RE: Consumer Reports Slammed for Creating 'Test' Viruses

[Nick FitzGerald <nick () virus-l demon co uk>]

Peter Kosinar wrote:

> Let's ignore the ethical point of view (= that they shouldn't have created 
> the viruses; regardless of the purpose of doing so) for now... 

OK...

> ... There are 
> two things which I find rather interesting from the scientific standpoint 
> (they might reveal what this test actually measured, if anything ;-) ).
>
> Quoting from the original announcement:
>
> > To pit the software against novel threats not identified on signature
> > lists, we created 5,500 new virus variants derived from six categories
> > of known viruses, the kind you'd most likely encounter in real life.
> >
> > That done, we unleashed the new viruses in our labs to see how well
> > the products detected them while scanning. Then we infected our lab
> > computer with each of 185 of them to see whether the products could
> > better detect viruses that were actively executing, based on their
> > behavior.
>
> Question #1: So, they created 5500 new variants but infected the computer 
> with 185 viruses... Why? What was so special about them?

Perhaps (presumably?) they were the 185 best-, or worst-, detected in 
the on-demand test?  The phrasing of the second part of your quoted 
second para suggests that the point of this small sub-test was to 
compare the effectiveness of detection of "new viruses" on-demand 
(passive scanning) vs. on-access (active scanning).

> Question #2: -How- did they create these "new variants"? I've seen tons of 
> amateur attempts to "evaluate how AVs can detect modified variants of 
> existing malware". In most cases, it turned out that they simply took an 
> existing piece of malware and got rid of -visible strings- by replacing 
> them with spaces/their own strings/etc... including strings like 
> "KERNEL32.DLL" or "GetProcAddress" used for imports ;-)

This is, of course, the $64,000 question and the one that will have 
immediately been raised in the minds of AV developers everywhere...

Your comments and observations are quite accurate as far as they go, 
but there are further, common, testing idiocies we see from ignorant 
testers.  A couple that immediately spring to mind both stem from the 
"let's grab several of the 'virus generator' kits and run them over and 
over then take all the pulp that falls on the floor and use that as our 
test-set" idea.  This approach has at least two very major flaws in 
terms of building a "detection test set".

First, many of these generators are themselves, very buggy.  Recall 
VBS/VBSWG.J (aka the "Anna Kournikova worm")?  If memory serves me 
right, the version of the VBSWG (VisualBasic Script Worm  Generator) 
kit the Dutch moron behind VBSWG.J used to "write" that virus for him 
had something like an 85% _misfire_ rate -- due to bugs in the kit 
itself, about 85% of the .VBS files it could generate would produce 
compile-time or early runtime errors that meant the files were non-
replicative.  (VBSWG was written by such a brilliantly talented haxor 
that there are versions that: cannot produce any code at all; produce 
.VBS files that are 100% guaranteed to not work; etc, etc.)  Thus, Jan 
de Wit (??), who later admitted he was too scared of screwing up his 
father's computer to even try the most rudimentary of tests of the code 
he produced with VBSWG before he released it, only had about a one-in-
six chance of actually gaining the infamy he garnered...

IMNSHO Markus Schmall's analysis at SecurityFocus:

   http://www.securityfocus.com/infocus/1287

is lacking in historical perspective of the kit's development and 
grossly overrates its "sophistication" and complexity, despite his 
opening comments about the kit's author being quite perceptive...

Second, some of these generators produce source-only (usually ASM) 
output, which is also non-infectious and thus (generally) not detected 
by virus scanners.

And, if you're going to make real sense of their justification of this 
approach to testing, you have to make sense of this claim:

   we created 5,500 new virus variants derived from six categories
   of known viruses, the kind you'd most likely encounter in real
   life.

It is true that, in general, most new viruses are variants of existing 
ones, usually made by taking the source of an earlier variant, tweaking 
it, rebuilding and possibly repacking and/or obfuscating (possibly 
multiple times).  However, it is NOT true that _only_ doing that 
produces new viruses that are in any sense representative of the actual 
new viruses that will be seen.  Of late it has been quite obvious that 
the "bad guys" go to a great deal of effort to test their new creations 
are NOT detected, at least by the subset of AV products they see as 
being most likely to thwart their evil "too early".  This is a real-
world bias that the developers of such a test could not possibly 
justify including in their test methods, as they would therefore be 
admitting to deliberately weighting the test against specific products, 
yet in not doing it they are deliberately producing what we know to be 
unrepresentative results which is sadly exactly the opposite result 
they (presumably) were expected (and contracted?) to produce.

Yep -- damned if you do bias the results aginst the biggest/most 
admired/etc vendors, and damned if you don't, so simply don't even try 
to do that kind of test.

Regardless of any of the other technical issues mentioned by Peter or 
me, the tester's lack of understanding of this point alone makes one 
question their suitability for performing any of the tests.

> Does anyone have more information about the approach taken by 
> ConsumerReports?

Presumably these guys do:

   http://securityevaluators.com/people.html

as ConsumerReports acknowledges ISE as the independent security experts 
consulted about (contracted to do?) the testing.  (I don't have a CR 
account and now can't find where I read that from the various bits of 
the actual report I've seen quoted various places...)

Avi Rubin is justifiably (in)famous for analysing the accidentally 
discovered source code of early Diebold voting machines, and his 
subsequent analyses of such machines and systems, but I have generally 
been far from impressed with the limited work on virus and antivirus 
issues I've seen from him.  The rest are unknown to me as far as being 
involved in AV issues goes.


Regards,

Nick FitzGerald

_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.