Re: Consumer Reports Slammed for Creating 'Test' Viruses

[Peter Kosinar <goober () nuf ksp sk>]

Hello,

The following response is based on my observation, referring to "people" 
does not imply that it reflects my or your point of view. :-)

> *I* don't think that, generally speaking.  (I seriously doubt that no 
> one, ever, working for an AV company hasn't written or modified some 
> malware.  But generally, no, I don't believe they are creating the 
> malware.)

Actually, there are documented cases when former virus-writer(s) were 
employed by AV companies. On the other hand, I seriously doubt that there 
are people who wrote a virus -while- working for an AV company. Once 
someone sees the amount of malicious software (s)he needs to fight 
against, the slightest thoughts of adding to it get kicked out very 
quickly. :-)

> However, that is a HUGE reason why AV people are so paranoid about 
> creating malware, because of 20 years of people waiting to pounce the 
> moment there is a hint that they do.

Indeed; it's happens too often that the first reaction of laymen to people 
working in the AV industry is like this: "I work for an AV company." "Oh 
really? So, just between us, you do write viruses, don't you? So that 
you'll get more profit... How many of them have you written?". It's a 
shame, but it is so :-(

> > people think that AV products should be made so they don't need updates. 
> *I* don't think that.  I think that AV relies almost entirely on signature 
> updates.  However, if there is going to be any claim for detection for 
> unknown malware, then that claim is fair game for testing.

I have to agree with Drsolly here -- it seems that large fraction of 
people really thinks that AV is a kind of "fire-and-forget" thingy. On the 
other hand, they tend to think the same of "security" in general and it's 
usually quite difficult to make them understand the Schneier-attributed 
quote: "Security is a process, not a product.".

Blue Boar is right in assuming that AVs rely on updates (the distinction 
between the "scanning engine", "signatures", and other parts of the AV is 
disappearing over the time). Undoubtedly, it is -necessary- to update the 
AV once an undetected piece of malware appears. And it will... unless the 
AV detects everything; in that case, it'll get updated when their users 
bash them because of false positive :-)

> > A lot of the comments are "The AV companies don't like to see third party 
> > testing, and that's why they're against this test."
>
> I can see where some people would think that, but I don't find it to be a 
> particularly strong argument.

Let's make a mental experiment... There is a comparative test of a similar 
product from three vendors (A,B,C), each of them claiming "totally great 
ultimate functionality" (sorry, can't make up any meaningful market-speech 
:-) ) in their advertisements and the results of the test are, on 
100-point scale, as follows:

1. Vendor A - 83 points
2. Vendor B - 62 points
3. Vendor C - 11 points

If Vendor C starts criticizing the test as being performed by incompetent 
idiots, what do you think that majority of people will think? My 
experience is that it'll be: "They just performed much worse than anyone 
else in the test, it's obvious they're bad; they're just trying to save 
their bums.". Depending on the nature of test, vendor A might choose one 
of two stances: Either they'll use the results of this test for their 
marketing ("See? We outperformed everyone else in an -independent- test!") 
or they'll complain about the quality of the test, along with B and C.

In that case, still a large percentage of people (although much lower than 
in the case of vendor C) will think that A' critics is unfounded; that 
vendor A is just complaining because their results weren't in the 97-99 
range (as they used to be in other tests).

> I don't think I'm particularly worrying about the ethical question, I'm 
> trying to find out why the test is not valid, strictly for determining 
> functionality.

I have no reasons to doubt the validity of their test. It's the conclusion 
which they draw from it, that can be invalid. Without knowing more 
details, the only conclusion I can draw from the test is: "The test show 
the detection ratio of AVs on a set of 5500 files." Somehow, I don't 
consider this result to be very useful.

As Nick and I pointed out in our previous replies, it's -incredibly- easy 
to make the test meaningless from the technical point of view. Quick 
summary: Did they check each of those 5500 pieces of malware and actually 
verified that they work -and- that they perform their malicious activity?

If they did not, the results do not show the actual detection rates, 
rather some vague lower values. If you want to argue that the actual 
number of malicious files does not matter in relative order (a better AV 
will always detect more malware pieces than the worse one), read 
further...

One easy-to-think-up bad idea is to take the files, run all the AVs on 
them and delete the files which weren't detected by any of the 
participating AVs. This "ensures" that your testing set consists only of 
malware. Or not?

The answer is obvious -- each and every useful AV has false positives 
(yes, I can make an AV which has no false positives but it'll be a huge 
monster, thus not -useful-), so your testing set is, most likely,
contamined by clean files. In fact, if one of the tested products is 
extremely false-positive-rich, it'll probably score very well in the 
subsequent scanning of the testing set prepared in this way.

In other words, you shall -not- use the AVs products you're testing for 
creation of the test-set. Unfortunately, it also means that the testers 
need to put a lot more effort into performing the test than they might 
have expected.

If CR did so, they get my respect, they've performed better than dozens of 
other "amateur" tests. They can safely claim that: "Our test shows how the 
AVs performed on a set of 5500 dangerous files which are similar to the 
malware commonly found on the Internet."

If they want to be honest, they should also point out that the results of 
their test do NOT necessarily reflect how the AVs perform on real-life 
unknown malware (unfortunately, people tend to jump to this conclusion 
very quickly). It's the distribution of the actual families of malware 
which wasn't taken into account. As Larry pointed out, detecting a variant 
of some very rare family (e.g. a plain file infector infecting just files 
in current directory on April 1) is not the same as detecting a new 
variant of a worm spreading via a largely-unpatched hole without any kind 
of user-interaction. However, if you put them side-by-side in your test, 
their weight will be the same and missing the former will be considered 
as bad as missing the other one.

If CR did take this into account as well (which I -really- doubt), I'd say 
they deserve a medal of honor :-)

> > > OK, I'm not sure what would be qualitatively different about me the
> > > virus author, versus the natural self-selected population of virus
> > > authors, but at least I understand your position better.  For the
> > > record, I wasn't trying to hint that I could write some
> > > uber-polymorphic-super virus.  I'm under the impression that I could
> > > write some 80's-style file infecter, and as long as it's original, it
> > > wouldn't be detected.

Actually, Ryan, assuming that by 80's-style file infector you mean an 
infector for MS-DOS-running machines of those days -and- using the 
techniques common in those days, I doubt it'll be undetected by all the 
AVs. Yes, it is possible to write such a thing (and it is not all that 
difficult) with current knowledge and ideas but if you really adhered to 
the virus-writing principles used then, the result will be quite likely to 
be detected.

Peter

--
[Name] Peter Kosinar   [Quote] 2B | ~2B = exp(i*PI)   [ICQ] 134813278
_______________________________________________
Fun and Misc security discussion for OT posts.
https://linuxbox.org/cgi-bin/mailman/listinfo/funsec
Note: funsec is a public and open mailing list.