Dailydave mailing list archives
Re: Graphing: Don't believe everything you see.
From: jf <jf () danglingpointers net>
Date: Thu, 8 Feb 2007 03:03:05 +0000 (UTC)
Really, almost all of these metrics are flawed- of the critical vulnerabilities listed many of them are things like critical bug in OpenSSL, problems in ftp proxy with IPv66 sockets, et cetera; which I guess depending on who you are, may or may not be critical, but to most of us who aren't using any type of proxy or IPv66 sockets, it's not so important. This is important to take into account when reviewing those number of critical bugs comparisons. If we compare MS Office to OpenOffice in this light, it would show that OO is greatly superior in security to MS Office because of the number of critical flaws found, but I'd be willing to bet that many of us may not necessarily agree with that conjecture. The number of reported bugs are just that, and shouldn't be used as a metric to determine if a product is secure or not. (however, when a bug is reported and then some time in the distant future another similar bug affecting the same region of code does indicate a failure on the vendors part to really care at all, which IMHO is a much better metric) Then we have things like 'time to patch' metrics, which are also flawed, for instance does MS release patches for third-party products, or rather if there is (yet another) bug in a CA product and MS doesn't patch it, do we count that against them? Why do we do that for Redhat? Maybe that isn't the best point as Redhat did indeed ship with a product, but where does responsibility lie? What if the bug is on the 'extras' CD in an unstable directory, do we count that? How about if it took organization Y several weeks to produce a patch for their product and then in less than Z hours the OS vendor provides the patch to their customers, do we count the time as 'several weeks' or Z hours? That all said, because of different models, comparing time to patch for Windows to Linux/BSD/any of the OSs that comprise of mostly third party applications provides a false view of the situation. As for the graphs, they provide an idea of the potential amount of bugs, but provide no real firm data. Speaking in a sense of probability of course. To declare however that one product is more secure than another simply based off of a graph like that is absurd and silly, and I think everyone realizes this. -- Success is not final, failure is not fatal: it is the courage to continue that counts. -- Sir Winston Churchill On Wed, 7 Feb 2007, Robert E. Lee wrote:
Date: Wed, 07 Feb 2007 17:05:46 +0100 From: Robert E. Lee <robert () dyadsecurity com> To: dailydave () lists immunitysec com Subject: Re: [Dailydave] Graphing: Don't believe everything you see. George Ou wrote: > Take a look at Microsoft SQL 2005 and you'll see that's been ROCK SOLID with > ZERO vulnerabilities. > http://secunia.com/product/6782/?task=advisories > Compare that to the mess of Oracle over the same time period. > > So let's not base our analysis on some stupid trumped up diagram and let's > not make stupid generalizations about platforms. Let's try and be objective > and factual. In the spirit of "[silly] generalizations".... the number of vulnerabilities publicly disclosed for a product doesn't seem to be a valid metric for measuring security between products. There are different disclosure policies for every organization/product. Some applications are just going to get more attention than others. Closed source vs Open Source changes the methods available to an outside researcher for testing. For results to be compared, the same tests have to be run equally for both projects. Comparing the end result (vulnerability count) without taking into account how we got to the end result (testing methodology) reminds me a bit of: "If... she... weighs... the same as a duck,... she's made of wood. And therefore? A witch!!!" Cheers :), Robert
_______________________________________________ Dailydave mailing list Dailydave () lists immunitysec com http://lists.immunitysec.com/mailman/listinfo/dailydave
Current thread:
- Graphing: Don't believe everything you see. Dave Aitel (Feb 06)
- Re: Graphing: Don't believe everything you see. Felix von Leitner (Feb 06)
- Re: Graphing: Don't believe everything you see. dan (Feb 07)
- Re: Graphing: Don't believe everything you see. Adam Shostack (Feb 07)
- Message not available
- Re: Graphing: Don't believe everything you see. Adam Shostack (Feb 08)
- Re: Graphing: Don't believe everything you see. Douglas F. Calvert (Feb 09)
- Re: Graphing: Don't believe everything you see. dan (Feb 07)
- Re: Graphing: Don't believe everything you see. Felix von Leitner (Feb 06)
- Re: Graphing: Don't believe everything you see. Robert E. Lee (Feb 07)
- Re: Graphing: Don't believe everything you see. jf (Feb 07)
- Re: Graphing: Don't believe everything you see. LMH (Feb 07)
- Re: Graphing: Don't believe everything you see. Dave Aitel (Feb 09)
- Re: Graphing: Don't believe everything you see. Ed Ray (Feb 12)