Bugtraq mailing list archives
Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?)
From: "Steven M. Christey" <coley () mitre org>
Date: Mon, 27 Nov 2006 17:14:43 -0500 (EST)
Large-scale comparisons using historical data, while suggestive, have certain limitations. I touched on many of these in my open letter on the interpretation of vulnerability statistics [1] when talking about trend analysis in vulnerability databases, but many of the points apply here. For example, there appears to be distinct difference in editorial policy between Oracle and Microsoft in terms of publishing vulnerabilities that the vendors discovered themselves, instead of third parties. This might produce larger numbers for Oracle, which appears to include internally discovered vulnerabilities in their advisories, whereas this is not necessarily the case for Microsoft [2], [3]. In both cases, the lack of details can mean that multiple issues wind up with one public identifier; for example, Oracle Vuln# DB01 from CPU Jul 2006 (CVE-2006-3698) might involve 10 different issues, and this is not an isolated case. This can further muddy the waters. Another difficulty that I originally mentioned involved possible differences in research community bias. I don't think we really know how many researchers are looking at which product, what their skill levels are, and how much effort they're putting into looking. David touches on this topic briefly: Do the SQL Server 2005 results have no flaws because no-one is looking at it? No. I know of a number of good researchers are looking at it. SQL Server code is just more secure than Oracle code. It would be nice if the research community could begin to quantify the level of effort they are putting into their analyses. If researchers are putting 10 times more effort into one product than another, then you might expect to find 10 times more issues (assuming the products started at the same baseline of quality). In short, we need better metrics before we can really compare the relative *inherent* security of products. (The work from Wing, Manadhata, and Howard for measuring relative attack surface shows promise). However, public stats are the best we have for now. - Steve [1] Open Letter on the Interpretation of "Vulnerability Statistics" http://lists.grok.org.uk/pipermail/full-disclosure/2006-January/041028.html [2] Skeletons in Microsoft.s Closet - Silently Fixed Vulnerabilities Steve Manzuik, eEye http://www.blackhat.com/presentations/bh-europe-06/bh-eu-06-Manzuik.pdf [3] Microsoft Patches: When Silence Isn't Golden Ryan Naraine, eWeek http://www.eweek.com/article2/0,1895,1951186,00.asp
Current thread:
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) stopmakingnoise (Nov 24)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Steve Friedl (Nov 25)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Tim Newsham (Nov 27)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) David Litchfield (Nov 27)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Tim Newsham (Nov 27)
- <Possible follow-ups>
- Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Thor (Hammer of God) (Nov 25)
- Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP] (Nov 25)
- Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Thor (Hammer of God) (Nov 25)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Steven M. Christey (Nov 28)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) David Litchfield (Nov 28)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) David Litchfield (Nov 29)
- RE: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Shawn Fitzgerald (Nov 29)
- Re: Re: "Which is more secure? Oracle vs. Microsoft" (is it a fair comparison?) Steve Friedl (Nov 25)