RE: How secure is software X?

["Ferguson, Justin (IARC)" <FergusonJ () nv doe gov>]

David,

One thing you have to keep in mind is that a lot of things are incredibly
variable when dealing with this subject. For instance, suppose you want to
ensure that the URI in a web server is not overflowable. So you test with
something like

GET /[AAAAAAAAA x 4096] HTTP/1.1
Host: foobar.com
Connection: close

This is all fine and well, unless at 8192 is where the overflow takes place,
or if I can stick as many characters as I want in, so long as I am using
HTTP 1.1 and not HTTP 0.9, or if I am using HTTP/1.1 and Host doesn't
contain a 36 backslashes, et cetera.

This is generally why fuzzing is mostly inconclusive because it often misses
a lot of conditions and you have essentially assured nothing. Without
in-depth analysis of each software package you are basically pushing snake
oil. There are just far too many variables to really standardize such a
thing.


Best Regards,

Justin Ferguson
Reverse Engineer
NNSA IARC
702.942.2539

"It is a capital mistake to theorize before one has data. Insensibly one
begins to twist facts to suit theories, instead of theories to suit facts."
-- Sir Arthur Conan Doyle

> -----Original Message-----
> From: Adam Shostack [mailto:adam () homeport org] 
> Sent: Friday, May 12, 2006 11:35 AM
> To: David Litchfield
> Cc: bugtraq () securityfocus com; 
> full-disclosure () lists grok org uk; 
> ntbugtraq () listserv ntbugtraq com; dbsec () freelists org
> Subject: Re: How secure is software X?
>
>
> Hi David,
>
> Very briefly because I'm swamped today:  Please consider 
> bringing some of this to Metricon
> (https://securitymetrics.org/content/Wiki.jsp?page=Welcome)
>
> Also there's a project of US DHS/NIST and probably others 
> called SAMATE Software Assurance Metrics and Tool Evaluation 
> http://samate.nist.gov/index.php/Main_Page
>
> which might be of interest.
>
> Adam
>
> On Fri, May 12, 2006 at 02:59:17AM +0100, David Litchfield wrote:
> | How secure is software X?
> | 
> | At least as secure as Vulnerability Assessment Assurance 
> Level P; or Q 
> | or
> | R. Well, that's what I think we should be able to say. What 
> we need is an 
> | open standard, that has been agreed upon by recognized 
> experts, against 
> | which the absence of software security vulnerability can be 
> measured - 
> | something which improves upon the failings of the Common 
> Criteria. Let's 
> | choose web server software as an example. When looking for 
> flaws in a new 
> | piece of web server software there are a bunch of well 
> known checks that 
> | one would throw at it first. Try directory traversal 
> attacks and the 
> | several variations. Try overflowing the request method, the 
> URI, the query 
> | string, the host header field and so on. Try cross site 
> scripting attacks 
> | in server error pages and file not found messages. As I 
> said, there's a 
> | bunch of checks and I've mentioned but a few. If these were 
> all written 
> | down and labelled with as a "standard" then one could say 
> that web server 
> | software X is at least as secure as the standard - 
> providing of course the 
> | server stands up.
> | 
> | For products that are based upon RFCs it would be trivial 
> to write a 
> | simple
> | criteria that tests every aspect of the software as per the 
> RFCs. This 
> | would be called Vulnerability Assessment Assurance Level: 
> Protocol. If a 
> | bit of software was accredited at VAAL:Protocol  then it 
> would given a 
> | level of assurance that it at least stood up to those attacks.
> | 
> | Not all products are RFC compliant however. Sticking with 
> web servers, 
> | one
> | bit of software might have a bespoke request method of 
> "FOOBAR". This opens 
> | up a whole new attack surface that's not covered by the 
> VAAL:Protocol 
> | standard. There are two aspects to this. Anyone with a 
> firewall capable of 
> | blocking non-RFC compliant requests could configure it to 
> do so - thus 
> | closing off the attack surface - from the outside at least. 
> As far as the 
> | standards go however - you'd have to introduce criteria to 
> cover that 
> | specific functionality. And what about different 
> application environments 
> | running on top of the web server? And what about more 
> complex products such 
> | as database servers? I suppose at a minimum for DB software 
> you could at 
> | least have a standard that simply checks if the server 
> falls to a long 
> | username or password buffer overflow attempt and then fuzz 
> SQL-92 language 
> | elements. It certainly makes standardization much more 
> difficult but I 
> | think by no means impossible.
> | 
> | Clearly, what is _easy_ is writing and agreeing upon a VAAL:Protocol
> | standard for many different types of servers. You could 
> then be assured 
> | that any server that passes is at least as secure as 
> VAAL:Protocol and for 
> | those looking for more "comfort" then they can at least 
> block non-RFC 
> | compliant traffic.
> | 
> | Having had a chat with Steve Christey about this earlier 
> today I know 
> | there
> | are other people thinking along the same lines and I bet 
> there are more 
> | projects out there being worked on that are attempting to 
> achieve the same 
> | thing. If anyone is currently working on this stuff or 
> would like to get 
> | involved in thrashing out some ideas then please mail me - 
> I'd love to hear 
> | from you.
> | 
> | Cheers,
> | David Litchfield
> | http://www.databasesecurity.com/
> | http://www.ngssoftware.com/