RE: An Open Letter (and Challenge) to the Application Security Consortium

["Michael Silk" <michaels () phg com au>]

Hi,

        I agree: I personally wouldn't purchase one either.

        However, the point of my note was that your (owasps') stance
seemed to be "use our criteria, it is far superior, and you may never be
able to meet it!" but also "don't use a criteria ... there is no silver
bullet.".

        I think that there probably needs to be a tougher criteria (but
I haven't seen the one proposed) however I think it would benefit the
community and employers more if a standard was legally *enforced* on
companies to prevent them over-estimating their abilities.

        Hence, rather then providing these companies with an
impossible-to-attain stamp, why not try and get laws passed to actually
*enforce* it. Because, while a lot of people in the security field might
be aware of OWASP, many aren't ... And even if they are, they may not
remember to think: "did this company pass the OWASP test ... ?" before
purchasing an IDS. If, however, it was a requirement for a company to
list the level of "security issues" (or whatever) it was able to
identify, then they wouldn't really have to think too hard at all.

        Furthermore, why would I, commercially, bother advertising the
fact that I can discover 3/536 tests on the OWASP test suite ? It could
only harm me; customers would have to investigate more about what I
*did* pass or may just do somewhere that has a higher number, prettier
box, nice-sounding receptionist or no mention of OWASP at all ...

        Of course, the difficulty with a legally enforcable standard is
that there are many issues to consider, and as new programming languages
and development platforms come out, old vulnerabilities disappear and
are replaced by new ones.

        And on the car analogy, it's not *really* the software
developers fault - it's the customers. We put up (and most of the time,
we have to..) with crappy, crashing, bug-ridden software - the software
developers just follow the path of least resistence and most money. If
the customers really pushed for legally liable software that always
worked the developers would have no choice ... of course, it's a bit
late for that now ... or is it ? :)

-- Michael


-----Original Message-----
From: Mark Curphey [mailto:mark () curphey com] 
Sent: Wednesday, 17 November 2004 1:36 PM
To: Michael Silk; owasp () owasp org; webappsec () securityfocus com
Subject: RE: An Open Letter (and Challenge) to the Application Security
Consortium

You have done lots in commentary over the years sir.

Its not about providing more issues, its about providing representative
issues. Its about a criteria that is truly representative of what the
common web security issues are. I agree these things are not *TOTALLY*
useless (although I would never buy one personally). Its about finding a
way to get the facts out there about what this set of technology can
actually do and holding that up against the big picture of web security
issues for all to make a fair unbiased judgement of their usefullness.

Imperva published a press release saying they cover the OWASP Top Ten a
while back if I am not mistaken. Thats just rubbish. Do they decrypt or
flag bad cryptography used in a cookie? No way. Do these products
protect from a case where object affinity results in one user getting
another account balance? No way. Its a data stream tech and only a small
subset of issues can be detected and protected in the data stream.

The challenge is about building a benchmark that is representative of a
real world site and the real world issues it faces. Those issues are not
always via a front-end web app. Its not about front-impact testing only.


This is saying if you are going to hold your products up as examples of
what you think the industry should accept, then hold them up against a
real set of criteria and not a convoluted set of things you know you
will shine against. 

You don't buy a car for safety because it has dent resistent side panels
!
And on that note now for some relevant humor as I read this again today
and laughed hard..

If Cars were built like applications (from Denis Verdon) 70% of all cars
would be built without following the original designs and blueprints.
The other 30% would not have designs.
Car design would assume that safety is a function of road design and
that all drivers were considerate, sober and expert drivers.
Cars would have no airbags, mirrors, seat belts, doors, roll-bars,
side-impact bars, or locks, because no-one had asked for them. But they
would all have at least six cup holders.
Not all the components would be bolted together securely and many of
them would not be built to tolerate even the slightest abuse. 
Safety tests would assume frontal impact only.  Cars would not be roll
tested, or tested for stability in emergency maneuvers, brake
effectiveness, side impact and resistance to theft.
Many safety features originally included might be removed before the car
was completed, because they might adversely impact performance.
70% of all cars would be subject to monthly recalls to add major
components left out of the initial production.  The other 30% wouldn't
be recalled, because no-one would sue anyway.
The after-market for safety devices would include such useful products
as training wheels, screen doors, elastic seatbelts and devices that
would restrict the car's top speed to 3mph, if found to be unsafe (which
would be always).
Useful safety could be found, but could only be custom retro-fitted,
would take six months to fit and would cost more than the car itself.
A DOT inspection would consist of counting the wheels and making
recommendations on wheel quantity.
Your only warning indicator would be large quantities of smoke and flame
in the cab.
You could only get insurance from one provider, it would be extremely
expensive, require a duplicate DOT inspection, and you might still never
be able to claim against the policy.



---- Michael Silk <michaels () phg com au> wrote:
> Hi,
>
>       Firstly let me say I would consider myself a member of owasp 
> (although I haven't really done much :)) but I'm not taking sides
here.
>       In this letter, you raise the point of " ... Building secure
software 
> requires deep changes in our development culture, including people, 
> processes, and technology ... ". Valid point, I thought.
>
>       In the next paragraph, however - and seemingly the main point of
this 
> article - you go on to say that your "solution" is to create a tool 
> that just covers more vulnerabilities then that provided by the ASC
> (btw: are there any links to what they suggested?) and that this tool 
> should be used by these application firewalls (which you previously 
> suggested a 'not very' useful). This leaves me confused ... What was 
> the point here ? To say that they aren't covering enough technical 
> vulnerabilities (sql injection, etc) or that they are ignoring the 
> most important factor - a sound security design.
>
>       To me, it seems you are attempting to provide them with, to use
your 
> words, the "... elusive silver bullet" that you claim they shouldn't 
> be searching for. If, however, your submission succeeds and these 
> corporations use the OWASP Testing Application it would be more 
> powerful then it otherwise would've been: they can get a big shiny 
> stamp from OWASP saying "We Passed!".
>
>       Don't get me wrong, I'm not against having OWASP, or whoever, 
> providing a comprehensive application that mimics common 
> vulnerabilities
> - but I'm just not sure what the point of this letter was and I am 
> wondering whether OWASP really wants to provide the companies with 
> these stamps of approval.
>
>
> ------------------
>
>
>       The issue, however, seems to be that of companies rating
themselves. 
> Other industries have covered this issue  ... Enegry ratings, car 
> safety, etc ... Perhaps there could be some discussion of how to 
> formalise these rating for the application security community ...
> In Australia something like this would be enforced by the ACCC 
> (Australian Competition and Consumer Commission), I'm sure America 
> would have a similar organisation .. Perhaps proposals could be made ?
>
> -- Michael
>
>       
>
>        
>
> -----Original Message-----
> From: The OWASP Project [mailto:owasp () owasp org]
> Sent: Tuesday, 16 November 2004 2:34 PM
> To: webappsec () securityfocus com
> Subject: An Open Letter (and Challenge) to the Application Security 
> Consortium
>
> An Open Letter (and Challenge) to the Application Security Consortium
>
> Since its inception in late 2000 the Open Web Application Security 
> Project (OWASP) has provided free and open tools and documentation to 
> educate people about the increasing threat of insecure web 
> applications and web services. As a not-for-profit charitable 
> foundation, one of our community responsibilities is to ensure that 
> fair and balanced information is available to companies and consumers.

> Our work has become recommended reading by the Federal Trade 
> Commission, VISA, the Defense Information Systems Agency and many 
> other commercial and government entities.
>
> The newly unveiled Application Security Consortium recently announced 
> a "Web Application Security Challenge" to other vendors at the 
> Computer Security Institute (CSI) show in Washington, D.C. This group 
> of security product vendors proposes to create a new minimum criteria 
> and then rate their own products against it.
>
> The OWASP community is deeply concerned that this criteria will 
> mislead consumers and result in a false sense of security. In the 
> interest of fairness, we believe the Application Security Consortium 
> should disclose what security issues their products do not address.
>
> As a group with a wide range of international members from leading 
> financial services organizations, pharmaceutical companies, 
> manufacturing companies, services providers, and technology vendors, 
> we are constantly reminded about the diverse range of vulnerabilities 
> that are present in web applications and web services. The very small 
> selection of vulnerabilities you are proposing to become a testing 
> criteria are far from representative of what our members see in the 
> real world and therefore do not represent a fair or suitable test
criteria.
> In fact, it seems quite a coincidence that the issues you have chosen 
> seem to closely mirror the issues that your technology category is 
> typically able to detect, while ignoring very common vulnerabilities 
> that cause serious problems for companies.
>
> Robert Graham, Chief Scientist at Internet Security Systems, recently 
> commented on application firewalls in an interview for CNET news. When

> asked the question "How important do you think application firewalls 
> will become in the future?" his answer was "Not very."
>
>
> "Let me give you an example of something that happened with me. Not 
> long ago, I ordered a plasma screen online, which was to be shipped by

> a local company in Atlanta. And the company gave me a six-digit 
> shipping number. Accidentally, I typed in an incremental of my 
> shipping number (on the online tracking Web site). Now, a six-digit 
> number is a small number, so of course I got someone else's user 
> account information. And the reason that happened was due to the way 
> they've set up their user IDs, by incrementing from a six-digit 
> number. So here's the irony: Their system may be so cryptographically 
> secure that (the) chances of an encrypted shipping number being 
> cracked is lower than a meteor hitting the earth and wiping out 
> civilization. Still, I could get at the next ID easily. There is no
application firewall that can solve this problem.
> With applications that people are running on the Web, no amount of 
> additive things can cure fundamental problems that are already there 
> in the first place."
>
> This story echoes some of the fundamental beliefs and wisdom shared by

> the collective members of OWASP. Our experience shows that the 
> problems we face with insecure software cannot be fixed with
technology alone.
> Building secure software requires deep changes in our development 
> culture, including people, processes, and technology.
>
> We challenge the members of the Application Security Consortium to 
> accept a fair evaluation of their products. OWASP will work with its 
> members (your customers) to create an open set of criteria that is 
> representative of the web application and web services issues found in

> the real world. OWASP will then build a web application that contains 
> each of these issues. The criteria and web application will be 
> submitted to an independent testing company to evaluate your products.

> You can submit your products to be tested against the criteria 
> (without having prior access to the code) on the basis that the 
> results are able to be published freely and will unabridged.
>
> We believe that this kind of marketing stunt is irresponsible and 
> severely distracts awareness from the real issues surrounding web 
> application and web services security. Corporations need to understand

> that they must build better software and not seek an elusive silver 
> bullet.
>
> We urge the Consortium not to go forward with their criteria, but to 
> take OWASP up on our offer to produce a meaningful standard and test 
> environment that are open and free for all.
>
> Contact: owasp () owasp org
> Website: www.owasp.org
>
>
>
>
>
>
> **********************************************************************
> This email message and accompanying data may contain information that
is confidential and/or subject to legal privilege. If you are not the
intended recipient, you are notified that any use, dissemination,
distribution or copying of this message or data is prohibited. If you
have received this email message in error, please notify us immediately
and erase all copies of this message and attachments.
> This email is for your convenience only, you should not rely on any
information contained herein for contractual or legal purposes. You
should only rely on information and/or instructions in writing and on
company letterhead signed by authorised persons.
> **********************************************************************