WebApp Sec mailing list archives
Re: benchmarking the web app scanners
From: Dinis Cruz <dinis.cruz () googlemail com>
Date: Sat, 21 Jan 2006 01:47:41 +0000
Thomas, Thanks for you kind comments :) I was going to wait a little bit before I posted this ( http://owasp.net/forums/428/ShowPost.aspx) information here (webappsec () securityfocus com), but since the genie is now out of the bottle, here are a series of quick comments: 1) this tool has been sponsored by Foundstone, BUT (and it is a bit but) it is being released under the Owasp .Net Project and an Open Source Licence. So Kudos for Foundstone for doing this and I hope they get good exposure from it 2) The main objective of the tool is to create dynamic websites based on XML files which will 'map' to a database containing hundreds of different vulnerabilities (some simple to detect/exploit, some harder) covering multiple languages and web development architectures (for example navigation: Html, JavaScript, Flash, Java, etc...) 3) There are many ways this tool can be used, here are just a couple starting ideas: a) As a training tool since it allows the creation of multiple websites with multiple variations of vulnerabilities b) As a Web Application Honeypot (since we are able to create dynamic ( i.e. false) websites and track / monitor in real-time all requests made) c) As a test ground for newly discovered vulnerabilities types and its exploit vectors d) As a benchmark for Web Security Scanners 4) The Web Security Scanner benchmarking and testing is the most obvious short-term application for this tool, but I think that as it evolves the others will be proven to be as (if not more) valuable 5) On the Web Security Scanner issue: a) My main hope is that the Web Security Scanner Companies will see this tool as an opportunity and work with the Owasp .Net project (and other groups that want to be involved) in a productive and constructive way. b) Although in the short term some Web Security Scanners might have some bad results (well, at least when compared with what their Marketing machine publishes :) in the medium term, as they adapt and improve their scanning techniques, everybody will benefit c) One of the core objectives of the tool (when thinking about benchmarking Web Security Scanners) is to be able to create real and measurable metrics. For example: - Scanner X was able to detect 65% of the vulnerabilities where Scanner Y was able to detect 90% - Scanner X made 10000 to detect those 65% (over a period of 16h) where scanner Y made 4000 request (over a period of 10h) - 20% of Scanner X results where false positives, where Scanner Y had 50% false positives - Scanner X was able to deal with Html and JavaScript navigation, Scanner Y was able to deal with Html, JavaScript and Flash, and both where NOT able to deal with Java based navigation systems - Scanner X is not able to go more than 40 levels deep, Scanner Y is able to go up at least 100 levels deep (if note more) - etc, etc, etc. d) There will be two main types of tests that can be done in the short term: i) provive the links to all different types of vulnerabilities existent in the database, and see how many can the scanner correctly identify? and ii) when multiple types of website architectures and navigation techniques are used, how many vulnerabilities is the scanner able to detect? e) In order to test (and further improve the tool) I want to take this opportunity to ask the Web Application Security Scanners that subscribe to this list (which I believe all do) to give the Owasp .Net project a temporary licence to they product so that we can use it during development and during some basic benchmarking that we might do (and NO, I will not sign an NDA that doesn't allow me to publish the data collected, in fact I will not sign ANY NDA with ANY web application security scanner company) f) note that at the moment I (Dinis) have no plans to do a full benchmarking exercise since I don't have the time required, but I know of at least one group of experienced security consultants which is starting such project (and I will be supporting them). If anybody else is interested in doing a similar benchmarking project please contact me directly 6) Regarding how the tool works, here is a brief technical description: There are two main components: A webserver (which can be IIS or a custom webserver) and a GUI application (written in C# 2.0). The Gui Application is responsible for handling all mappings (from the virtual requests to the actual pages on disk) and there is an unmanaged C++ DLL loaded by both which implements a Shared Memory to send and receive data between them. The current version is hardcoded to IIS, although in the code there is support for using a custom .Net webserver. This IIS version uses an HttpHander to capture all requests and communicate with the GUI Application (called SiteGeneratorGUI). The previous version used C++ Detours to hook all sorts of functions in either IIS or the Custom Webserver (this worked ok, but ultimately I decided to use IIS since it was much more robust and scalable) The dynamic websites are defined by XML files like this (which are edited on the GUI Application using the WYSIWYG Altova Authentic Browser Object (SPS files created via Altova's StyleVision application)): <?xml version="1.0" encoding="utf-8" ?> <SiteGenerator name="SiteGenerator Demo" xmlns:ipo="http://www.altova.com/IPO" xmlns=" http://www.xmlspy.com/schemas/orgchart" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <site> <folder name=""> <file mappedTo="aspx/Default.aspx" name="HelloWorld.aspx" /> <folder name="htm" /> <folder name="aspx"> <file mappedTo="aspx/pages.htm" name="pages.htm" /> <file mappedTo="aspx/xss.aspx" name=" xss.aspx" /> <file mappedTo="aspx/SqlInjection_Easy.aspx" name="SqlInjection.aspx" /> <file mappedTo="aspx/SqlInjection_Hard.aspx" name=" SqlInjection2.aspx" /> </folder> <folder name="flash"> <file mappedTo="flash/cromas_xml.swf" name="cromas_xml.swf" /> <file mappedTo="flash/cromas_xml.htm" name="menu.htm" /> <file mappedTo="/flash/cromas_menu.xml" name="cromas_menu.xml" /> </folder> </folder> </site> </SiteGenerator> SiteGeneratorGUI.exe and IIS will map the virtual name "HelloWorld.aspx" to the file on disk "aspx/Default.aspx" . For example: http://localhost/HelloWorld.aspx --> F:\Owasp SiteGenerator\SiteGenerator_ContentPages\aspx\Default.aspx So to create new websites all you need to do is to create a new XML file And to create new vulnerabilities type, all you need to create in an Aspx page and map it to the xml file 7) the tool is still in Beta, so please be patient with it. The code is still a bit in mess, since there are multiple past experiments in there which I will need to clean up 8) What we need now is: a) comments on how the tool works b) new vulnerabilities (i.e. new Aspx pages) c) help in the development of the missing features (namely an Xml Metadata section to contain details about each vulnerabilities (risk rating, remediation, exploit difficulty level, etc...). I also want to add support for other types of languages and servers (at the moment it is limited to .Net Framework and IIS (I had ASP Classic in one of the earlier betas, but didn't include it in this one since it was implemented using the detours technique)). I have many more ideas on what is comming next, but first I want to make sure that what this version is fully functional and usable I hope this make since (it's 1:30am here and I just wanted to do a quick brain dump with some ideas and pointers about SiteGenerator) Best regards Dinis Cruz Owasp .Net Project On 20/01/06, thomas.jones () hushmail com < thomas.jones () hushmail com> wrote:
This tool looks awesome. About time someone built this! Has anyone got a plan in place to provide some independent, objective results rather than the marketing hype I have to read from the likes of eWeek? Dinis you rock ! http://owasp.net/forums/428/ShowPost.aspx Concerned about your privacy? Instantly send FREE secure email, no account required http://www.hushmail.com/send?l=480 Get the best prices on SSL certificates from Hushmail https://www.hushssl.com?l=485 ------------------------------------------------------------------------- This List Sponsored by: Watchfire Watchfire's AppScan is the industry's first and leading web application security testing suite, and the only solution to provide comprehensive remediation tasks at every level of the application. See for yourself. Download AppScan 6.0 today. https://www.watchfire.com/securearea/appscansix.aspx?id=701300000003Ssh --------------------------------------------------------------------------
------------------------------------------------------------------------- This List Sponsored by: Watchfire Watchfire's AppScan is the industry's first and leading web application security testing suite, and the only solution to provide comprehensive remediation tasks at every level of the application. See for yourself. Download AppScan 6.0 today. https://www.watchfire.com/securearea/appscansix.aspx?id=701300000003Ssh --------------------------------------------------------------------------
Current thread:
- benchmarking the web app scanners thomas.jones (Jan 20)
- Message not available
- Re: benchmarking the web app scanners Dinis Cruz (Jan 23)
- Message not available