Webscarab development continues

["Dawes, Rogan (ZA - Johannesburg)" <rdawes () deloitte co za>]

Hi folks,

After some discussion with Ingo Struck about structuring packages, and so
on, I would like to invite you to look at my proposed structure (and initial
implementation) for the new WebScarab.

You will see obvious links back to the current Exodus code, as I have reused
what seems worth reusing from my previous efforts.

You can find the latest archive at
http://home.intekom.co.za/rdawes/webscarab-20030728-0820.jar. There is also
a link from my exodus page (http://home.intekom.co.za/rdawes/exodus.html)
which will be updated as I progress. (Look for it after the BOLD section
where I explain that future development will go into WebScarab :-)

This .jar should be runnable, and provides:

* the webscarab framework (user interface independent), 
* the WebScarabPlugin framework (user interface independent), 
* a Proxy WebScarabPlugin implementation (no SSL yet)
* the ProxyPlugin framework (user interface independent), 
* two sample ProxyPlugins (ManualEdit and RevealHidden) (not quite user
interface independent - see the comments in the ManualEdit proxyplugin)
* a sample Swing webscarab UI, with some panels that interact with the
underlying plugins

Part of the model is also implemented:
* "Conversation" holds what we know about a particular conversation,
including the Request and Response. It will eventually hold a parsed version
of the Response content (as flexibly as possible, to cope with various
content-types - I would appreciate help here!)
* "URLInfo" holds what we know about a particular URL. It is a summary of
all the Conversations that have been seen (analogous to the Site view panel
in Exodus, I guess). E.g. it will record the various methods seen, that
generated anything other than "method not supported", list the total
(content) bytes received as responses to requests for that URL, checksums of
the content, etc. 

Each WebScarabPlugin gets a chance to analyse a Conversation as it is seen,
and can summarise whatever information it wants to into the URLInfo. The
presentation layer will then need to show a column with that information in
it, or save it out, or whatever. This is currently implemented as a Property
class, so you can use a fairly arbitrary string to index your information.

Major things that need to be implemented still:

* HTML parser - I'm thinking of a Tokeniser approach, that could return an
array of Tags, which each plugin can iterate through. The Tags will be used
to extract Links (for use by the Spider), find XSS, ODBC error messages, etc
* Readers and Writers (so we can save and resume a session)
* a decent conversation cache, so we can dump the raw requests and responses
to save memory, but read them back if requested.
* Various views into the model - showing conversation history (a table of
Conversations, effectively), URL properties, etc
* Various plugins - such as those from the current Exodus, as well as
others. In particular the Spider will be a good one to get started on.
* a "shared browser state", that can be used by the Spider and Proxy plugins
to synchronise Cookies (if the Proxy sees a Set-Cookie, the Spider can use
it for future requests, if the Spider sees a Set-Cookie, the Proxy will
inject it into future requests, as well as back to the browser)
* interfaces to the above

Any volunteers?

All comments are welcome!

Rogan

P.S. I originally sent this mail on Monday morning, but it didn't go
through. Since then Ingo and I have been busy checking the code into the
Sourceforge CVS repository, under webscarab. However, the build scripts have
not yet been updated to reflect the new code. Consequently, it is unlikely
that everything will build successfully at this point. The code in CVS is
essentially the same as that in the .jar mentioned above, so please get that
if you want to see how it works so far. Thanks!

-- 
"Using encryption on the Internet is the equivalent of arranging an 
armored car to deliver credit card information from someone living 
in a cardboard box to someone living on a park bench."
  - Gene Spafford
-- 
Deloitte & Touche Security Services Group
Tel: +27(11)806-6216     Fax: +27(11)806-5202     Cell: +27(82)784-9498
-- 

Important Notice: This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") 
that must be accessed and read by clicking here or by copying and pasting the following address into your Internet 
browser's address bar: http://www.Deloitte.co.za/Disc.htm. The Disclaimer is deemed to form part of the content of this 
email in terms of Section 11 of the Electronic Communications and Transactions Act, 25 of 2002. If you cannot access 
the Disclaimer, please obtain a copy thereof from us by sending an email to ClientServiceCentre () Deloitte co za.