Re: Handling of arbitrary content in analysis tools (WebScarab) Was RE: SOAP protocol analyzer

["Jeff Williams @ Aspect" <jeff.williams () aspectsecurity com>]

Rogan,

We built something like this into our PointGuard
(http://www.aspectsecurity.com/point).  Our design uses protocol "adapters"
to handle all communications with the client and server. The adapters' job
is to do the parsing into a form that the rule engine could handle. You can
program the rule engine to make decisions about what to do with the message
based on any part of the parsed content.

For arbitrary content, I'd recommend some sort of chaining (like J2EE
filters) so that you can stack a series of parsers together for a particular
client-server protocol.  For example, you might want a Blowfish decryptor,
an unzipper, a Java object deserializer, and and XML parser for a particular
stream.

Now it would be very cool if the parser engine could automatically recognize
what step to do next. Kind of like how Winzip recognizes tar files and asks
you if you want to untar them.  In PointGuard, we ordered the parsers, and
built them so they could quickly tell whether they were appropriate for the
data. The first match gets used. But we haven't chained them yet.

I like your ideas about custom editors. You could start with raw, xml, and
binary editors and get more sophisticated when you need it.  One problem to
wrestle with is if you want to change information in a stream at two
different layers of the protocol. Like if you want to change headers *and*
the file content itself in an HTTP file upload.

In PointGuard, we handled this by parsing into a big data structure and
allowing references into it from the rule language.  So you might say...if
(message.header.referer = 'abc') and (message.content.file.data = '12') then
reject( "abc referer can't have 12 in data field);

Maybe something like this would work in WebScarab if you could hook together
all the custom editors into a big tree.  Did you ever see ResEdit on the old
Mac OS?  Something like that with custom editors for 'known' formats, and
default string and byte editors for unknown ones.

--Jeff

Jeff Williams
Aspect Security
http://www.aspectsecurity.com



----- Original Message ----- 
From: Dawes, Rogan (ZA - Johannesburg)
To: 'Olivier G. Gaumond' ; webappsec () securityfocus com
Sent: Monday, October 13, 2003 4:19 AM
Subject: Handling of arbitrary content in analysis tools (WebScarab) Was RE:
SOAP protocol analyzer


Unfortunately I do not know of a tool that allows intercepting and analysing
SOAP packets.

However, I AM interested in including support for fairly arbitrary content
in WebScarab. I would like ideas on how best to implement this, though,
since I am not an experienced Java programmer.

For example, a client of mine is sending twofish encrypted serialised
objects across an HTTP connection, and I would like to be able to support
intercepting, interpreting, modifying, and releasing content of this nature.

Obviously, it is also important to be able to report on (summarise?) what
has been seen and what has been sent.

Does anyone have any suggestions on how to handle fairly arbitrary data
formats in a clean extensible way in Java?

Complications that I am seeing (over and above what WebScarab does
currently) are:

* arbitrary data formats, such as the twofish encrypted objects mentioned
above, XML (SOAP?) and whatever else developers come up with.
* Multi-part content, which could have "variable=value" pairs, as well as
instances of the arbitrary formats mentioned above.

Scenarios where it would be used/needed:

* WebScarab intercepts a request or response, and allows the operator to
modify it
* WebScarab records the conversation, and the operator wants to review it.
* The operator wishes to create such a request manually
* WebScarab wants to supply "known bad" data (fuzz) to see how the web
application responds.

I am leaning towards the following approach:

Have a operator-configurable list of "ContentParsers", which are essentially
java classes that implement the ContentParser interface, mainly the
following method:

Object parseContent(byte[] content) throws ParseException;

The operator would be able to select from the list of parsers, and ask it to
parse the message body (entity-body, content, whatever ;-) to an object. If
it succeeds (does not throw an Exception), the Object could be passed to an
appropriate editor.

Example parsers would be:

org.owasp.webscarab.parser.MultiPartFormDataParser returning a
MultiPartFormData object
org.owasp.webscarab.parser.WWWFormUrlEncodedParser returning a
WWWFormURLEncoded object

for the two default FORM submission methods
(application/x-www-form-urlencoded and multipart/form-data).

But it would be entirely possible for someone to put a class such as

za.org.dragon.exodus.parser.TwoFishSerialisedObjectParser returning Object

in the list, (and the CLASSPATH) and have that appear as an option in a
ComboBox, or whatever. The operator could then ask WebScarab to attempt to
parse the content using that particular parser, and return an Object of an
appropriate type.

Once we have an Object representation of the content, we can look at an
operator-configurable list of ObjectEditors (analogous to
javax.swing.CellEditor) which would allow us to visualise(?) and modify the
Object. The tool could use reflection to identify the Object's class
hierarchy and any interfaces that the Object implements, and then identify
the Editors that are able to edit objects of that type.

An illustration might be the MultiPartFormDataEditor that shows the list of
parts, allowing the operator to modify, insert or delete parts. The
MultiPartFormDataEditor would then use Class-specific editors for each part,
such as a FilePartEditor (load a different file), NameValuePairEditor,
javaUtilVectorEditor, etc.

Does this seem like a reasonable way of doing things?

Rogan

> -----Original Message-----
> From: Olivier G. Gaumond [mailto:olig () monimap com]
> Sent: 10 October 2003 03:21 AM
> To: webappsec () securityfocus com
> Subject: SOAP protocol analyzer
>
>
> Do someone know a tool who can dissect web services protocols
> like SOAP.
> What I am looking for is a tool that decomposes the traffic
> sniffed on the
> network.  I want to know which method was called, the
> parameters, etc.  Like
> Ethereal does with many other protocols.
>
> The tool can also be proxy based and if it can analyze .NET
> Remoting also it
> would be a bonus.
>
> Thanks.
>
> Olivier
>
>
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.516 / Virus Database: 313 - Release Date: 2003-09-01

Important Notice: This email is subject to important restrictions,
qualifications and disclaimers ("the Disclaimer") that must be accessed and
read by clicking here or by copying and pasting the following address into
your Internet browser's address bar: http://www.Deloitte.co.za/Disc.htm. The
Disclaimer is deemed to form part of the content of this email in terms of
Section 11 of the Electronic Communications and Transactions Act, 25 of
2002. If you cannot access the Disclaimer, please obtain a copy thereof from
us by sending an email to ClientServiceCentre () Deloitte co za.