Re: automatic signature generation

["Eric Hacker" <focus () erichacker com>]

On 5/20/07, Sanjay R <2sanjayr () gmail com> wrote:
> Hi List:
> There have been few studies to propose the automatic generation for
> misuse based IDS, like snort (in fact, it is the hot area of research
> among IDS researchers). Suddenly, it came into my mind, whether is it
> feasible to generate (Good) signatures for all types of attack in an
> automatic way (in a black-box environment, where we don't have the
> source-code of the vulnerable application)?

Sanjay, please define feasible.

Let's look just at network IDS. There is no doubt that one can take a
packet or session and create a signature to match. The question is
will it match other packets exploiting that vulnerability or match
other packets not. Thus what is a feasible rate of false positives
(F+) and false negatives (F-). The problem is that depending on
circumstances, acceptable F+ and F- are variable.

For instance, immediately after a worm exploiting an unpatchable zero
day vulnerability is released, one might have a fairly high tolerance
for false positives in detection mode and find false negatives
unacceptable. "Better safe than sorry" so to speak. After the risk has
died down somewhat, then F+ becomes annoying and hampers the
effectiveness of the analysts. F- might be unacceptable with a high
risk, and more tolerable when the risk goes down, especially if a
victim makes a whole lot of noise anyway.

> The main cause of my doubt is the observation that it is not
> feasible to generate attacks automatically. Usually, an attacker spend
> hours to analyze the application and then write an exploit.

Just because you haven't seen it done doesn't mean it can't be done
(or isn't being done). Look at the evolution of reverse engineering
patches and creating exploits in recent years. Automated generation of
exploits is a matter of resource allocation. If this was a wartime
need, it would be done already. It will happen; we should be thankful
it hasn't yet.

> We don't
> have any tool that take, as an input,  the application to be
> exploited, and gives us an working exploit (of course, Metasploit
> helps us to create exploit). Therefore, the early thought that comes
> into my mind is "creating an automated signature generation tool is as
> difficult as creating an automated attack generation tool". I would
> like to know your opinion on this.

The main problem I see is that we don't have widespread tools to
automate the test/QA process for verifying that the generated
signature is feasible. Regardless of what your definition of feasible
is, if testing the signature is manual (or only live production) then
one can hardly say that the process is automated.

Where is the "make test" for IDS signatures?

There are necessary pieces, but I don't think that there is anything
available to tie them together except unique shell scripts for each
one. There is tcpreplay for playing captures, but openpacket.org is
still waiting to be born. One can attach files to signatures at
doc.bleedingthreats.net but that doesn't help much for normal network
traffic.

Sometimes a packet replay isn't good enough in any case. One really
needs real time coordinated activities between the attacker, victim
and IDS is order to support automated testing of a signature. The
process is:
 Launch attack
 Get IDS response.
 Get victim effect.
 Does response match effect?
 Lather, rinse, repeat.....

Packet captures can sometimes represent an known victim effect, but
sometimes they can't. And unless our friendly list advertiser (Core)
has updated their testing capabilities with real time log coordination
recently, then I don't think a "feasible" automated IDS testing tool
is available commercially or open.

However, I am getting close to making one available. I've got
proposals in the works for end of summer security conferences. Even if
my system for IDS signature QA doesn't really meet the need, it will
be open source and easy for others to learn from my mistakes. And I'm
probably wrong about there being other tools too; I'm not perfect.

So, Sanjay, if by the end of the summer you have implemented a cool
algorithm for automated IDS signature generation, I'll set you up with
a "make test" script that will automatically coordinate the attacker,
IDS and victim to see if it passes.

The only thing really in the way is not believing it can be done.

Peace,
--
Eric Hacker, CISSP

aptronym (AP-troh-NIM) noun
A name that is especially suited to the profession of its owner

I _can_ leave well enough alone, but my criteria for well enough is
pretty darn high.

------------------------------------------------------------------------
Test Your IDS

Is your IDS deployed correctly?
Find out quickly and easily by testing it 
with real-world attacks from CORE IMPACT.
Go to http://www.coresecurity.com/index.php5?module=Form&action=impact&campaign=intro_sfw 
to learn more.
------------------------------------------------------------------------