Re: Efficiently detecting obfuscated shell code

[Paul Robertson <proberts () patriot net>]

On Wed, 4 Feb 2004, Don Parker wrote:

> Hey guys/gals, I have been sending this question around some of the lists, and have had
> little real discussion on it. Question being; is it possible to reliably detect an
> obfuscated egg? Many of the ids signatures I have seen are a little loose, and always go
> for the nop sled with some port matching.

I think it depends on the obfuscation.  Just like with polymorphic
viruses, or indeed packed viruses, it's possible to grab the thing and
sometimes flag on the packer or the extraction engine, and sometimes on
payload characteristics, but sometimes you have to go to the exectution point.

The packer/extractor will be detectable, it's just a question of if it's
detecable without too high a false positive rate (I've been meaning to
look at those things, but real work keeps getting in the way...)

> The problem though is that it is a relatively trivial matter to sub the nop with an
> ascii character. Or someone who has a little more skill can insert another 1 byte
> function that won't affect the egg itself. These ids evasion attempts are becoming more
> widely known. With the prevalence of such programs as ADMutate and phiral.c simplifying
> the task as it were this will probably become more prevalent.

In detecting malcode, you have to go after the indcator, not the
instantiation if you want good generic detection.  "This unwraps itself"
is always a good leading indicator.

> Its not every company which has layered defences which includes application level
> firewalls, and a properly tuned ids with good signatures. This is not even taking into
> account an analyst who will recognize what they are seeing. Snort's fnord does a good
> job of detecting shell code actually, and known obfuscated variants too. Any thoughts on
> this?

To really do things right, you want to be at the point of execution.  I
wrote an interesting proof-of-concept kernel module a couple of years ago
that trapped execve calls to about 20 things, including /bin/sh for
processes who had a listening socket (or who's parents had a listening
socket) -- you could opt out a process with a signal, but kill() was also
wrappered- it was pretty good for "this stops 99% of "tunnel in/tunnel
out" trojans and shellcode in the wild (for the primary shellcode vectors
such as Apache, Sendmail...)- but there's nothing that's going to be 100%.

Obviously MAC in the kernel is the "right way" to do this, but my module
was a drop-in- I just never got the time to take it to where it really
needed to be.

I don't think you can do good detection on the wire for obfuscation
without significant slowdown and either a sandboxed execution or emulation
engine when you're talking actual executable content, but the indicators
of it probably have enough similarity that you could get pretty darned
close, but that many depend heavily on what it is you're trying to
protect.

If you run 1000 samples through an obfuscator, and can't get a reliable
detector for more than 40% of them, it's probably a lost cause.  The good
news is that most of the bad guys use the same tools, so if you can get
"today's obfuscator," it's probably good enough for prime time, even
though you might not get generic detection.

I have been meaning to go through the common shellcode libs and see if
they code match with any distribution binaries- that would give a good
starting point for on-machine protection.

Paul
-----------------------------------------------------------------------------
Paul D. Robertson      "My statements in this message are personal opinions
proberts () patriot net      which may have no basis whatsoever in fact."
probertson () trusecure com Director of Risk Assessment TruSecure Corporation
_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards