Re: one of my servers has been compromized

[Tim <tim-security () sentinelchicken org>]

For future reference, and for the benefit of people searching for
solutions to similar problems: You've made the most common rookie
mistake.  You have already trashed potentially critical information
about the attack by trying to clean up the server first.  Don't do
that.


> I've run the "find" commands and found a number of file with the first 
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl 
> processes that where trying to connect to port 6959 (only trying because 
> I've now set up iptables so that they actually can't), but those processes 
> kept spawning. Checked crontab of www-data, found the launcher, removed it.


Instead, your first step should be trying to preserve the evidence.
You should obtain forensic images of the disk and if possible,
physical memory of the host before taking any corrective action on the
host itself.  

If you don't have budget to bring in a professional to do the
investigation, then capturing memory is probably not practical (it is
easy to do it wrong and trash useful information on disk).  So in that
case, shut the server down and use a boot disk to obtain a DD image of
your drive(s).  If your server has any useful interactions with other
hosts, such as relying on an LDAP server for authentication or sending
logs to a syslog server, then capture all of the logs you can from
those hosts as well.  (Simple copying of those files is sufficient,
unless you have reason to believe they are compromised as well.)

If you really need the server back in production ASAP, you can build a
new copy of it in parallel with these preservation activities.  I
definitely recommend against trying to "clean" a compromised server.
You might say: "gosh, then I'll need more hardware to do that, and
hardware is expensive!"  No, hardware is cheap.  Allowing an attacker
continued access to your infrastructure is much more expensive.

Ok, that's the "what".  Now the "why":

By doing what you did (deleting "evil" files, running a find across
the filesystem), you may have lost the back door files that tell you
exactly how the malware works and also altered
created/modified/accessed dates of the malicious files themselves.
Those dates, while not always reliable, can be very useful in
determining the timeframe of the attack.  In addition, each time you
do anything on the host, more data is written to disk (logs,
.bash_history, you name it) and that new information could be
overwriting previously deleted files that could be useful to the
investigation.  

Suffice it to say, you've probably already trashed a wealth of
information that would have helped identify how the attacker got in.
Enough information may still be available to determine what the
vulnerability is that they exploited, but you've certainly made it a
lot harder to isolate the event.

tim

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/