Recovering from a penetrator, the easy way

["Harrington, Perry" <pedward () WEBCOM COM>]

Hi all,

I must confess that I recently had a home machine compromised.  It was part my
fault, and others.  It was simply a box the hosts my personal web pages and
some email.

I noticed that it was hosed when I couldn't ping it from my inside network.
It turns out that the cracker (orangehaw) had created an alias interface
for every ip on my network.  However I have a /29, and he created one for
a /24, which didn't make it much happy.  It didn't affect my ability to
ssh into the box.

First, I did a find of the drive, with -ls, to get a listing of the files.

Installed was the t0rnkit, a sniffer and rogue sshd.

He also left wuftpd-god on the box, of which I was not susceptible.  He got
into my box using an account that was vestigial from a previous machine I
migrated to that box.  It turns out he got root though old NFS utils.

The purpose of my message is to detail how I found his 'updated' programs
and how to replace them.

First, there are 2 utils in RH that are really handy: find and stat.

stat is a program that does a stat() for each file given and prints out the
information.  stat() returns 3 different times for each file, atime (last
accessed), mtime (last modified), and ctime (last changed, or creation time).

The atime and mtime are modifiable using touch or utime().  The ctime is not
changable in userspace, because it's a function of the filesystem.  It's this
field that I used to track down the files.

First, I ran 'stat' against all the important utilites, and found that these
had been replaced:

ifconfig
ls
ps
netstat
find
nscd (rogue, didn't have it installed)
inetd

I renamed all of them and downloaded known good versions from another box.

His version of ls, ps, find, and netstat were to hide his rootkit, ip addrs,
and utils, in /usr/src.

The find command is very helpful to locate rogue files, you specify a -ctime
with the number of days since the breakin, it will print out all the files
that have a ctime within that 24 hour period.  I used this to pick up all his
leftovers.

This whole message is born out of the neccessity I had to keep the machine online
and not reinstall the OS.  If you know the breakin time and catch it soon enough,
this will help you find the files.

Now, this method is not without a couple of holes.  You can modify the ctime of
a file by directly modifying the filesystem internals, this requires a remount
readonly to do, but it's possible.  Also, It might work if you reset the system
time every time you update a file, just before creation, this will have other
affects to the system, so it can be tracked.

On the latter, perhaps we really need a secure way to ensure the system time isn't
mucked with, or use something like the 'timestamp' in Sybase; it doesn't fluctuate
with the system time, it's an always incrementing timestamp.  This method is used by
programs that do journalling, perhaps the ext3 journalling extension does this too?

--Perry

PS, Orangehaw, I had that machine rate limited to 64Kbps on aDSL, why where you trying
to launch a DDoS on it?  Stachel. Grr.

--
Perry Harrington                 Director of                   zelur xuniL  ()
perry () webcom com             System Architecture               Think Blue.  /\