/etc/inittab - The Most Overlooked Cracker Haven

[InfoSec News <isn () c4i org>]

+------------------------------------------------------------------+
|  Linux Security: Tips, Tricks, and Hackery                       |
|  Published by Onsight, Inc.                                      |
|                                                                  |
|  05-December-2002                                                |
|  http://www.hackinglinuxexposed.com/articles/20021205.html       |
+------------------------------------------------------------------+

This issue sponsored by: Hacking Linux Exposed, Second Edition.

Hacking Linux Exposed, Second Edition will be hitting the stores any
day now. This new version of the best-selling first edition includes
almost 200 new pages of hacking and cracking materials. Also, if you
order online through our web page (via Amazon or Barnes and Noble),
we'll donate any commissions to the Electronic Frontier Foundation.

For more information, visit http://www.hackinglinuxexposed.com/

--------------------------------------------------------------------

/etc/inittab - The Most Overlooked Cracker Haven
By Brian Hatch

Summary: Crackers can cause their software to be run by adding
entries to /etc/inittab, a file frequently missed by administrators.

Last week I provided a challenge, which can be summarized as follows:

  * A cracker broke into a server,
  * Said cracker installed various back doors,
  * The administrator removed everything he could find, but,
  * Each time the machine was booted, even in single user mode,
    mysterious processes ran from /tmp/.qw,
  * The /tmp directory was created clean from scratch at every
    bootup, so the files could not be stored there and usable between
    reboots,
  * All xinetd and /etc/rc?.d services were cleaned out and nothing
    therein could have started these mystery processes, or start
    processes that started those processes, etc,
  * No LKMs or kernel modifications were used to start these
    processes[1],

Many[2] people wrote in with ideas which fell into the following
categories:

  * changes to the master boot record
  * changes to /etc/inittab
  * changes to bootloader (lilo/grub) command-line arguments
  * changes to initrd ram disks
  * changes to /root/.bashrc, /etc/profile, and such.

I told folks to discount any change which required kernel
modifications. A change to the MBR probably doesn't qualify in that
regard, but it would certainly seem out of scope nonetheless.[3]
Similarly, a change to the bootloader or initrd ram disk[4] fall
under kernel changes to some degree, and thus weren't the correct
answer, based on my phrasing.

Many folks suggested that it was root's .bashrc, or global /etc/
profile and similar files that was the culprit. By logging in, root's
shell would execute commands that were hidden in those files. That's
definately a possibility, but the fact that moniker had PID 15 meant
that it had to have started before root logged in, so that's not the
culprit.

In fact, the problem on this compromised box was caused by cracker
changes to /etc/inittab. To explain, let's recap some of the
evidence. The processes were visible via ps as:

  # ps -ef | grep qw
  UID      PID  PPID  C STIME TTY          TIME CMD
  root     15    1    0 11:24 ?        00:00:00 /tmp/.qw/moniker
  root     242   15   0 11:25 ?        00:00:00 /tmp/.qw/cr8
  root     244   1    0 11:25 ?        00:00:00 /tmp/.qw/rucc

Several folks said that it is quite obvious that 'monkier' and 'rucc'
were started from init since their parent is process #1. (Init always
has process ID #1). Unfortunately, that's faulty logic. Any
daemonized process will be inherited by init when as seen in this "ps
-ef" snippet:

$ ps -ef | grep '  1  '
  UID        PID  PPID  C STIME TTY          TIME CMD
  root         1     0  0 Jan23 ?        00:00:19 init [2]
  root         2     1  0 Jan23 ?        00:00:00 [powerd]
  root         3     1  0 Jan23 ?        00:00:00 [keventd]
  root        10     1  0 Jan23 ?        00:00:00 [scsi_eh_0]
  root        11     1  0 Jan23 ?        00:00:00 [khubd]
  root        88     1  0 Jan23 ?        00:00:04 perl /etc/swatch/swatch
  root       483     1  0 Jan23 ?        00:00:00 /sbin/klogd
  root       574     1  0 Jan23 ?        00:02:23 /usr/lib/postfix/master
  daemon     612     1  0 Jan23 ?        00:00:00 /usr/sbin/atd
  root       619     1  0 Jan23 ?        00:00:13 /usr/sbin/cron
  root       632     1  0 Jan23 ttyS0    00:00:00 /sbin/getty 9600 ttyS0
  root       859     1  0 Jan23 ?        00:02:15 /sbin/syslogd
  root       893     1  0 Jan23 ?        00:28:28 /usr/sbin/sshd

These processes also have a PPID of 1, but most were started by
standard /etc/rc?.d scripts. The fact that our cracker's processes
had init as the parent doesn't prove they were started by init.
However the 'monker' program has a PID of 15 - a very very low number
for processes. Assuming it was started as the 15th process (as
opposed to being several thousand process IDs later when the PIDs
wrapped around) then it must be extreemly early in the boot process,
before many rc?.d scripts run, and likely is started from init.

The administrator had re-installed all his software to be safe. By
design, these re-installs did not wipe out his configuration files,
so though he reinstalled init, it did not wipe /etc/inittab clean.

He scanned the machine for any files that had been touched since the
breach, but this is not reliable. A cracker can easily change the
modified-date timestamp on a file using the 'touch' command. Doing so
will change the inode change time, however:

  $ cd /tmp
  $ ls -l somefile; ls -cl somefile
  -rw-------    1 bri      bri         20476 Sep 17 14:49 somefile
  -rw-------    1 bri      bri         20476 Sep 17 14:49 somefile
  
  $ date
  Wed Dec  4 13:31:37 PST 2002

  $ echo "blah" >> somefile

  $ ls -l somefile; ls -cl somefile
  -rw-------    1 bri      bri         20481 Dec 04 12:00 somefile
  -rw-------    1 bri      bri         20481 Dec 04 12:00 somefile


  $ touch 09201419 somefile
  $ ls -l somefile; ls -lc somefile
  -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
  -rw-------    1 bri      bri         20481 Dec  4 13:32 somefile

Note that after our 'touch' command, the file modification time
(mtime) was set to the original Sept 17th date. However the inode
change time (ctime) was not. The ctime is always updated whenever the
inode has any changes made, such as file modifications, chmod
permissions changes, etc. The administrator was smart by looking for
files with mtime or ctime changes during the window where the cracker
had access.

If the cracker modified /etc/inittab, even though he could reset the
mtime with touch, he shouldn't be able to change the ctime back. Thus
you'd think that his find process would have caught the /etc/inittab
changes. Unfortunately, the cracker can still modify the ctime of a
file simply by changing the system clock:

  $ date 09201419 
  $ touch 09201419 somefile
  $ date 12041200 
  $ ls -l somefile; ls -lc somefile
  -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
  -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile

So, faking both the mtime and ctime is trivial, and this is why he
missed the changes to /etc/inittab - his find command didn't indicate
any change, so he didn't examine the file at all.

So, what were those changes? Here's a snippet of the file

  # The default runlevel.
  id:2:initdefault:
  
  # Boot-time system configuration/initialization script.
  # This is run first except when booting in emergency (-b) mode.
  si::sysinit:/etc/init.d/rcS
  
  # What to do in single-user mode.
  ~~:S:wait:/sbin/sulogin
  C1:S:boot:/dev/st0z

  # /etc/init.d executes the S and K scripts upon change
  # of runlevel.
  #
  l0:0:wait:/etc/init.d/rc 0
  l1:1:wait:/etc/init.d/rc 1
  l2:2:wait:/etc/init.d/rc 2
  l3:3:wait:/etc/init.d/rc 3
  l4:4:wait:/etc/init.d/rc 4
  ...

The line that begins with C1 above was the new entry. It tells init
to run the program /dev/st0z at boot time, regardless of runlevel.
This entry came before the /etc/init.d scripts (the l# entries) which
is how it was able to get such a low process id.

The /dev/st0z file was the cracker's program. Sticking the file in /
dev/ made it less likely the administrator would find it, since many
admins don't understand what all the /dev devices do. This file was a
simple shell script:

  # head /dev/st0z
  #!/bin/sh
  cat <<'EOF' > /tmp/st0z.uu
  begin 700 st0z
  M?T5,1@$"`0`````````````"``(````!``%-$````#0``SSX```````T`"``
  M`````@`"=W``!'=P`````````+@`````````!P`````O=7-R+VQI8B]L9"YS
  M.````4P`````````J@```"L```!Q````/0```6(```#2```!2`````````$1
  ...
  M````````````````````F0```+H``````````````/8`````````]````7$`
  end
  EOF
  cd /tmp
  uudecode st0z.uu
  tar xzf st0z
  rm  st0z.uu st0z
  exec /tmp/.qw/moniker

This script contains a uuencoded file within it, which it puts in /
tmp. This uuencoded file is just a compressed tarball, so it
uudecodes and untars it. It then deletes the temporary files, and
then runs the moniker program (another shell script). The actual
script had more comments at the top to make it look more like a
system program, and obfuscated the commands a bit, but you get the
idea.

Monkier was responsible for running the cr8 program -- obvious from
ps output, since moniker is cr8's parent -- and for running rucc,
which is not obvious from ps, but obvious if you read monkier's
source.[6]

This st0z script also explains why the files in /tmp/.qw were always
coming back, even though they weren't there between reboots. The
files were copies out of the /dev/st0z file each time, so wiping /tmp
had no effect.

So what were the problems this administrator faced when trying to
track down and remove the mysterious cracker code?

  * He blindly trusted timestamps on files, thus missing /etc/
    inittab.
  * Had no file integrity tools on the system, thus missing inittab
    and /dev/st0z.
  * He didn't understand the Linux boot process enough to know that
    programs can be launched by init directly.

Of course, you may argue having a week root password was another
problem. There were a few other backdoors found later, and eventually
they decided to wipe the machine clean and start from scratch - a
very good idea given how long the machine had been out of their
control.

Thanks to the many folks who wrote in with ideas - you came up with
some things I hadn't though of either. And congratulations to Mandi
Walls, who nailed pretty much everything that the cracker did. A
beautiful copy of Hacking Linux Exposed, Second Edition will be
coming your way.

Also, thanks to Simon Walter who pointed out the 'From Power Up To
Bash Prompt' webpage at http://www.tldp.org/HOWTO/
From-PowerUp-To-Bash-Prompt-HOWTO.html. If you want to understand the
bootup process in more depth, take a look.

NOTES:

[1] This wasn't obvious based on the facts presented, but was offered
by me since I already knew the actual cause, in order to narrow the
scope of the challenge.

[2] Proof that a free book is much better incentive than a postcard -
I shall have to remember this.

[3] A change to the MBR would need to run the kernel in a way
different than the default, and really is no different than modifying
the kernel or the bootloader. Thus a theoretical MBR change would
seem to fit within the other alternatives anyway.

[4] Inirtd ram disks are typically used to help a Linux kernel load
needed modules, and thus this fits into the kernel-changes category
again.

[5] Becoming a daemon process involves code such as "fork && exit;
fork && exit" which creates a child process and kills off the parent
processes, such that they have no running parent process, and are
inherited by init.

[6] Rucc contained code to daemonize, which is why it's parent is
init.

                            -------------                            
Brian Hatch is Chief Hacker at Onsight, Inc and author of Hacking
Linux Exposed and Building Linux VPNs. He launches programs from /etc
/inittab quite regularly, or at worst through DJB's supervise. It's a
heck of a lot easier than writing your own daemon checking and
restarting code. Brian can be reached at
brian () hackinglinuxexposed com.

--------------------------------------------------------------------
This newsletter is distributed by Onsight, Inc.

The list is managed with MailMan (http://www.list.org). You can
subscribe, unsubscribe, or change your password by visiting
http://lists.onsight.com/ or by sending email to
linux_security-request () lists onsight com.

Archives of this and previous newsletters are available at
http://www.hackinglinuxexposed.com/articles/

--------------------------------------------------------------------

Copyright 2002, Brian Hatch.



-
ISN is currently hosted by Attrition.org

To unsubscribe email majordomo () attrition org with 'unsubscribe isn'
in the BODY of the mail.