Interesting webserver intrusion (apache 1.3.31, mod_ssl 2.8.18, php 4.3.7)

["nathan c. dickerson" <nathan () pro net>]

Greetings,

A few weeks ago, a webserver at my location was broken into. I caught 
the attacker in progress before any damage was done, but the attack 
itself was quite stealthy and surprising.

At the time, we were using a version of mod_ssl which was venerable to a 
client  certificate buffer overflow(apache 1.3.29 mod ssl 2.8.16). This 
attack occurred roughly 25 days after the release of the security fixed 
version. At the time I thought it was too advanced for what the 
attackers were doing on the system. After patching the hole, and setting 
all temporary directories on the system to non executable to avoid 
binary execution from the standard places, I generally thought 
everything was fine.

We are now running Apache 1.3.31 with mod_ssl 2.8.18, which I thought 
was secure. However; a few days ago, I noticed another successful 
intrusion via Apache. Same callback scripts to get around the firewall, 
same people.

This annoyed me greatly, since they managed to execute commands via Apache.

The standard error of their exploited httpd process would show up in the 
error log. I believe the execution is done blindly.

sh: line 1: /usr/sbin/apachectl: No such file or directory
sh: line 1: cd: /var/tmp/...: No such file or directory
sh: line 1: cd: /var/tmp/...: No such file or directory
cat: /tmp/cmdtemp: No such file or directory
rm: cannot remove `/tmp/cmdtemp': No such file or directory

During the first attack, they uploaded binaries and a perl script in 
/var/tmp/.../ , but since then,  with the addition of non executable 
temporary directories, they have been simply using a perl script which 
connects back to spawn a shell on efnet IRC (uploaded in /dev/shm !) -- 
fairly clever.  The IRC callback shell leads me to believe the access is 
blind. Again, they didn't have time to get root, as a combination of 
spotting the activity quickly and running the latest kernel.

I have obtained decently accurate time ranges of intrusion from the temp 
files left around. After searching through alot of logs around those 
time ranges, the only strangeness in the logs seemed to be (ips somewhat 
hidden): 

64.110.x3x.217 - - [02/Jul/2004:12:04:17 -0700] "\x80L\x01\x03" 501 -
64.110.x3x.217 - - [02/Jul/2004:12:04:23 -0700] "\x80L\x01\x03" 501 -
216.90.2xx.3 - - [02/Jul/2004:12:20:39 -0700] "\x80L\x01\x03" 501 -
142.173.xx.6x - - [02/Jul/2004:17:33:26 -0700] "\x80L\x01\x03" 501 -
68.227.xx.2x7 - - [03/Jul/2004:08:16:38 -0700] "\x80L\x01\x03" 501 -
68.227.xx.2x7 - - [03/Jul/2004:08:16:38 -0700] "\x80L\x01\x03" 501 -
68.227.xx.2x7 - - [03/Jul/2004:08:16:38 -0700] "\x80L\x01\x03" 501 -
68.227.xx.2x7 - - [03/Jul/2004:08:16:38 -0700] "\x80L\x01\x03" 501 -
80.39.22x.6x - - [05/Jul/2004:07:26:05 -0700] "\x80L\x01\x03" 501 -
64.114.2xx.xx0 - - [05/Jul/2004:15:26:18 -0700] "\x80L\x01\x03" 501 -
80.39.22x.x1 - - [05/Jul/2004:22:21:03 -0700] "\x80L\x01\x03" 501 -
68.63.17x.2x2 - - [08/Jul/2004:07:25:58 -0700] "\x80L\x01\x03" 501 -
66.114.14x.17x - - [08/Jul/2004:11:30:55 -0700] "\x80L\x01\x03" 501 -
157.191.xxx.17 - - [08/Jul/2004:14:42:14 -0700] "\x80L\x01\x03" 501 -
38.119.13x.x2 - - [08/Jul/2004:14:42:14 -0700] "\x80L\x01\x03" 501 -
66.114.14x.1x4 - - [08/Jul/2004:17:14:50 -0700] "\x80L\x01\x03" 501 -
24.237.13x.xxx - - [08/Jul/2004:17:27:00 -0700] "\x80L\x01\x03" 501 -
217.217.xx.xx - Invalid method in request \\x80b\\x01\\x02\\xbd

This occurred several times from several different IPs at different 
times, and 30 minutes before one of the scripts was uploaded. However, 
it makes me wonder if it is a worm. I hope it is a worm, because if its 
not, they are using a non disclosed apache hole, which is disturbing.

If it is a hole in PHP, or one of the scripts on one of the 120+ sites, 
it is very difficult to detect. It also wouldn't generate anything in 
the error_logs (unless they do something silly), and the useful 
variables GET and POST get tructuated in the access log, making my job 
more difficult.

If I could get the full GET and POST request data, I could perform 
searches for interesting execution strings. Does anyone have any 
suggestions on this?

I am fairly sure the normal-user accessible php scripts on the server do 
not access exec() type functions, however it is possible since php 
scripts will write their standard error to the apache error log as well, 
and I didn't write them all, so I can not be certain they are 100% 
secure.  PHP running is 4.3.7. There are no cgi perl scripts running on 
the server.

From the scripts, I have confirmed the attackers are a team of some 
kind, and last night I paid them a visited on IRC. I was kicked from the 
channel within 5 minutes, and some civil words were exchanged which 
ended in a decent sized DOS against my ADSL, and afterwords, a brief 
conversation with one of the intruders.

I set up a acid/snort monitoring system at the router on wednesday which 
is fairly effective, and have been monitoring for further attacks. The 
attacks performed did not show up with snorts shell code detection, but 
then again, I've got to go in and tweak alot of settings still, which 
points to PHP again. Perhaps I'll look into monitoring all http requests 
on the network for occurences of the perl, sh, cat, rm and various other 
commands.

Does anyone have any suggestions or pointers that would help me narrow 
down the security hole? I have brought the system into a state where 
system access is fairly complex to achieve with blind access, but not 
impossible to the dedicated.

I can sleep well again once I find out how these guys are getting in,

Thanks,
Nathan Dickerson
Pro.NET Communications