Re: one of my servers has been compromized

[Dave <mrx () propergander org uk>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/12/2011 10:44, Lucio Crusca wrote:
> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I never 
> believed it could happen to me until it actually happened: they compromized 
> one of my servers. It's a Ubuntu 10.04 server with all security patches 
> regularly applied. I'm inclined to believe they used some hole in the web 
> application, which is a old customized Virtuemart version (1.1.3), which is 
> not upgradable because of the invasive code customizations (I'm not the 
> author of that code, so I have no clue about what had been changed back 
> then).
>
> Now the problem for me is to track down the security hole. Here is the email 
> my provider received and forwarded to me:
>
> > Subject: ISP Report; botnet activity on irc.undernet.org
> > [...]
> >  
> > Hello, I am an operator on the irc chat network, 
> > irc.undernet.org and i would like you to investigate the 
> > owner of the Ip addresses that are listed at the foot of this 
> > email.
> >  
> > This/These host(s) have likely been compromised, and had an
> > altered/rogue process installed on it, and was part of a botnet
> > that was found on our network. 
> >  
> > The exploit or compromise running on this system is likely 
> > to be an irc bot. Can you please alert the person who is 
> > responsible, for its security to patch/upgrade, remove the 
> > irc process and secure their system.
> >  
> > = Unix System owners = 
> > A favourite place for hiding the bot(s) is in tmp 
> > and in /var/tmp/ or /dev/shm/ or in a users home directory
> > sometimes it may be hidden like /tmp/".  ."/ or similar.
> >  
> > The bot files can usually be found by running these one line 
> > commands as the root user.
> >  
> > find / -exec grep -l "undernet" {} +
> > find / -exec grep -l "sybnc" {} +
> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> >  
> > netstat -tanp
> > lsof -i tcp:<Port number>
> >  
> > *netstat looking for connections to remote port 6667 or the 
> > range of ports between 6660-7000 once you find the port you 
> > can use the command, lsof -i tcp:portnumber to determine
> > which process/user it is running under, and terminate it. 
> >  
> > = Windows System Owners = 
> > most windows bots are mIRC scripted bots and generally 
> > need a file called mirc.ini to run, you should search for 
> > this file. Run a good antivirus scanner and firewall.
> >  
> > This Ip/host may be removed from our Irc network due to the
> > risks it presents to our users.
> >  
> > Should you need any help with removing the files or bot
> > process, feel free to contact me by mail or on our network,
> > which you connect to using any irc client and issuing
> > /server irc.undernet.org
> >  
> > I look forward to your reply
> > Scot
> >  
> > * Affected host/IPs, capture time is GMT+1: United kingdom
> > and servers they were connected to.
> >  
> > Please note: when resolving server names to IP Addresses
> > that all our servers end with .undernet.org (for example)
> > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org 
> >  
> > Important: If you reply to this mail needing further
> > information, please leave this mail intact, or supply us 
> > with the IP Address(es) in question, as we reference these 
> > mails by the unique IP Address
> >  
> > Time of Capture: DECEMBER 3, 2011 10:03:48 PM
> >  
> > List of IP address(es) and server it connected to:
> > my.server.ip.address (CHICAGO.IL.US
> >  
> > BUDAPEST.HU.EU
> >  
> > MONTREAL.QC.CA.undernet.org)
> >  
>
> I've run the "find" commands and found a number of file with the first 
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl 
> processes that where trying to connect to port 6959 (only trying because 
> I've now set up iptables so that they actually can't), but those processes 
> kept spawning. Checked crontab of www-data, found the launcher, removed it.
>
> Now the problem is: how do I pervent further abuse? What should I search in 
> the logs (if anything) to spot the security hole?
>
> TIA
> Lucio.

Much of what can be done to secure your server has been mentioned in the replies.

I would just like to add this:
You may want to consider running a rootkit check as a daily cron job and having the results emailed to you.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBTtz/uLIvn8UFHWSmAQL1GAgA4JWPXX1klCE6Tbi3rYuYzHnr7nU739V8
/5+1dKq/30Sq3MeKOxRWbrJRMJkEdf9Dg+61wZUq1YMvNFjX+ISmGBQzY8oay2y2
iRhzgE14YWFisZH4xEMliioC7kjRDl64+sOKahCImOmnJNChdpoQWwsheNn/gqa9
sR6g8A13ERN0Ngm9TT+9C4t4OukZGU01B9mI+9XMOY4cryKyb8jseqOrvyxYcU2d
vBdKQeUuMSo6d+069bb2y7nmojdBcdgj/wr1pJuo2wxe2QS9hqHBNOnYmy6Qiios
N4nd3wHiWIaUlwVSuC8BsT+d+G/rGGy/dh1vI3RIOoVvvJUXs5/JyQ==
=nqLC
-----END PGP SIGNATURE-----

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/