Re: one of my servers has been compromized

[Ferenc Kovacs <tyra3l () gmail com>]

On Mon, Dec 5, 2011 at 11:44 AM, Lucio Crusca <lucio () sulweb org> wrote:

> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I
> never
> believed it could happen to me until it actually happened: they compromized
> one of my servers. It's a Ubuntu 10.04 server with all security patches
> regularly applied. I'm inclined to believe they used some hole in the web
> application, which is a old customized Virtuemart version (1.1.3), which is
> not upgradable because of the invasive code customizations (I'm not the
> author of that code, so I have no clue about what had been changed back
> then).
>
> Now the problem for me is to track down the security hole. Here is the
> email
> my provider received and forwarded to me:
>
> > Subject: ISP Report; botnet activity on irc.undernet.org
> > [...]
> >
> > Hello, I am an operator on the irc chat network,
> > irc.undernet.org and i would like you to investigate the
> > owner of the Ip addresses that are listed at the foot of this
> > email.
> >
> > This/These host(s) have likely been compromised, and had an
> > altered/rogue process installed on it, and was part of a botnet
> > that was found on our network.
> >
> > The exploit or compromise running on this system is likely
> > to be an irc bot. Can you please alert the person who is
> > responsible, for its security to patch/upgrade, remove the
> > irc process and secure their system.
> >
> > = Unix System owners =
> > A favourite place for hiding the bot(s) is in tmp
> > and in /var/tmp/ or /dev/shm/ or in a users home directory
> > sometimes it may be hidden like /tmp/".  ."/ or similar.
> >
> > The bot files can usually be found by running these one line
> > commands as the root user.
> >
> > find / -exec grep -l "undernet" {} +
> > find / -exec grep -l "sybnc" {} +
> > find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> > find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> >
> > netstat -tanp
> > lsof -i tcp:<Port number>
> >
> > *netstat looking for connections to remote port 6667 or the
> > range of ports between 6660-7000 once you find the port you
> > can use the command, lsof -i tcp:portnumber to determine
> > which process/user it is running under, and terminate it.
> >
> > = Windows System Owners =
> > most windows bots are mIRC scripted bots and generally
> > need a file called mirc.ini to run, you should search for
> > this file. Run a good antivirus scanner and firewall.
> >
> > This Ip/host may be removed from our Irc network due to the
> > risks it presents to our users.
> >
> > Should you need any help with removing the files or bot
> > process, feel free to contact me by mail or on our network,
> > which you connect to using any irc client and issuing
> > /server irc.undernet.org
> >
> > I look forward to your reply
> > Scot
> >
> > * Affected host/IPs, capture time is GMT+1: United kingdom
> > and servers they were connected to.
> >
> > Please note: when resolving server names to IP Addresses
> > that all our servers end with .undernet.org (for example)
> > Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
> >
> > Important: If you reply to this mail needing further
> > information, please leave this mail intact, or supply us
> > with the IP Address(es) in question, as we reference these
> > mails by the unique IP Address
> >
> > Time of Capture: DECEMBER 3, 2011 10:03:48 PM
> >
> > List of IP address(es) and server it connected to:
> > my.server.ip.address (CHICAGO.IL.US
> >
> > BUDAPEST.HU.EU
> >
> > MONTREAL.QC.CA.undernet.org)
>
> I've run the "find" commands and found a number of file with the first
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl
> processes that where trying to connect to port 6959 (only trying because
> I've now set up iptables so that they actually can't), but those processes
> kept spawning. Checked crontab of www-data, found the launcher, removed it.
>
> Now the problem is: how do I pervent further abuse? What should I search in
> the logs (if anything) to spot the security hole?
>
> TIA
> Lucio.
>
>
>
>
>
> _______________________________________________
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/

If you take security seriously, you should remove that box from the
network(or take a snapshot and wipe everything and reinstall from scratch),
and start the investigation according to your (security) incident response
plan.
In the meantime you can start restoring the services on a clean server, but
you should consider the compromised server as fully compromised, so you
shouldn't restore data from that server, until you can't guarantee without
a proof that the data is intact/genuine.
http://en.wikipedia.org/wiki/Computer_security_incident_management
Based on your area of business, you can be obligated to report the breach
to some kind of authority and co-operate with them resolving the issue.

If you have offsite backups and/or externals logs, which you can trust,
that can help you to pinpont that when did the breach happen, and what
extent did your system got compromised(worst case you can also try
comparing your system with a vanilla install of the same OS/services/in
house applications, etc).

ps: "I neverbelieved it could happen to me until it actually happened: they
compromizedone of my servers." this is a really bad attitude.

-- 
Ferenc Kovács
@Tyr43l - http://tyrael.hu

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/