Re: one of my servers has been compromized

[Paul Schmehl <pschmehl_lists () tx rr com>]

Don't be silly.  You can run static binaries off a thumb drive without 
taking the system down.  And that includes md5sum.  You can put everything, 
including the script, on a thumb drive and be perfectly comfortable that 
the results are reliable, because statically compiled binaries don't use 
system libraries.  As a quick check to see if system binaries have been 
altered, it's hard to beat.

And I wasn't responding to the OP.  I already did that the other day, when 
I told him he had a web app that got hacked and the scripts were run with 
perl, so noexec, nosuid for /tmp wouldn't have helped.

I will agree with you that we don't typically see the old style hacks any 
more, but some hackers are still installing trojaned sshd binaries. 
Nowadays it's mostly app hacks and primarily (as it's always been) to 
Windows boxes.

--On December 6, 2011 10:49:14 AM -0800 Gage Bystrom 
<themadichib0d () gmail com> wrote:

> But the problem with that is it is a mentality roughly a little more then
> a decade old. What you described is a userland rootkit detector. Problem
> is no one uses userland rootkits anymore! Sure there was some recent
> development in managed code rootkits but it really hasn't home anywhere
> and is Windows centric. Not to mention your plan is totally flawed. You
> assume md5sum is safe to begin with. Meaning that to be remotely safe
> with this you have to run the script for a livecd. Meaning you have to
> bring down the server everytime you suspect you MAY have been
> compromised. Completely unacceptable for anyone other than a home user.
> The only way to circumvent such issues is to recreate tripwire, in which
> you still have the same fundemental problems tripwire has always has.
>
> I know ya mean well, but your first block of advice isn't pratical or
> effective. The second one the OP already did so alls well for that.
>
> :)
> On Dec 6, 2011 10:19 AM, "Paul Schmehl" <pschmehl_lists () tx rr com> wrote:
>
> A "poor man's" root kit detector is to take md5sums of critical system
> binaries (you'd have to redo these after patching), and keep the list on
> an inaccessible media (such as a thumb drive).  If you think the system
> is compromised, run md5sum against those files, and you will quickly
> know. You could even keep statically compiled copies on the thumb drive
> to use in an investigation.
>
> Start with things you use to check for problems; ls, ps, fstat, sockstat,
> netstat, wtmp, nc, sshd, etc.
>
> It would be fairly trivial to create a simple shell script that would
> compare the md5sums of system binaries to the saved copies and flag
> anomalies.
>
> And, of course, if you can take a system offline, there are a number of
> bootable security distros that allow you to do extensive analysis of
> systems.
>
> <http://www.darknet.org.uk/2006/03/10-best-security-live-cd-distros-pen-t
> est-forensics-recovery/>
>
> In general, on Unix systems, look for oddly named directories in odd
> places (like /tmp, /dev, etc. and review logs that have been syslogged
> elsewhere for telltale signs of compromise.
>
> It's surprising how few times the shell history logs get wiped, but there
> are some kits out there that do that for you.  Web apps and improper
> permissions (world writeable) are the two most frequent causes of
> compromises that I've seen on Unix systems.
>
> --On December 5, 2011 1:53:21 PM +0000 Dan Ballance
> <tzewang.dorje () gmail com> wrote:
>
>
> Thanks for the heads-up on rkhunter Gage.
>
>
> Is there anything else out there atm that works as a reasonable root kit
> detector or is such a thing considered impossible now? I realise a
> skilled attack will be able to bury itself without a trace, but I'm
> thinking of something that can be used in less skilled breaches such as
> the one thought to have been identified in this thread. Sometimes
> something imperfect is still better than nothing I think.
>
>
> Also, am I correct to think that using something like tripwire is the
> best way to detect root kits properly, but that it obviously needs
> installing when the box is fresh and before it has been physically
> connected to a network?
>
>
> thanks to everyone for their valuable contributions here - much
> appreciated!
>
>
> dan :)
>
>
>
> On 5 December 2011 11:13, Gage Bystrom <themadichib0d () gmail com> wrote:
>
>
> If it was a rootkit then trying to run the outdated rkhunter would be a
> moot point. Whatever seizes the kernel first wins, hands down.
>
> Fortunately for him, since the bot was so easy to find in the first place
> and such a simple way of maintaining it, the box was clearly seized by
> someone who didn't give a rats ass about it. Probably a skiddie or an
> automated attack to begin with.
>
> As for plugging any security holes, check your httpd error logs. If you
> noted down the time of the bot files creation date you would look around
> the same time for suspicious log entries. If they were as careless in
> scrubbing the logs as they were holding the box it would give you a look
> into how it may have been compromised. If you're getting things like
> ../.../../../../etc/passwd then some sort of lfi vuln was likely
> exploited, start grepping your php files for stuff like include(), or if
> you're getting something like into outfile then check your mysql user
> permissions and don't let it have file perms, and then start grepping
> down for sql vulns.
>
> If it comes down to being too much of a hassle to get all the obvious
> vulns at least then go to your boss, admit there is an issue and that
> time needs to be taken to remove such legacy code as this could have been
> a far worse incident if it had been more targetted and the end goal
> wasn't a botnet.
>
>
> On Dec 5, 2011 3:02 AM, "Dan Ballance" <tzewang.dorje () gmail com> wrote:
>
> I'm no expert, but here's something to get you started while you wait
> for more experienced replies. Check for root kits:
>
>
> sudo apt-get install rkhunter
> sudo rkhunter --update
> sudo rkhunter --check
>
> On 5 December 2011 10:44, Lucio Crusca <lucio () sulweb org> wrote:
>
> Hello *,
>
> I'm not new here, but I've mostly lurked all the time through gmane. I
> never
> believed it could happen to me until it actually happened: they
> compromized
> one of my servers. It's a Ubuntu 10.04 server with all security patches
> regularly applied. I'm inclined to believe they used some hole in the web
> application, which is a old customized Virtuemart version (1.1.3), which
> is
> not upgradable because of the invasive code customizations (I'm not the
> author of that code, so I have no clue about what had been changed back
> then).
>
> Now the problem for me is to track down the security hole. Here is the
> email
> my provider received and forwarded to me:
>
>
> Subject: ISP Report; botnet activity on irc.undernet.org
> [...]
>
> Hello, I am an operator on the irc chat network,
> irc.undernet.org and i would like you to investigate the
> owner of the Ip addresses that are listed at the foot of this
> email.
>
> This/These host(s) have likely been compromised, and had an
> altered/rogue process installed on it, and was part of a botnet
> that was found on our network.
>
> The exploit or compromise running on this system is likely
> to be an irc bot. Can you please alert the person who is
> responsible, for its security to patch/upgrade, remove the
> irc process and secure their system.
>
> = Unix System owners =
> A favourite place for hiding the bot(s) is in tmp
> and in /var/tmp/ or /dev/shm/ or in a users home directory
> sometimes it may be hidden like /tmp/".  ."/ or similar.
>
> The bot files can usually be found by running these one line
> commands as the root user.
>
> find / -exec grep -l "undernet" {} +
> find / -exec grep -l "sybnc" {} +
> find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
> find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
>
> netstat -tanp
> lsof -i tcp:<Port number>
>
> *netstat looking for connections to remote port 6667 or the
> range of ports between 6660-7000 once you find the port you
> can use the command, lsof -i tcp:portnumber to determine
> which process/user it is running under, and terminate it.
>
> = Windows System Owners =
> most windows bots are mIRC scripted bots and generally
> need a file called mirc.ini to run, you should search for
> this file. Run a good antivirus scanner and firewall.
>
> This Ip/host may be removed from our Irc network due to the
> risks it presents to our users.
>
> Should you need any help with removing the files or bot
> process, feel free to contact me by mail or on our network,
> which you connect to using any irc client and issuing
> /server irc.undernet.org
>
> I look forward to your reply
> Scot
>
> * Affected host/IPs, capture time is GMT+1: United kingdom
> and servers they were connected to.
>
> Please note: when resolving server names to IP Addresses
> that all our servers end with .undernet.org (for example)
> Tampa.FL.US. is actually  Tampa.FL.US.undernet.org
>
> Important: If you reply to this mail needing further
> information, please leave this mail intact, or supply us
> with the IP Address(es) in question, as we reference these
> mails by the unique IP Address
>
> Time of Capture: DECEMBER 3, 2011 10:03:48 PM
>
> List of IP address(es) and server it connected to:
> my.server.ip.address (CHICAGO.IL.US
>
> BUDAPEST.HU.EU
>
> MONTREAL.QC.CA.undernet.org)
>
>
>
> I've run the "find" commands and found a number of file with the first
> "find", under /tmp/.m
>
> Deleted them, looked up remote connections with netstat, killed perl
> processes that where trying to connect to port 6959 (only trying because
> I've now set up iptables so that they actually can't), but those processes
> kept spawning. Checked crontab of www-data, found the launcher, removed
> it.
>
> Now the problem is: how do I pervent further abuse? What should I search
> in
> the logs (if anything) to spot the security hole?
>
> TIA
> Lucio.
>
>
>
>
>
> _______________________________________________
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>
>
>
> _______________________________________________
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/
>
>
> _______________________________________________
> Full-Disclosure - We believe in it.
> Charter: http://lists.grok.org.uk/full-disclosure-charter.html
> Hosted and sponsored by Secunia - http://secunia.com/



-- 
Paul Schmehl, Senior Infosec Analyst
As if it wasn't already obvious, my opinions
are my own and not those of my employer.
*******************************************
"It is as useless to argue with those who have
renounced the use of reason as to administer
medication to the dead." Thomas Jefferson
"There are some ideas so wrong that only a very
intelligent person could believe in them." George Orwell

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/