RE: security advice

["Grant, Richard (KYTC)" <richard.grant () ky gov>]

Edmund I think you may need a different approach to Incident handling. We have experienced a couple compromised 
servers. With over 3,000 servers physical and virtual we are too big of a target to miss forever. First the horse is 
already out of the barn. Slamming the door shut will not change that. 

This is a time for the Incident Handling Team to be called into action. This team would develop  a plan for dealing 
with the incident. Some of the questions to be addressed include:

Who was the intruder?
When was the server compromised? How long have they had access to the machine?
How did they compromise the server?
What did they want?
Was any user data compromised? 
Finally, how can future intrusions of this nature be prevented.

The firewall logs will not answer many of these questions. So it is a time for a forensic evaluation of the server.
Active memory needs to be imaged.
A list of active processes needs to be generated.
A forensic image of the servers drive systems needs to be generated of at least the OS drive.

Now it is time to pull the plug.
So can you just remove the known infected files and go back to work?
Did you remove all of the cancer?
Did the intruders rootkit the server? CAN it be trusted in its' current installation?
Conventional wisdom dictates that the server must be rebuilt. Can it be trusted after it has been violated?
The forensic analysis will help you determine the answers to many of these questions. A timeline analysis will help 
determine what files were altered when the server was compromised. When all is said an done the Incident handling team 
will get together for a lessons learned. 
Good luck!

 
Richard G.
 

-----Original Message-----
From: listbounce () securityfocus com [mailto:listbounce () securityfocus com] On Behalf Of Edmund
Sent: Tuesday, August 24, 2010 5:17 AM
To: security-basics () securityfocus com
Subject: security advice

Hi,

Just yesterday, I found out that my company's e-mail server had been
compromised.  This fact, for some reasons, didn't seem to be a 'big
deal' to others.  I'm still stunned; but, considering how lax I had
become, it shouldn't be surprising.  *sigh*

[story mode]
Basically, the incident started out with an innocuous "there is
something wrong with sending e-mail" from a co-worker.  I looked
at the e-mail server and everything seemed to be ok, so I decided
to check the firewall.  That's when I noticed it was running
very sluggish.  "Uh oh."

I couldn't figure out which program was making it go slow.  I
thought it was the proxy, but it wasn't.  I rebooted the
firewall.  It was ok, up until a certain point and that's
when it slowed down.  I tcpdump'd one ethernet nic, and
noticed a huge amount of packets being sent to a remote
site from my e-mail server.  (Capital UH OH)

Checking out the |ps ax| I noticed a very suspicious
file "./s <ip#>".  Immediately I knew someone had
accessed the system.  I started to become a little
panicky.  I searched for the './s' file.  Then looking
up online, I found that I could go into the /proc
filesystem and find the pid and then the exe will
be shown.  Found the full path.  Looking at the
files within the folder "/var/tmp/.b", it was
confirmed.

I shouldn't have done what I did next.  I killed
the running program and deleted the folder.  :(
In hindsight, I should have killed the program
and zipped up the darn folder for analysis.
I'm still regretting that move.  *banging head
on table*

Cleaned up a few extra items and it seems normal.
I ran 'rkhunter' and filled out the necessary
warnings it found.

[story mode off]

I'm still very reprimanding myself for being
so careless. This is one lesson that I gotta
have imprinted in my thick skull.

Anyway, given this lesson,  can someone offer
any methodologies/programs that I can use to
protect the company system?   I'm now going
through the firewall rules to find out what
holes the intruder might have entered through.

Thanks.

Ed

------------------------------------------------------------------------
Securing Apache Web Server with thawte Digital Certificate
In this guide we examine the importance of Apache-SSL and who needs an SSL certificate.  We look at how SSL works, how 
it benefits your company and how your customers can tell if a site is secure. You will find out how to test, purchase, 
install and use a thawte Digital Certificate on your Apache web server. Throughout, best practices for set-up are 
highlighted to help you ensure efficient ongoing management of your encryption keys and digital certificates.

http://www.dinclinx.com/Redirect.aspx?36;4175;25;1371;0;5;946;e13b6be442f727d1
------------------------------------------------------------------------


------------------------------------------------------------------------
Securing Apache Web Server with thawte Digital Certificate
In this guide we examine the importance of Apache-SSL and who needs an SSL certificate.  We look at how SSL works, how 
it benefits your company and how your customers can tell if a site is secure. You will find out how to test, purchase, 
install and use a thawte Digital Certificate on your Apache web server. Throughout, best practices for set-up are 
highlighted to help you ensure efficient ongoing management of your encryption keys and digital certificates.

http://www.dinclinx.com/Redirect.aspx?36;4175;25;1371;0;5;946;e13b6be442f727d1
------------------------------------------------------------------------