Fwd: network problem...?

["Chuck Swiger" <chuck () codefab com>]

Hi, all--

Here's the email audit trail of a denial-of-service attack from a system at  
Princeton, which occurred yesterday evening.

We're using a multihomed FreeBSD 3.4 system as our firewall, which is  
explicitly configured to block traffic to our network broadcast addresses,  
and it did it's job-- our internal machines were completely unaffected.   
Regrettably, the machines on our external network were so badly  
network-fugued out that I couldn't keep an SSH connection alive long enough  
to run snoop, so I couldn't capture packets for forensic analysis.

I'd like to do better in the future.

The upstream router managed by our ISP (AT&T) does implement a reasonable  
degree of packet filtering for our external subnet as well as blocking  
spoofed internal addresses.  However, I'm considering creating an external  
subnet containing only the external interfaces of firewalls and our ISP's  
ethernet interface, with no other machines there, so that I can achieve  
better control over traffic.  Any thoughts about this network architecture  
would be of interest to me.  :-)

However, what I'd really like to have is a firewall box which snoops all  
network traffic to a large local disk buffer in a FIFO fashion so that I can  
always look at what was going on.  Can anyone recommend a solution?

[ Of course, something based on free tools and FreeBSD or OpenBSD would be  
great, but I'm willing to spend money, too. ]

Thanks,
-Chuck


Begin forwarded message:

From: "Rita Seplowitz Saltz" <rita () Princeton EDU>
To: <chuck () codefab com>
Subject: Denial of service
Date: Thu, 17 May 2001 08:22:10 -0400
X-Priority: 3 (Normal)
X-Mailer: Microsoft Outlook 8.5, Build 4.71.2173.0
Importance: Normal

Hello, Chuck Swiger.

I am responding on behalf of this office's security team to your note of
yesterday evening, for which much thanks.

The host from which the DoS traffic issued is a shared Solaris system
with thousands of user accounts.  Thanks to your report, the compromised
account was discovered and blocked, as was the host (in a Brazilian
domain) behind the assault.  Unfortunately, early this morning, similar
traffic was observed, originating from another account on
phoenix.princeton.edu, and investigation revealed the next step back was
yet another host from the same Brazilian domain.

In addition to blocking the local accounts (and leaving voice-mail
messages for the account-holders, who will need to arrange for new, more
secure passwords), the outside domain now has been blocked entirely for
the time being, to discourage further forays from that location.

Please accept my apologies on behalf of Princeton University for this
incident--and thank you again for bringing it to our attention.

for the team:

Rita Saltz
Policy and Security Advisor
Computing and Information Technology (CIT)
Princeton University


Begin forwarded message:

X-Nextstep-Mailer: Mail 4.2mach_patches [m68k] (Enhance 2.2p3, May 2000)
Sender: "Charles W. Swiger" <chuck () codefab com>
From: "Chuck Swiger" <chuck () codefab com>
Date: Wed, 16 May 2001 21:44:10 -0400
To: abuse () princeton edu
Subject: Fwd: network problem...?
cc: chuck () codefab com

Hello--

My company may have experienced a network DoS attack from  
phoenix.Princeton.EDU (128.112.128.42).  The two messages copied below  
contain the details available at this time.

It is entirely possible that this machine had nothing to do with the  
incident...however, I would be grateful if you would investigate.

Thanks,

-Chuck Swiger; Network Operations Manager @ CodeFab


Begin forwarded message:

X-Nextstep-Mailer: Mail 4.2mach_patches [m68k] (Enhance 2.2p3, May 2000)
Sender: "Charles W. Swiger" <chuck () codefab com>
From: "Chuck Swiger" <chuck () codefab com>
Date: Wed, 16 May 2001 21:22:04 -0400
To: all () codefab com
Subject: Fwd: network problem...?

Hi, all--

We had a network incident Wednesday night involving a denial-of-service  
condition for our T1 link and most of the machines on the external subnet.

Basicly, we were seeing 95+ % packet loss for over an hour, and most of the  
machines on the external subnet were unresponsive or very slow due to the  
external subnet being flooded with traffic.  Our firewall performed its job  
properly, so the internal subnet was not affected.

However, external email and our mailing lists were non-functional during  
this problem.  Also, iota.codefab.com was completely frozen-- almost as if it  
had been reset to the OpenFirmware prompt, since even the blinking  
"heartbeat" LED was off-- and not pingable from 19:44 to 20:21 PM (thanks,  
Big Brother!)

There was an odd syslog message during the interval we were experiencing problems:

May 16 19:30:04 iota netmsgserver[365]: datagram_main.netipc_receive  
data_size invalid, data_size = 0.
May 16 19:30:35 iota netmsgserver[365]: srr_main.netipc_receive invalid  
data_size 0 from host 128.112.128.42
May 16 19:32:18 iota netmsgserver[365]: datagram_main.netipc_receive  
data_size invalid, data_size = 0.
May 16 19:37:49 iota netmsgserver[365]: srr_main.netipc_receive invalid  
data_size 0 from host 128.112.128.42
May 16 19:39:49 iota last message repeated 1 time
May 16 20:20:40 iota unix: envctrltwo0: ignoring debug enter sequence

...and this may or may not have been coincidental.

[ The 'envctrltwo0' message for iota was when I did enter a serial-console  
'break' sequence, which unfroze that machine. ]

Does anyone recognize this machine, phoenix.Princeton.EDU (128.112.128.42)?   
I've changed our firewall to explicitly filter out all traffic from that IP,  
just in case this was somebody trying a smurf/teardrop-like network  
amplification DoS attack or whatever.  I'll also get in touch with the  
network admins at Princeton to investigate that machine.

-Chuck Swiger

        =====


Begin forwarded message:

X-Nextstep-Mailer: Mail 4.2mach_patches [m68k] (Enhance 2.2p3, May 2000)
From: "Chuck Swiger" <chuck () codefab com>
To: awmis <RM-awmis () ems att com>
cc: sysadmin_list () codefab com
Subject: [SysAdmin_List] network problem...?
Sender: sysadmin_list-admin () shot codefab com
Date: Wed, 16 May 2001 20:14:23 -0400

Hi--

For some reason, our Cisco router decided to generate a whole lot of traffic  
both across the T1 and into our external subnet.  This traffic was not being  
generated by local machines-- the AT&T router ethernet interface goes into  
an 8-port hub (which is connected to our firewall and so forth), so it was  
easy to isolate it from any other machine.

Here's what we were seeing trying to ping to your access router,  
attcodefab.customer.ip.att.net (12.124.107.57):

pi# ping 12.124.107.57
PING 12.124.107.57 (12.124.107.57): 56 data bytes
64 bytes from 12.124.107.57: icmp_seq=24 ttl=254 time=2939.213 ms
64 bytes from 12.124.107.57: icmp_seq=36 ttl=254 time=2848.182 ms
64 bytes from 12.124.107.57: icmp_seq=68 ttl=254 time=5.451 ms
64 bytes from 12.124.107.57: icmp_seq=69 ttl=254 time=1955.235 ms
64 bytes from 12.124.107.57: icmp_seq=94 ttl=254 time=3803.131 ms
64 bytes from 12.124.107.57: icmp_seq=112 ttl=254 time=2920.443 ms
^C
--- 12.124.107.57 ping statistics ---
134 packets transmitted, 6 packets received, 95% packet loss
round-trip min/avg/max/stddev = 5.451/2411.942/3803.131/1201.532 ms

And here are some traceroutes which failed into "host unreachables":

pi# traceroute mail-in.apple.com
traceroute to mail-in.apple.com (17.254.0.58), 30 hops max, 40 byte packets
 1  att-gw (12.38.161.129)  1.790 ms  1.469 ms  1.568 ms
 2  * * *
 3  * * *
 4  * * *
 5  att-gw (12.38.161.129)  1.574 ms !H *  1.498 ms !H
^C

pi# traceroute -n www.apple.com
traceroute to www.apple.com (17.254.0.91), 30 hops max, 40 byte packets
 1  12.38.161.129  1.547 ms  1.491 ms  2.369 ms
 2  * 12.38.161.129  3.638 ms !H *
 3  *^C

I decided to power-cycle the Cisco 1600, and it came back up in a normal  
state.  This interruption lasted roughly 7:00 to 8:00 PM, May 16.  Please  
investigate any log information which might be available on the router to try  
and determine what happened.

-Chuck

       Chuck Swiger | chuck () codefab com | All your packets are belong to us.
       -------------+-------------------+-----------------------------------
       She said, you've taken me for granted because I please you.  -P Simon
_______________________________________________
SysAdmin_List mailing list
SysAdmin_List () shot codefab com
<<<http://shot.codefab.com/mailman/listinfo/sysadmin_list
_______________________________________________
firewall-wizards mailing list
firewall-wizards () nfr com
http://www.nfr.com/mailman/listinfo/firewall-wizards