Re: Bogon IPs traffic only seen by netflow, confined within a VLANonly

[Roland Dobbins <rdobbins () cisco com>]

You can clear the NetFlow cache manually, FYI - 'clear ip cache', I  
believe.

What IOS/platform are you running on the box that's seeing the  
NetFlow?  At what point in the topology is it located?

If the router is running a recent T-train image, there's a feature  
called 'RITE' which is essentially SPAN for the router - you could  
connect your tcpdump box there and sniff the traffic to see if the  
traffic is present.

l2trace should work if you have the actual MAC address of the box(es)  
in question; but again, that info might be spoofed or mangled.

Again, on what input/output interfaces are you seeing the traffic in  
NetFlow?  Can you paste 'sh ip cache flow' output?

Feel free to contact me 1:1 if that would help.


On Apr 11, 2006, at 7:12 AM, Stef wrote:

> UPDATE:
>
> Thanks to all who replied, and continue(d) with suggestions. We still
> have not been able to isolate the problem - the attempt is now to shut
> down one port at a time, and watch netflow to see when it stops (we
> are waiting for each port for a 2 * cache expiration, so that we do
> not risk to move past the "guilty" port). No span/monitor session
> revealed the needed info, either going through each port, entire VLAN,
> etc. Shutting down ports has an impact on production, though, so it
> has to be done fast, and coordinated with the end points, so it has
> been moving slow (thus the lack of a follow-up, lately). We are beyond
> 66% of the ports, and still have not isolated it, so it may be - in
> the end - a faulty flow record, as someone already suggested.
>
> As far as this latest recommendation - I wish we had the MAC ;) - this
> whole exercise is an attempt to identify it, after which I am pretty
> sure "game over"!
>
> Stef
>
> On 4/11/06, AJ Cochenour <ajc () mytcpip net> wrote:
> > Assuming CatOS on the C4506:
> >
> > 1. Issue the following to locate port if host may be directly  
> > connected:
> >      'sh cam dynamic | include <Questionable Source MAC --
> > FF-FF-FF-FF-FF-FF>'
> > 2. If operating within distributedswitch network issue the following
> > (assuming Cisco/Foundry topology):
> >      'l2trace <Switch MAC -- FF-FF-FF-FF-FF-FF> <Questionable  
> > Source MAC
> > -- FF-FF-FF-FF-FF-FF>' to reveal L2 path to device(s) in question.
> >
> > If using IOS on C4506 issue the following to locate port if host  
> > may ne
> > directly connected:
> >      'sh mac-address-table | include <Questionable Source MAC --
> > FF-FF-FF-FF-FF-FF>'
> >
> > In the absence of recent CatOS/IOS firmware to support the commands
> > provided  TCL scripting would be a viable alernative.  Happy hunting
> >
> > AJC
> > ajc () mytcpip net
> >
> > > Combining the below from Nicolai with setting up the port in  
> > > promiscuous
> > > mode and running a Network Sniffer tool would give you enough  
> > > data to
> > > track it down, I would think.
> > > -
> > > Jean-Raymond Xavier Pierre
> > > Scientific Computing and Computing Services
> > > Stanford Linear Accelerator Center
> > >
> > > -----Original Message-----
> > > From: Nicolai van der Smagt [mailto:nicolai.vandersmagt () bbned nl]
> > > Sent: Monday, April 10, 2006 2:12 AM
> > > To: stefmit () gmail com
> > > Cc: incidents () lists securityfocus com
> > > Subject: Re: Bogon IPs traffic only seen by netflow, confined  
> > > within a
> > > VLANonly
> > >
> > >
> > > Stef,
> > >
> > > Why don't you just span the entire VLAN to a machine capable of  
> > > running
> > > tcpdump, use tcpdump -e to find the hardware address of the  
> > > station(s)
> > > sending the traffic, and look up that address in the CAM table of  
> > > your
> > > switch? Would be quicker than spanning 1 port at a time..
> > >
> > > Kr,
> > > Nicolai van der Smagt
> > >
> > >
> > > -----
> > > I have a question, that - in case someone has seen this somewhere  
> > > - may
> > > save us a lot of work: our netflow tool has been reporting lots  
> > > of traffic
> > > (100s of MB/day) between some bogon IPs: 0.10.94.27 to a few IPs  
> > > in the
> > > 37.245.0.0/24 network (e..g 37.245.0.64,  37.245.0.18,  
> > > 37.245.0.14, etc.).
> > > The report comes exclusively for one VLAN, from a
> > > 4506 switch. The IP protocols being reported are not among the  
> > > well known
> > > ones (TCP, UDP, ICMP, etc.), but rather #140 (for the majority of  
> > > traffic)
> > > and #63 (and some other ones). We have tried to reach (ICMP echo,  
> > > nmap,
> > > etc.) those IPs from various stations from the same VLAN, with no  
> > > success.
> > > Monitoring a few ports (span to a probe), at random, have not  
> > > revealed any
> > > ARP traffic for those IPs, either, thus
> > > - at this stage - being unable to determine who is responsible  
> > > for that
> > > traffic. The default gateway for all the systems on that VLAN  
> > > does not see
> > > any of this traffic, either - and neither any other systems form  
> > > that
> > > point on, upstream, al the way to the internal interface of the  
> > > firewall,
> > > which makes us think that somehow that odd traffic is really  
> > > confined to
> > > that specific VLAN (thus - probably - some sort of spoofing,  
> > > combined with
> > > systems aware of each other's MAC, thus no need to hit the  
> > > gateway ...).
> > >
> > > The next step is to write a script on the switch itself (TCL -
> > > probably) or on an external probe, so that we could span/monitor  
> > > one port
> > > at a time, and go through all the ports on all blades from that
> > > 4506 (4 modules * 48 ports), until the probe hooked up t the  
> > > destination
> > > port will report the traffic we are looking for (as netflow data  
> > > reports
> > > this traffic going on continuously - so it must exist at all  
> > > times), from
> > > one of the ports. Another simpler approach (but unfeasible,  
> > > unfortunately,
> > > in our scenario, due to the need for machines to be up 3 shifts/ 
> > > day) would
> > > be to shut down one system at a time, and watch netflow when it  
> > > stops
> > > reporting this junk.
> > >
> > > So ... short of doing one of the above - has anybody seen this  
> > > before?
> > > Do you have an idea of how else we could trace down the perpetrator?

----------------------------------------------------------------------
Roland Dobbins <rdobbins () cisco com> // 408.527.6376 voice

     Everything has been said.  But nobody listens.

                   -- Roger Shattuck