Re: Recording slow scans

[Darren Reed <darrenr () reed wattle id au>]

In some email I received from Stephen P. Berry, sie wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
>
>
> > Darren Reed <darrenr () reed wattle id au> wrote:
>
> > > > How many times do people need to reinvent the wheel ? 
>
> > > Until there's a GPL'd wheel out there that I happen to like.  Because
> > > sometimes all I want is the wheel, and most vendors are car salesmen.
>
> > Well, why don't you write one for us ?  I'm sure we'd all appreciate
> > your time and effort spent on such a project.
>
> I did code up the changes I mentioned previously (twiddling libpcap(3)
> and tcpdump(8) to accept multiple filters and output the results to multiple
> files).  When I have a few more spare cycles, I'll document the changes
> and submit them to the libpcap/tcpdump maintainers.  I have no idea
> how receptive said maintainers are to unsolicited changes, so I have
> no idea whether or not the changes are likely to make it into future
> releases.

well, in the mean time, why don't you make the patches available to the
rest of us anyway ?  I'm sure others would like to use/abuse/hack on them
to make them better.

> Anyway, this twiddle and a couple like it really clear up the
> bottleneck in first-order filtering---the major bottleneck which seems
> to be characteristic of most tcpdump-based IDSen.

Yup.

> But the reason why I'm mentioning all this here is to point out
> that with comparatively minor coding one can produce fairly
> usable tools with materials which are already freely available.

For some definition of "usable".

I still find it intriguing that "0 packets lost" is different on each of
NFR/tcpdump/lananalyzer.

[...]
> Unless we're talking at cross purposes, I'm pretty sure there's
> a bottleneck in the database.  If one is interested in analysing
> data over fairly long periods of time (in terms of months rather
> than hours, say), even reasonably narrow pipes are going to be
> producing enough data to bog down most analysis engines.

Bah, that's not a "bottleneck", as such.  That's just a matter of
sizing your data to be analyzed correctly with a machine to analyze
it.

> > One alternative might be to (say) have a box with 5 ethernet ports on it,
> > 1 being the data "tap" and the other 4 for syhponing data off to boxes
> > for processing.  For example, you might have one box dedicated to doing
> > TCP processing, another for UDP, another for ICMP/IGMP and another for
> > the remainder.
>
> I've considered something similar, but discarded the idea.  When I'm
> doing long-term analysis, I'm very interested in correlations.

The first problem is data collection.  Nothing stopping you doing that
once you have the data.