RE: parsing logs ultra-fast inline

["Tina Bird" <tbird () precision-guesswork com>]

marcus has been sufficiently saying what i do that i've not felt obliged to
participate in this thread, until finally:

> -----Original Message-----
> From: Marcus J. Ranum [mailto:mjr () ranum com] 
> Sent: Wednesday, February 01, 2006 1:04 PM
> To: firewall-wizards () honor icsalabs com
> Subject: [fw-wiz] parsing logs ultra-fast inline

<<juicy stuff clipped for brevity>>

> It comes back to what I was saying in my last few postings on this
> topic:
>       - look at your data
>       - then decide what you want to collect from it
>       - then collect it
>       - count the stuff you throw away
>       - vector the rest to a human's attention

or take this list back one step further. **before** you look at your data,
think about what you'd like to know! as has been said repeated, it's hard to
build anything comparable to a human brain (of whatever generation) for
extracting useful data; on top of that, log analysis gets really hard when
you take a month's worth of data and say "tell me what i need to know from
this."

but there's a big way to simplify, both in terms of technical complexity and
in terms of getting motivated to do it, *especially* if you're in the
position of just starting out. 

WHAT DO YOU WANT TO KNOW?

so f'r instance, imagine i've landed in a new job at a company without a
centralized logging infrastructure. the network is the usual conglomeration
of file servers, mail, web stuff, firewalls, routers, remote access. and
databases, of course. and some custom code. i'd go MAD if i tried to build
the uber-logging facility all in one go.

but i have one or two types of "reports" (if you want to call them that)
that i've found particularly useful in the past, based on my experience as a
firewall and VPN admin. here, i want to start by building a logging system
that will give me daily reports on VPN access (who used it, and where'd they
come from). very targetted, very finite, very plausible.

when i've got that little bit working, i can add reports on administrative
activity on the VPN concentrators, and then the firewalls.

you see where i'm going. the people who run the boxes *know the kinds of
events they care about*. they may not have any idea about how to tell
whether that thing happened, but that's a different issue.

the actual point being that if you use what you know at the very beginning,
to target how you build things, you can make it a lot easier. i suppose you
run the risk of building something so specific to VPN logs that you can't
generalize it, but at least for me, i try to work with reasonable
architecture tools that let me branch out.

and even if it's completely specific to VPN logs, it has the great advantage
of getting that particular task done, rather than spending a lot of time
trying to start by building the perfect generic parser, and getting bogged
down, and never delivering.

cheers - tbird

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards