Re: traffic analysis

[jseymour () LinxNet com (Jim Seymour)]

Vladimir Parkhaev <vladimir () arobas net> wrote:
> Quoting R. DuFresne (dufresne () sysinfo com):
> > In the unix world, sed/awk/grep are your friends, iin the windows world
> > there are ports of those unix tools that can help in this area...
>
> Well, you can have many friends but only one is the best, right?
> That will be perl. It is cross-platform as well... 

I agree.  After battling with complex this-piped-to-that and temp files
for intermediate storage for years, I finally broke down and learned
Perl several years ago.  For logfile analysis, Perl's a God-send,
IMHO.

As for the original question (looking for a log analysis tool)...

True, there are a lot of pre-rolled solutions out there.  But I've
found that many of them, by including everything *and* the kitchen
sink, seem to have about as steep a learning curve as learning an
entire new language.  Which, to my way of thinking, rather misses the
point ;).

Here's how I go about it...

I log *everything* in excruciating detail.  I ensure my logs have
accurate time stamps.  (Inaccurate timestamps can render certain
logfile information nearly [?] worthless.)  I archive the raw logfiles
forever.  I never alter the logfiles.  (Other than compressing them for
storage.)

As for processing logfiles for (summary) reports, I do this:

    1. Identify which logfile entries are definitely of interest,
       determine which bits are the important/interesting ones, and
       write expressions to snag 'em out and stick those bits in either
       arrays or hashes.  

       The stuff in the arrays/hashes is then used to generate summary
       reports, grouped into related sections.  Some sub-reports may be
       sorted in chronological order, some by "how many times <blurfl>
       occurred" and some sorted by what I feel to be another field
       that's of greatest interest.

       I try to keep the "interesting bits" to the absolute minimum.  I
       figure that if I need more detail, I can always go back to the
       raw logs and dig it out.  I use these "summary" reports for an
       overview of what's going on and to alert me that something needs
       closer examination.  Too much in the summary and one might just
       as well just look at the raw logs in the first place, IMO.  Too
       much in the summary might cause the more important things to be
       missed, I feel.

    2. Identify which entries are "noise" and write expressions to
       throw these away.  (Basically: "next if(expression);"

       I'm *very* careful to write these to be quite specific so that
       nothing I don't *want* skipped is accidentally skipped.

    3. Everything else goes into an "unknown" array and is
       reported as such, in original, raw format.  As things appear
       here, I decide whether they belong in (1) or (2), and modify my
       script appropriately.

Hope this helps.

Regards,
Jim
-- 
Jim Seymour                  | PGP Public Key available at:
jseymour () LinxNet com         | http://www.uk.pgp.net/pgpnet/pks-commands.html
http://jimsun.LinxNet.com    |
_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards