Re: Base Barnyard and Unified Logs

[Jerry <gll () inel gov>]

Just saw the discussion about barnyard and DB's.  Here is some info I gained in having to deal with
consolidating data from two snort DB's in to a single application.

Now that generators have been assigned to various parts of snort, they need to be employed in the DB
schema (generator:sid:rev) as a key to a signature.  The generator-id is needed since the
pre-processors usually start the SIDS=1!  The problem becomes more complicated in that the
signature, sensor, reference, and classification tables are built on the fly by the DB-plugins.  The
plugins first try to grab the signature from the DB using msg (sig_name), Rev (sig_rev) and SID
(sig_sid). If found then use the assigned (via MySql auto-increment) sig_id.  If not, create the
record.  Note that the generator-id is never mentioned in the DB.

The signature, sensor, reference and classification tables are "normalized" tables created
on-the-fly by the database plugin.  Their ordinal (created by the order of insertion) is used in the
other tables (eg. event) to save time and space.
If you are only using a single DB, there isn't any problem, except as Joel wrote below, if you have
to clean the DB, your mapping between SID ->(sig_name,sig_sid,sig_rev) is lost.  If you are
combining the two DB's, for example an inside and an outside, into a single application/DB like we
are, you run in to data collisions and race conditions.


To solve these issues, I ended up writing scripts to insert (read preload) the following tables:
        .signature, from all of the rules
        .sensor (including the 'read from file' entries)
        .reference (reference.config), and
        .classification (classification.config)
The input to the scripts will never shrink.  Thus I will maintain the mapping.


As a side note.
I also discovered that not all of the sid/msg-s from the pre-processors were notated in the
gen-msg.map.




Wes Young wrote:
> I think it's an issue on both sides.... I don't think it was just you
> guys... although to fix it just look at the SID instead of the sig_id.
>
> I think the sql output plugin might need some editing down the line, but
> as far as I can tell, it might break the overall schema of the snort db
> (if the tables rely on teh sig_id)...
>
> just thinking outloud, i'll post to your site tomorrow.. if you have
> anymore ideas, let me know, i'd be happy to help...
>
> wes
>
> Joel Esler wrote:
>
> > Let us (the development team) look into this, and we'll get back to
> > you.  In the meantime if you don't mind, could you open a ticket at
> > www.sourceforge.net/projects/secureideas so we may track and notify
> > you of the fix.
> >
> >
> > Joel Esler
> > BASE Project Lead
> > http://secureideas.sourceforge.net
> > On Mar 14, 2005, at 17:49, Wes Young wrote:
> >
> > I realize this. Which is why I stated (more than once) that Aanval
> > (another analyzation tool) resolves sids from the snort database w/o any
> > issue or needing to know where sid-msg.map is, even when re-initiallized.
> >
> > I use snortcenter2x to manage my sensors, from this I have created a
> > script that autogenerates my sid-msg.map everytime barnyard starts from
> > my rules database.
> >
> > IMO: I think I might need to re-write the mysql plugin for barnyard,
> > there are too many tedious ID's in there that are helping confuse the
> > problem. Everything *SHOULD* revolve around the rule SID... it seems
> > like everythin in the db has it's own type of ID, some needed,
> > some...over duplicated, it seems.
> >
> > BASE alone seemes to look at teh SIG_ID and not the SID when it looks up
> > the sig name to generate its cache.... why would there be a need to
> > generate a seperate id for each sig in the signature table? To compound
> > that, barnyard doesn't generate the entire sig list into the DB on
> > runtime, only when it's needed, seems feasible, but what happens if you
> > clear one of the tables.... you just F'd your entire setup becuase the
> > SIG_ID starts back at 0 and your SID stays the same, so BASE read's the
> > SIG_NAME incorrectly (if at all) and you're hosed... may not pose a
> > problem for smaller db's that don't need that sort of flexibility, but
> > (again, IMO) seems like centralizing anything dealing with signature
> > resolution, evertying should revolve around the SID....
> >
> > I'm thinkin the reason why aanval seems to work is because it doesn't
> > even look at the SIG_ID, which BASE might.... I just can't find the code
> > to prove anything....(in BASE).
> >
> > Esler, Joel CNTR/Sytex wrote:
> > | BASE gets it's info from the database.  What you put in the database is
> > | up to you.  BASE reads it raw out of the database.  I agree with
> > | everyone else, I think your sid-msg.map is messed up.  I would point
> > | barnyard at your sid-msg.map that is updated.  (I would also recommend
> > | using IDSPM to manage your rules and auto-fix your sid-msg.map)
> > |
> > | BASE does not read raw files, it will not read your sid-msg.map.  I had
> > | a discussion with Marty recently about possibly generating the
> > | sid-msg.map on startup, or some kind of method to autogenerate it so
> > | this type of thing does not happen.
> > |
> > | Joel Esler
> > | BASE Project Lead
> > |
> > | On Mon, 2005-03-14 at 17:30 -0500, Wes Young wrote:
> > |
> > | I know... I have done that... which is why Aanval works...
> > |
> > | but Base Does not.... trying to figure that part out (where base gets
> > | all it's info)
> > |
> > | Paul Schmehl wrote:
> > | | --On Monday, March 14, 2005 04:05:36 PM -0500 Wes Young
> > | | <wcyoung () buffalo edu <mailto:wcyoung () buffalo edu>> wrote:
> > | |
> > | |>
> > | |> I thought barnyard uses the sid-msg.map to read the sid and then
> > inserts
> > | |> ~ the sig details to the DB, no? I don't specify the sid-msg.map
> > anywhere
> > | |> else, hense why Aanval works perfectly, but base, does not.
> > | |>
> > | | You *do* have to tell barnyard where the sid-msg.map is. 
> > Otherwise it
> > | | will not be able to parse the sids to msgs.
> > | |
> > | | You do it one of two ways:
> > | |
> > | | In the config file:
> > | | config sid-msg-map: /path/to/sig-msg.map
> > | |
> > | | On the commandline:
> > | | barnyard -s /path/to/sid-msg.map
> > | |
> > | | Paul Schmehl (pauls () utdallas edu <mailto:pauls () utdallas edu>)
> > | | Adjunct Information Security Officer
> > | | The University of Texas at Dallas
> > | | AVIEN Founding Member
> > | | http://www.utdallas.edu
> > | |
> > | |
> > |
> > | --
> > | Wes Young
> > | Network Security Analyst
> > | University at Buffalo
> > | GPG Key: http://saxjazman9-security.blogspot.com/2005/01/gpg-key.html
> >
> > -------------------------------------------------------
> > SF email is sponsored by - The IT Product Guide
> > Read honest & candid reviews on hundreds of IT Products from real users.
> > Discover which products truly live up to the hype. Start reading now.
> > http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> > <http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click>
> > _______________________________________________

>


-- 
------------------------------------------------------------------
Jerry Litteer
Cyber Security Office             e-mail:  gll () inel gov
Idaho National Laboratory (INL)
POB 1625 M.S. 3640                Phone: (208) 526-9117
Idaho Falls, Id. 83415-3640       FAX:   (208) 526-5700


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users