Re: barnyard to db

[Martin Roesch <roesch () sourcefire com>]

Jed Pickel wrote:
> > there are not currently any database output plugins available in barnyard.
> >  I have contemplated porting Jed's database output plugin over to
> > barnyard, but there are other things that need to be completed first.
>
> Note that I made an attempt at porting the db plugin over to barnyard
> beta2 a couple weeks ago and --- while I was able to get it partially
> working there were some problems that do not have obvious solutions
> (listed below). If you know of any simple workarounds or if any of
> this is addressed in beta3 please let me know.
>
> * The unified log format does not store any information about where
>   an event originated from (no host, interface, bpf filter string,
>   etc); thus, the database plugin is not able to determine which
>   sensor the event data is associated with. There are ways around
>   this, (eg. adding a new sensor each time barnyard executes, having a
>   single sensor for barnyard, etc...) but they all involve tossing out
>   the current notion of a sensor resulting in broken functionality in
>   one or more of the analysis apps that use the current db format.

Hmm, oversight here, we'll fix that.

> * User defined rules don't log any message. This comes from the fact
>   that barnyard requires the use of the sid-msg.map file and that all
>   user defined rules actually have a "sid". The obvious workaround is
>   for users to maintain their rules in two places, but I personally am
>   not in support of this. Any chance snort could auto-generate this
>   file and auto-assign sid's for rules that don't have them?

The program to build the sid-msg.map file should be available in CVS. 
(it's called regen-sidmap and it's in the contrib directory)

All the messages and cross reference data get loaded at run-time from
the sid-msg.map into internal structs.  Here's how to get the message
data from the system:

Sid *tmp = NULL;
tmp = GetSid(ad->log.event.sig_generator, ad->log.event.sig_id);

The message is available at tmp->msg.
 
> * Alert / Log - I'm not interested in maintaining a separate database
>   plugin for each logging facility. In beta2 there was no way to
>   connect an output plugin to more than one input type. As it stands,
>   I connect only to "log".

All the event data is in with the logs anyway, so there's no need to
write a plugin to handle the alert type for the purposes of ACID.  The
alert spool (event spool) is made just for notification, you're more
likely to want the log data...

> > > I noticed that in the lecture you gave at IO Wargames,
>
> > you were mentioning 20,000 writes per
> > > second though barnyard, compared to +-800 with MYSQL output
> > > plugin from snort, and was wondering if that was to a
> > > database, and if so, which one?  Is this something for the future?
> >
> > The benchmark mentioned was obtained using a high speed embedded database.
> >  I will let Marty talk more about that if he wants to.
>
> I've heard others mention these numbers. I am interested in seeing
> some supporting data for this. Also, has anyone measured the
> performance difference between barnyard and simply replaying pcap
> through another instance of snort?

Roman sent me the data on the max insert rate for MySQL a few months
ago.  Batch-replaying the data thru Snort into the db is fine, but then
you're getting into signifigant lag periods from the time an event
happens and when it's made available in the DB.  Barnyard is
specifically designed to read the binary spools as they occur and
translate the data to the final output format.  This is a much better
way of handling things than having Snort act as a monolithic process
doing all the NIDS tasks + DB writes.

There's some confusion over the numbers above.  I said Snort can *spool*
20k+ alerts per second to the disk (i.e. to the unified binary spool
files), barnyard writes them out to the output facilities at whatever
rate they support (which is not all that important once the data has
been output from snort).  The DB we're using at Sourcefire supports
several thousand events per second inserted, but I'd have to check the
exact numbers.  [Caveat: YMMV, we're running on 866MHz P3s with 512MB
RAM and U160 10k RPM SCSI drives, you won't see the same performance on
a 300MHz Celeron.]

The fact of the matter (and the whole point of barnyard at this time) is
that insert performance ceases to matter once you're using barnyard. 
The events can be spooled to disk in their binary form *much* faster
than they can be inserted into any database and can either be spooled in
"real-time" to the output system from there or saved for later.  I'm not
saying that you *have* to use barnyard, but if you're going to be
writing to DBs you should really think hard about it.

      -Marty

--
Martin Roesch - President, Sourcefire Inc. - (410)552-6999
roesch () sourcefire com - http://www.sourcefire.com 
Snort: Open Source Network IDS - http://www.snort.org

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users