Re: v2.8.4 incorrect logging to MySQL

[Martin Roesch <roesch () sourcefire com>]

Hi Jack,

I don't especially like polling either but it got the job done in BY
originally and as with most things I thought that people who wanted
better or faster methods could certainly write their own.

I'll agree with your point that using IPC methods is probably the way
to do these days, as I recall I was worried about context switch
thrashing between userland processes if the IPC mechanisms got flooded
by event data.  Flooding is a real possibility if Snort is
misconfigured which seems to be the norm instead of the exception in
many cases.  Since instructions are "practically free" on modern x86
boxes these days and there's lots more RAM available I think context
switching overhead and flooding is much less of a concern than it was
back in 2000-2001.

The unified and unified2 code was setup to do the minimal amount of
manipulation on the data as possible and to get it out of Snort ASAP
so that we could get back to the packet processing loop.  The goal
within Snort is to communicate the data it needs to get out as fast as
possible and get the next packet.  Packet loss is the thing we're
really looking to avoid.

I felt that files were preferable because you could run the spooler at
your leisure instead of having to have a listener operating
continuously and losing data in the event that it went down or stopped
working for whatever reason.  Additionally, if the listener became
overwhelmed with data then you could potentially lose events.  You
could also replay the data later, etc ad nauseum.

I'd certainly be willing to revisit it today and see how much improved
things are.  The system you wrote sounds pretty interesting!  What
happens if the listener process goes away for a few minutes or hours
in your design?

Marty


On Tue, Apr 14, 2009 at 11:44 PM, Jack Pepper
<pepperjack () afferentsecurity com> wrote:
> Quoting Martin Roesch <roesch () sourcefire com>:
>
> > Jack Pepper:
> >
> > I believe the latency between detection and spooling to a database
> > using the unified/Barnyard method is preferable to losing packets and
> > missing detections altogether using the direct-to-database method.  We
> > use Unified2 here at Sourcefire for our commercial devices.
>
> Yeah, I don't use direct to DB either.  My earier comments were just
> to say that for a guy who has been programming real time embedded
> systems for 30 years, unified is not my thing.  not bad code, just not
> my cup of tea.  As I tell the young guys who come here, "any time you
> think loop polling for data sounds like a good idea, you have made a
> design error."  And using a flat file on a journaled file system as a
> spooling pipe is creepy.  it looks like ketchup on ice cream.  But
> that's just me.  If you want a pipe, why not use a pipe?
>
> I happen to really like writing weird realtime multi-threaded apps
> with lots of IPC stuff embedded in them.  For my own use, I wrote a
> distributed correlator system wherein the snort output method writes
> to a shared memory handle where an external (non snort) process was
> waiting on the mutex. then it goes though a massively parallel system
> of "escalator" daemons that have a separate system and configuration
> elements for each type of event escalation (SMS, DB, text-to-skype
> (via festival), email, Full-Alert-Logging, tcpdump, syslog, etc) and
> each type of criticality.  It's my own beast and it works only as long
> as snort doesn't change the handoff from detect to output.  At the end
> of the escalation tree, everything goes to a remote DB on a central
> system where it gets mined for possible nuggets.  (The mining code
> does not work nearly as well as it sounds).
>
>
> BTW Marty:  I have been a loyal snort user since 1.6 and I think that
> if you wrote the code then you have to answer only to yourself for
> what kind of licence you put on the code.  You should do what works
> for you.  thanks for snort.  Snort has made IDS and IPS real objects
> and not just sales blather.  The Snort community has much to be proud
> of as a model of how the open source community is supposed to work.
> If people disagree with your direction, they should start a fork.
>
> jp
>
>
> --
>
> Framework?  I don't need no stinking framework!
>
> ----------------------------------------------------------------
> @fferent Security Labs:  Isolate/Insulate/Innovate
> http://www.afferentsecurity.com
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by:
> High Quality Requirements in a Collaborative Environment.
> Download a free trial of Rational Requirements Composer Now!
> http://p.sf.net/sfu/www-ibm-com
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users



-- 
Martin Roesch - Founder/CTO, Sourcefire Inc. - +1-410-290-1616
Sourcefire - Security for the Real World - http://www.sourcefire.com
Snort: Open Source IDP - http://www.snort.org

------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users