Re: Re: ACID/BASE vs PRELUDE

[Gene R Gomez <gene () gomezbrothers com>]

Hey Kris,
Yeah, this has been a big issue for many people using the Open Source 
version of the software; enough that we have put some time into 
developing a new schema that's been deployed into the commercial version 
of the software.  We've tested this new schema up to about 480K+ events, 
and Prewikka can render that in about 3 seconds on decent hardware.
We're currently working on back-porting portions of the schema update 
functionality into the Open Source version.  Unfortunately, as a team we 
set a deadline for the 0.9 release (it's been in development for a very 
long time) that didn't allow for the inclusion of a schema update for 
the Open Source version.
Now, on another note, I did some research for the team on ways to 
mitigate this from the database server side.  Most of the default MySQL 
settings are pretty bad in terms of allotted RAM and cache space for 
both queries and indices.  I'll admit, the performance is still not so 
good after tweaking MySQL up, but it's far better than the numbers you 
give below.
At any rate, improvements are on their way.  We can only work so fast, 
though.  ;)

Gene R Gomez

Kris Karas wrote:

> We set up Prelude to funnel the output from several Snort sensors into 
> a backend DB (MySQL) from which we used Prewikka (the python-based 
> display engine) for viewing.  Prewikka is very nice in concept, 
> filters work well, does a decent job of organizing data visually using 
> color-coding.  The Prelude-Manager is, I think, the best part of 
> Prelude; it allows data from various sensors to be aggregated, stored, 
> and sent upstream to either another aggregator or else into a database 
> for display.  (This is nearly identical to the "event collector" that 
> we used in our ISS RealSecure deployment.)
>
> Back to Prewikka for a moment...  I'd prefer an interface that shows 
> aggregate counts in descending order of severity, with an ability to 
> drill-down by src/dst network/address and, eventually, down to 
> individual discrete events.  Prewikka instead shows all events in the 
> order in which they were received, but allows you to apply filters if 
> you have some clue of what you're looking for.
>
> On the negative side, Prelude-Manager's DB has serious performance 
> issues that need to be addressed.  The fundamental limiting factor is 
> its schema, which puts most forms of data into a single table, along 
> with a tag that identifies the nature of the item.  Just pulling up 
> one screenful of log data (50 signatures) could take several minutes.  
> Profiling to see where the bottleneck was shows MySQL fielding some 
> 500+ queries per second to satisfy just that one page view.  If you 
> pull up a thousand entries, for example, you might as well go out for 
> lunch.




-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users