Re: HELP ON SNORT

[Martin Holste <mcholste () gmail com>]

> @martin
> How are you building time
> based metrics on 1 million records in less then a second?
>
> This all sounds pretty intense.. i'll pull the source and give it a go.

Just run "contrib/install.sh node" and "install.sh web", and it should
pretty much install itself.  A VM of Ubuntu works great.

MySQL is only used for storage, not searching.  All queries hit Sphinx
(sphinxsearch.com, the engine that powers Craigslist).  Sphinx does
pseudo-map-reduce and full-text searching, and is orders of magnitude
faster at indexing than the public version of Google's BigTable
(Hbase).  Now that PostgresQL 9.1 has non-doublewrite-buffered tables
like MySQL, you could use it instead if you needed to.  The gist is
that MySQL can batch load at 100k rows/sec, and Sphinx can index at
100k rows/sec when reading from MySQL.  So, if you run 1-minute batch
jobs, your peak rate is 100k events/sec, with a sustained rate of
30k/sec (due to occasional consolidation which is necessary).

This is pretty technical, but I think it's a good thing to point out
on the list because Dustin's point is dead-on: you can't expect any
standard database to perform advanced analytics at high volumes
(unless you're throwing a massive amount of hardware at it, which is
out-of-scope for this discussion).  So, either you tune your sensors
so the volume is low, or you use something log-based, like ELSA or
Splunk.

Also, contact me off-list or on the ELSA list at
http://groups.google.com/group/enterprise-log-search-and-archive if
you have any questions on it.

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!