Re: HELP ON SNORT

[Dustin Webber <dustin.webber () gmail com>]

Martin,

I made a lot of assumption but in retrospec It's totally possible to
hit those benchmarks. I will pull down the source and play with it more.
Great work on this project and keep up the excellent work. It's refreshing
to see people getting into the research side on how we can solve the defsec
storage conundrum.

Dustin W. Webber
Dustin.Webber () gmail com


On Mon, Jan 30, 2012 at 11:09 AM, Martin Holste <mcholste () gmail com> wrote:

> > @martin
> > How are you building time
> > based metrics on 1 million records in less then a second?
> >
> > This all sounds pretty intense.. i'll pull the source and give it a go.
>
> Just run "contrib/install.sh node" and "install.sh web", and it should
> pretty much install itself.  A VM of Ubuntu works great.
>
> MySQL is only used for storage, not searching.  All queries hit Sphinx
> (sphinxsearch.com, the engine that powers Craigslist).  Sphinx does
> pseudo-map-reduce and full-text searching, and is orders of magnitude
> faster at indexing than the public version of Google's BigTable
> (Hbase).  Now that PostgresQL 9.1 has non-doublewrite-buffered tables
> like MySQL, you could use it instead if you needed to.  The gist is
> that MySQL can batch load at 100k rows/sec, and Sphinx can index at
> 100k rows/sec when reading from MySQL.  So, if you run 1-minute batch
> jobs, your peak rate is 100k events/sec, with a sustained rate of
> 30k/sec (due to occasional consolidation which is necessary).
>
> This is pretty technical, but I think it's a good thing to point out
> on the list because Dustin's point is dead-on: you can't expect any
> standard database to perform advanced analytics at high volumes
> (unless you're throwing a massive amount of hardware at it, which is
> out-of-scope for this discussion).  So, either you tune your sensors
> so the volume is low, or you use something log-based, like ELSA or
> Splunk.
>
> Also, contact me off-list or on the ELSA list at
> http://groups.google.com/group/enterprise-log-search-and-archive if
> you have any questions on it.
------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!