Re: Snort/Barnyard2 performance with remote DB

[Mike Lococo <mikelococo () gmail com>]

On 02/29/2012 08:47 PM, beenph wrote:
> Batch insertion is something already tought and will probably be
> available in the next release. For design/consistency reasons it will
> not be used with the new schema.

That's great news, and will likely completely clear up the latency
bottleneck.

> > My solution was to move the database local to the barnyard2
> > instance and use a more latency tolerant protocol to push the
> > events back to a central system.  In my case, an Arcsight connector
> > is doing that work and it employs both batching and multiple
> > transport threads to achieve latency tolerance.
>
> Well if you feel confortable having distributed db system and you
> think you have no issue synchronizing betwen all your deploiment then
> its fine but personally its only part of the problem. Also having a
> DB on the system that uses snort is kind of a performance hog in its
> self and personaly i would rather deal with latency than having
> potentialy dropped packets.

I'm aware of general best-practice advice in sensor design, but I
monitor my snort stack extensively and use that data to drive my design
decisions. I'm confident that it's working as expected.

An overview of my monitoring setup is at:
http://mikelococo.com/2011/04/snort-zabbix/

> As i said before the REAL issue with the "old" plugin was the
> incredible amount of time it was quering the DB for 1 event, this
> dramatically reduced kind of fix the problem of using it over a high
> latency network, unless you use barnyard2 in combinaison with a
> special snort ruleset that would generate 2mb of data every second
> and you try to force that data arround the world over a 128k/s link,
> then you might have other issue.

Below are theoretical maximum alert rates for a system that inserts 1 
alert per tcp round trip.  If I understand barnyard2's architecture, a 
single instance of barnyard2 cannot exceed these maximums without the 
addition of code to handle batching or parallel insert threads:

80-100 ms -> 10-12.5 alert/sec  - Typical latency for US east-cost to
                                   US west-coast or to western-europe
200 ms -> 5 alert/sec           - Typical latency for US east-coast to
                                   asia or australia
300+ ms -> 3 or fewer alert/sec - Typical latencies to locations with
                                   poor-quality internet infrastructure

These theoretical numbers are 5-10 times larger than what I measured in 
my last round of real-world tests, which were performed before the most 
recent round of optimizations.  I believe they already include the 
best-case improvement for the optimizations that are about to be 
released, but I haven't tested the newly optimized code.  I'd love to 
see someone else's numbers, but won't have time to investigate myself 
until batching lands, which is when I expect there might be enough 
improvement to facilitate wan DB's for my use-case.

Cheers,
Mike Lococo

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!