Re: Sizing of a box requiring 2x10Gbps

[Russ Combs <rcombs () sourcefire com>]

Good points Will.  Some clarifications / comments below.

Russ

On Wed, Jul 7, 2010 at 7:06 PM, Will Metcalf <william.metcalf () gmail com>wrote:

> For the most part I agree with what you guy's are saying although
> there are some things that we have learned from working on the
> "IDS-Who-Must-Not-Be-Named" that you could do to reduce packet loss of
> stock snort that I'm actually really surprised you haven't done
> to-date.
>
> 1. Support reading multiple packets per pcap_dispatch() call.

Snort has been doing this for a while (maybe since 2.7.0?).  Probably not
when you last looked at it though.

> 2. Along the same lines since libpcap-1.0 if the functionality is
> available in the linux kernel, libpcap will use a mmap'd ring
> buffer(essentially phil woods patch was integrated).  The size of this
> buffer can be set via pcap_set_buffer_size() but you don't allow this
> as an option to your users.

Snort has been bogged down with an old pcap for quite a while.  However, the
memory mapped pcap has been an option.  Snort 2.9.0 does in fact jump up to
libpcap 1.1.1 and provide a way to configure via pcap_set_buffer_size().

2.9.0 goes even further here.  It introduces a DAQ (for data acquisition),
basically an API and a suite of pluggable modules for packet acquisition and
injection.  In addition to using pcap, an afpacket module provides inline
memory mapped performance and there are a number of other modules as well,
including ipfw, ipq, and nfq.

> 3. Since I have a feeling you will never open source your in-kernel
> load-balancing juju that you reference here
>
> http://vrt-sourcefire.blogspot.com/2010/06/single-threaded-data-processing.html
> (I'm
> going to buy you a nehalem chip btw), why don't you guy's add support
> for PF_RING cluster-sockets?  This way you can be happy with your
> FUDless single thread while still allowing your users to load balance
> based on flow by simply firing up multiple disparate snort processes
> with the same PF_RING cluster id.  The time that it would take to
> build this new packet acquisition method would be minimal.

Creating a PF_RING DAQ module is worth investigating.


> Just my 2 cents..
>
> Regards,
>
> Will
>
> On Wed, Jul 7, 2010 at 6:18 AM, Joel Esler <jesler () sourcefire com> wrote:
> > It would be very difficult to achieve those kinds of speeds without a
> commercial Snort appliance like Sourcefire.
> > Sorry for the plug.
> >
> > --
> > Sent from my iPad
> >
> > On Jul 7, 2010, at 4:28 AM, "Sven Juergensen (KielNET)" <
> s.juergensen () kielnet de> wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > Hi list,
> > >
> > > I'm playing with the thought of implementing an
> > > IDS for our network. Now, for the box handling
> > > this, a bit of advice would be appreciated. It
> > > needs 2 10GE interfaces and would have to soak
> > > up a throughput of about 4GBps tops. The amount
> > > of accumulated data should last about a week.
> > >
> > > Does anyone know the rough specs for a box to
> > > deal with this?
> > >
> > > Thanks in advance and regards,
> > >
> > > Mit freundlichen Gruessen,
> > >
> > >       i. A. Sven Juergensen
> > >
> > > - --
> > > Fachbereich
> > > Netze und Rechenzentren
> > >
> > > KielNET GmbH
> > > Gesellschaft fuer Kommunikation
> > > Preusserstr. 1-9, 24105 Kiel
> > >
> > > Telefon : 0431 2219-053
> > > Mobil   : 0170 403 5600
> > > Telefax : 0431 2219-005
> > > E-Mail  : s.juergensen () kielnet de
> > > Internet: http://www.kielnet.de
> > >
> > > Geschaeftsfuehrer Eberhard Schmidt
> > > HRB 4499 (Amtsgericht Kiel)
> > >
> > > PGP details at
> > > http://pgp.kielnet.de/sjuergensen/
> > >
> > > -----BEGIN PGP SIGNATURE-----
> > > Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
> > >
> > > iEYEARECAAYFAkw0Or0ACgkQnEU7erAt4TLYvQCgro8f56KLyt6QH4gSql4GO8CS
> > > c+8AoPJfbeK3Ft+jgqmv3gFih7K41tkw
> > > =8R2s
> > > -----END PGP SIGNATURE-----
> ------------------------------------------------------------------------------
> > > This SF.net email is sponsored by Sprint
> > > What will you do first with EVO, the first 4G phone?
> > > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> > > _______________________________________________
> > > Snort-users mailing list
> > > Snort-users () lists sourceforge net
> > > Go to this URL to change user options or unsubscribe:
> > > https://lists.sourceforge.net/lists/listinfo/snort-users
> > > Snort-users list archive:
> > > http://www.geocrawler.com/redir-sf.php3?list=snort-users
> ------------------------------------------------------------------------------
> > This SF.net email is sponsored by Sprint
> > What will you do first with EVO, the first 4G phone?
> > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> > _______________________________________________
> > Snort-users mailing list
> > Snort-users () lists sourceforge net
> > Go to this URL to change user options or unsubscribe:
> > https://lists.sourceforge.net/lists/listinfo/snort-users
> > Snort-users list archive:
> > http://www.geocrawler.com/redir-sf.php3?list=snort-users
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://www.geocrawler.com/redir-sf.php3?list=snort-users
------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users