Some help with performance issues

[Jorge Cuevas <jcuevas () nesys-st com>]

Hi all:

We are now implementing OSSIM (www.ossim.net) on a clients network 
(snort, pads, p0f, ntop, and arpwatch running), and we have an issue 
with the amount of traffic we can monitor.

The actual phase of the project is about measuring that amount using the 
only server we have right now: Dell  PE2950 Xeon 5110 1.6GHz/4MB 1066FSB 
with 4GB FB 667MHz Memory and an Intel Pro/1000 PT NIC PCIe Card. There 
is a Debian etch 64bits running. We need another NIC only for 
administering purposes. The traffic in our production environment is 
injected from a Cisco 6500 core, and we have four trunked links of 1Gbit 
each, which is being inserted (Spanned) 1 to 1, 2 to 1, 3 to 1 and 4 to 
1 during the tests. In the 1 to 1 spanning we have a 17,5 Mb peaks and 
80 kbits average traffic, and in the 2 to 1 spanning we have 63 Mb peaks 
and 273 kbits average. Doesn't seem to much for PF_RING (even though the 
system will be overloaded with so many apps: all the sensors + database 
+ correllation engine ...).

We have a testing environment, with a Intel(R) Pentium(R) D CPU 3.00GHz, 
1 GB memory, a Realtek RTL-8169 NIC, where we want to reproduce the 
improvement of PF_RING functionality. It also has a separated NIC for 
administration. There is a Debian etch 32bits running. We will use 
tcpreplay for injecting traffic (using a real capture from the real 
environment)

In our testing environment, following the howtos about PF_RING, we have 
downloaded the code, edited and run mkpatch... We have a comment here... 
this script does the patch itself?? You don't need to execute "patch" 
afterwards as the instructions say. Just a point.

We execute make menuconfig and select the options (rx and tx polling for 
r8169 as a module, PF_RING sockets as a module) and generate the new 
kernel which we boot (Debian like). No problem

We insert the LKM ring.ko -> insmod ring.ko bucket_len=1500 num_slots=8192

Success:

"Welcome to PF_RING 3.7.8
(C) 2004-08 L.Deri <deri () ntop org>
NET: Registered protocol family 27
[PF_RING] Bucket length 1500 bytes
[PF_RING] Ring slots 8192
[PF_RING] Slot version 9
[PF_RING] Capture TX Yes [RX+TX]
[PF_RING] IP Defragment No
[PF_RING] initialized correctly
[PF_RING] registered /proc/net/pf_ring/"


Next we compile libpfing and libpcap, generating the new dynamic 
libraries in /usr/local/lib. Seems a success...

We go to ntop:

we use ./autogen.sh LDFLAGS="-L/usr/local/lib -lpfring -lpcap -lpthread" 
as the options an seems a success, the execution gives PF_RING loading 
options... But, executing ldd against the binary: where is the libpcap 
linking option??
# ldd /usr/local/bin/ntop
      linux-gate.so.1 =>  (0xffffe000)
      libpfring.so => /usr/local/lib/libpfring.so (0xb7ee7000)
      libntopreport-3.3.5.so => /usr/local/lib/libntopreport-3.3.5.so 
(0xb7e3e000)
      libntop-3.3.5.so => /usr/local/lib/libntop-3.3.5.so (0xb781e000)
      libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb780c000)
      libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb77de000)
      libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb76ad000)
      libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb766e000)
      libcrypto.so.0.9.8 => /usr/lib/i686/cmov/libcrypto.so.0.9.8 
(0xb7534000)
      librrd_th.so.2 => /usr/lib/librrd_th.so.2 (0xb74ed000)
      libgdbm.so.3 => /usr/lib/libgdbm.so.3 (0xb74e7000)
      libz.so.1 => /usr/lib/libz.so.1 (0xb74d3000)
      /lib/ld-linux.so.2 (0xb7ef4000)
      libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb74cf000)
      libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xb7465000)
      libpng12.so.0 => /usr/lib/libpng12.so.0 (0xb7441000)
      libart_lgpl_2.so.2 => /usr/lib/libart_lgpl_2.so.2 (0xb742b000)
      libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7406000)


Then we go to snort (maybe you have nothing to do with it...):

#./configure --enable-dynamicplugin

#export LDFLAGS="-L/usr/local/lib -lpcap"

Compiling seems successfull... But the execution takes up to 100% CPU 
and 85% RAM... And again, executing ldd against the binary: where is the 
libpcap linking option?? Any ideas??
# ldd /usr/local/bin/snort
      linux-gate.so.1 =>  (0xffffe000)
      libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7f03000)
      libpcre.so.3 => /usr/lib/libpcre.so.3 (0xb7edd000)
      libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7eb7000)
      libnsl.so.1 => /lib/tls/i686/cmov/libnsl.so.1 (0xb7ea1000)
      libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e9d000)
      libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d6c000)
      /lib/ld-linux.so.2 (0xb7f1f000)

Also, executing any of theses binaries gives some strange messages 
(dmesg), any ideas??

[PF_RING] successfully allocated 50925568 bytes at 0xf8afd000
[PF_RING] allocated 32770 slots [slot_len=1554][tot_mem=50925568]
[PF_RING] removed /proc/net/pf_ring/25491-lo.0      --> STRANGE!!!
[PF_RING] successfully allocated 50925568 bytes at 0xf8afd000
[PF_RING] allocated 32770 slots [slot_len=1554][tot_mem=50925568]
[PF_RING] removed /proc/net/pf_ring/25491-eth5.1
[PF_RING] successfully allocated 50925568 bytes at 0xf8afd000
[PF_RING] allocated 32770 slots [slot_len=1554][tot_mem=50925568]
[PF_RING] removed /proc/net/pf_ring/25491-eth0.2
[PF_RING] removed /proc/net/pf_ring/25491.0

Then we go to our tests (previously done with the standard Debian 
packages and kernel), and there is very little improvement: We have 
changed the module parameters num_slots: 8192 and 32768

These are our results:

*Tests*         Dropped by Libpcap
Standard Sockets        PF_ring: 8192   PF_Ring: 32768
*tcpreplay -C -t -l 60 -i eth0 capt_50MB (485.37 Mbps/sec)* 	189,50% 
118,00% 	130,00%
*tcpreplay -C -t -l 200 -i eth0 capt_50MB ()*   151,90%         106,00%         110,00%
*tcpreplay -C -M 20 -l 10 -i eth0 capt_50MB ()*         0,20%   0,90%   0,00%
*tcpreplay -C -M 70 -l 20 -i eth0 capt_50MB ()*         15,80%  3,70%   2,10%
*tcpreplay -C -t -l 60 -i eth0 capt_50MB ()*    289,20%         171,30%         161,10%
*tcpreplay -C -M 0.5 -i eth0 capt_50MB ()*      0,00%   0,00%   0,00%




Something to do with the BFP filters?
Now, do you think this is enough?
We did nothing about RT_IRQ and real-time patches to the kernel, should we?
Should we go and do the same on the production environment? Should it go 
better just for having a better NIC and the driver support (intel)?
Any change to these tests we should make?

Thanx a lot just for reading...

Regards

Jorge Cuevas

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users