Re: Performance issue in 2.9.8.0

[Hui cao <huica () cisco com>]

Hi Philip,

Thanks for the information! We want to know why this happens. Can you 
enable performance profiling when this happens? Here are the lines to 
add to your snort config. If you can run both versions, that will be great!

config profile_rules: print all, sort total_ticks
config profile_preprocs: print 100, sort total_ticks

Best,
Hui.

On 02/04/2016 08:55 AM, Phillip Deneault wrote:
> Hello,
>
> I've noticed that the memory profile and performance of snort 2.9.8.0 
> has drastically changed in 2.9.8.0 over 2.9.7.6.  I run a large 
> PF_RING enabled sensor running 20 sessions of snort in 'AC' mode (to 
> maximize performance) and each snort session now seems to consume 
> nearly 4 times the memory of the previous version.  After running the 
> two on the same sample file, I found drastic differences.
>
> I've attached the full runs of each to this message, but to summarize 
> (two runs on the same sample pcap).
>
> 2.9.7.6 <http://2.9.7.6>:
>
> [ Port Based Pattern Matching Memory ]
>
> +- [ Aho-Corasick Summary ] -------------------------------------
>
> | Storage Format : Full-Q
>
> | Finite Automaton : DFA
>
> | Alphabet Size : 256 Chars
>
> | Sizeof State : 4 bytes
>
> | Instances : 955
>
> | Characters : 4459913
>
> | States : 3382768
>
> | Transitions : 480562184
>
> | State Density : 55.5%
>
> | Patterns : 333707
>
> | Match States : 374584
>
> | Memory (KB) : -651916.19
>
> | Pattern : 34781.62
>
> | Match Lists : 71479.36
>
> +----------------------------------------------------------------
>
>
> Run time for packet processing was 28.3331 seconds
>
> Snort processed 804001 packets.
>
> Snort ran for 0 days 0 hours 0 minutes 28 seconds
>
> Pkts/sec: 28714
>
>
> Action Stats:
>
> Alerts: 443 ( 0.055%)
>
> Logged: 443 ( 0.055%)
>
>
>
>
> 2.9.8.0 <http://2.9.8.0>:
>
> [ Port Based Pattern Matching Memory ]
>
> +- [ Aho-Corasick Summary ] -------------------------------------
>
> | Storage Format : Full-Q
>
> | Finite Automaton : DFA
>
> | Alphabet Size : 256 Chars
>
> | Sizeof State : 4 bytes
>
> | Instances : 1710
>
> | Characters : 7831490
>
> | States : 5940237
>
> | Transitions : 834278207
>
> | State Density : 54.9%
>
> | Patterns : 590748
>
> | Match States : 660897
>
> | Memory (MB) : 1979.09
>
> | Patterns : 60.01
>
> | Match Lists : 122.54
>
> | DFA : 1795.65
>
> +----------------------------------------------------------------
>
>
> Run time for packet processing was 51.4997 seconds
>
> Snort processed 804001 packets.
>
> Snort ran for 0 days 0 hours 0 minutes 51 seconds
>
> Pkts/sec: 15764
>
>
> Action Stats:
>
> Alerts: 482 ( 0.060%)
>
> Logged: 482 ( 0.060%)
>
>
> So while I concede it did find 40 extra alerts, its created a 
> situation where I can't run as many snort instances without buying 
> significantly more RAM nearly halved my packet throughput for each 
> instance.  If I run my typical configuration with 2.9.8.0 in 
> production, I can't run 4 instances, the CPUs on the remaining 
> instances are pegged, and nearly 40% of my packets fall off my ring 
> unprocessed.  That just doesn't justify 40 more alerts IMHO.
>
> I was going to submit this as a bug, but its not really a bug if the 
> intention was to close potential gaps in the packet processing 
> stream.  I think people just crossed the 80/20 rule.  If someone from 
> SF wants to weigh in publicly or privately that its a bug and I should 
> go down that road, let me know.
>
> I have not repeated this test in other pattern matching modes because 
> the performance curve drops as many packets, if not more, in my 
> production configuration.  Others might have the same findings if they 
> are not running such a saturated link.  There seems to have been 
> tidbits on the mailing list about dropped packets in 2.9.8.0 without 
> much investigation behind it.  I might be mirroring those findings.
>
> Thanks,
>
> Phil
>
>
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
>
>
> _______________________________________________
> Snort-devel mailing list
> Snort-devel () lists sourceforge net
> https://lists.sourceforge.net/lists/listinfo/snort-devel
> Archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!