Re: Snort spikes to 100% CPU followed by network latency

["Russ Combs (rucombs)" <rucombs () cisco com>]

________________________________
From: Cody Brugh [cbrugh () gmail com]
Sent: Friday, May 23, 2014 10:06 AM
To: Russ Combs (rucombs)
Subject: Re: [Snort-users] Snort spikes to 100% CPU followed by network latency

Russ,

So PPM will control how much time snort can spend on a packet and in result will hopefully stop snort from chewing up 
100% CPU and causing network latency, correct?

* It can help, but the checks are 'passive', ie between processing steps, so any one step can take excessively long.  
The key thing is that it might trigger an alert which would help track down the root cause.

On Fri, May 23, 2014 at 9:40 AM, Russ Combs (rucombs) <rucombs () cisco com<mailto:rucombs () cisco com>> wrote:

________________________________
From: Cody Brugh [cbrugh () gmail com<mailto:cbrugh () gmail com>]
Sent: Friday, May 23, 2014 8:54 AM
To: Russ Combs (rucombs)
Cc: snort-users () lists sourceforge net<mailto:snort-users () lists sourceforge net>
Subject: Re: [Snort-users] Snort spikes to 100% CPU followed by network latency

Russ,

We did not change anything before this started happening... I did upgrade snort to the latest version once this 
happened but it continues to spike even after upgrading.

This is starting to happen about every 2 or 3 days now... I've had a couple times where it fixes itself and snort CPU 
usage goes down.  Other times it goes on for a long period and I end up killing the snort process.

I already have the preprocessor sensitive_data commented out, however I do see this in my disablesid.conf (I used 
pulled pork to fetch VRT rules).

pcre:fwsam
pcre:MS\d{2}-\d*
pcre:dce_iface

Should I adjust the PCRE stuff maybe?

* Can't comment on the above but the issue with pcre rule options that have /O is that they override the match limits 
that are configured and can therefore chew on a packet as long as needed.  You could check for such rules and comment 
them out.

Did you enable PPM?  That can both help limit your total maximum latency and catch the packets that trigger the problem.

Thanks!


On Fri, May 23, 2014 at 8:09 AM, Russ Combs (rucombs) <rucombs () cisco com<mailto:rucombs () cisco com>> wrote:

________________________________
From: Cody Brugh [cbrugh () gmail com<mailto:cbrugh () gmail com>]
Sent: Thursday, May 22, 2014 8:13 PM
To: snort-users () lists sourceforge net<mailto:snort-users () lists sourceforge net>
Subject: [Snort-users] Snort spikes to 100% CPU followed by network latency

Hello,

We have been running snort in-line for over a year now with no issues in terms of latency or CPU usage.  Recently (over 
the past month) snort will all of the sudden spike CPU usage up to 100% and network latency becomes real bad, 1000+ms.

I am really not sure where to start on figuring out what is causing this.  I am starting snort so it prints the 
alerts/drops on the console and don't see any specific rule that would be causing this.

Any advise on this issue?

* Did you change your Snort version or configuration around the time you started seeing the issue?  How frequently does 
this occur?  And when it happens does it resolve itself or do you restart or what?

You can turn on PPM (config ppm ...) and enable the PPM rules (gid 134).  That may catch the problem packet which you 
can log and examine for clues.

Without any clues I'd first check for SDF and PCRE.  If you have SDF (preprocessor sensitive_data) configured you can 
try commenting that out.  If you have any pcre/O rules (PCRE override) you can try commenting those out too.

Snort OS: CentOS, 64-bit

  o"  )~   Version 2.9.6.1 GRE (Build 56)
   ''''    By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
           Copyright (C) 2014 Cisco and/or its affiliates. All rights reserved.
           Copyright (C) 1998-2013 Sourcefire, Inc., et al.
           Using libpcap version 1.0.0
           Using PCRE version: 7.8 2008-09-05
           Using ZLIB version: 1.2.3

DAQ version: 2.0.2

Thanks!


------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!