Re: Snort spikes to 100% CPU followed by network latency

["Russ Combs (rucombs)" <rucombs () cisco com>]

________________________________
From: Cody Brugh [cbrugh () gmail com]
Sent: Wednesday, May 28, 2014 5:47 PM
To: Russ Combs (rucombs)
Cc: snort-users () lists sourceforge net
Subject: Re: [Snort-users] Snort spikes to 100% CPU followed by network latency

Ok, who can examine the core files once I have them?

* If you haven't done so already, it helps to try to (a) minimize the config that reproduces the issue and (b) capture 
a pcap that reproduces the issue.  Both of those are iterative, trial-and-error steps.  By studying the packets from 
the PPM events you may be able to craft a rule that triggers on protocol, addresses, and ports that minimizes the noise 
in the capture.  I would do that before getting a core.  But once you have a core, you can notify snort-devel and they 
can send you upload instructions.



On Wed, May 28, 2014 at 5:43 PM, Russ Combs (rucombs) <rucombs () cisco com<mailto:rucombs () cisco com>> wrote:

________________________________
From: Cody Brugh [cbrugh () gmail com<mailto:cbrugh () gmail com>]
Sent: Wednesday, May 28, 2014 5:40 PM

To: Russ Combs (rucombs)
Cc: snort-users () lists sourceforge net<mailto:snort-users () lists sourceforge net>
Subject: Re: [Snort-users] Snort spikes to 100% CPU followed by network latency

Also note that when we see these CPU/latency spikes we have no alerts or drops that would easily tell us what is 
causing the problem. If it's not a rule what should I start turning off to try eliminate possible causes?  It's 
something that doesn't log or anything.

* Another option you have is to compile with debug and generate a core by sending a sig abort to the Snort process when 
it is in the 100% CPU state.  And you should capture a few to ensure they are in the same basic area.  Then the cores 
need to be examined for clues.


On May 28, 2014, at 5:12 PM, "Russ Combs (rucombs)" <rucombs () cisco com<mailto:rucombs () cisco com>> wrote:


________________________________
From: Cody Brugh [cbrugh () gmail com<mailto:cbrugh () gmail com>]
Sent: Tuesday, May 27, 2014 6:30 PM
To: Russ Combs (rucombs)
Cc: snort-users () lists sourceforge net<mailto:snort-users () lists sourceforge net>
Subject: Re: [Snort-users] Snort spikes to 100% CPU followed by network latency

Russ,

I am still having latency/CPU spike issues even after enabling PPM configuration.... In the below logs where does it 
tell me which rule SIG ID so I can disable the rules that are causing me slowness?

* Actually, it doesn't tell you which rule.  Only which rule tree, which isn't terribly helpful.  That's because the 
rules are compiled into a different form to eliminate redundant checks.

It does tell you that the rule trees that are triggering the PPM events are around 55 usec although you have one shown 
which is higher.  What is your threshold set to?

Also, did you analyze the packets logged with the events to see what type of traffic this is?  You may be able to 
narrow it down and capture a whole session that triggers the problem and go from there.


PPM: Rule-Event Pkt[20570] address=0x0x8200f80 re-enabled 05/27-18:26:27.989936
PPM: Rule-Event Pkt[20570] suspended (72.194.88.31:53484<http://72.194.88.31:53484> -> 
10.2.14.21:80<http://10.2.14.21:80>).
PPM: Rule-Event Pkt[20570] address=0x0x8200f80 used=52.7766 usecs suspended 05/27-18:26:27.989936
PPM: Rule-Event Pkt[30776] address=0x0x3c28fb0 re-enabled 05/27-18:26:36.195293
PPM: Rule-Event Pkt[36115] suspended (10.2.13.17:80<http://10.2.13.17:80> -> 
164.113.217.51:41204<http://164.113.217.51:41204>).
PPM: Rule-Event Pkt[36115] address=0x0x3c28fb0 used=54.8616 usecs suspended 05/27-18:26:40.813898
PPM: Rule-Event Pkt[47155] address=0x0x8200f80 re-enabled 05/27-18:26:48.038002
PPM: Rule-Event Pkt[48185] suspended (66.87.133.205:4163<http://66.87.133.205:4163> -> 10.2.2.4:80<http://10.2.2.4:80>).
PPM: Rule-Event Pkt[48185] address=0x0x8200f80 used=53.992 usecs suspended 05/27-18:26:49.081371
PPM: Rule-Event Pkt[60578] address=0x0x3c28fb0 re-enabled 05/27-18:27:00.867186
PPM: Rule-Event Pkt[71509] suspended (10.2.13.48:80<http://10.2.13.48:80> -> 
96.44.123.180:1046<http://96.44.123.180:1046>).
PPM: Rule-Event Pkt[71509] address=0x0x3c28fb0 used=54.4075 usecs suspended 05/27-18:27:08.522285
PPM: Rule-Event Pkt[72989] address=0x0x8200f80 re-enabled 05/27-18:27:09.766234
PPM: Rule-Event Pkt[76364] suspended (10.2.13.1:35718<http://10.2.13.1:35718> -> 
66.135.58.62:80<http://66.135.58.62:80>).
PPM: Rule-Event Pkt[76364] address=0x0x8200f80 used=76.2302 usecs suspended 05/27-18:27:15.130825
PPM: Rule-Event Pkt[77634] suspended (66.8.180.174:64983<http://66.8.180.174:64983> -> 
10.2.13.17:80<http://10.2.13.17:80>).
PPM: Rule-Event Pkt[77634] address=0x0x820c0b0 used=53.126 usecs suspended 05/27-18:27:17.180899




On Fri, May 23, 2014 at 8:09 AM, Russ Combs (rucombs) <rucombs () cisco com<mailto:rucombs () cisco com>> wrote:

________________________________
From: Cody Brugh [cbrugh () gmail com<mailto:cbrugh () gmail com>]
Sent: Thursday, May 22, 2014 8:13 PM
To: snort-users () lists sourceforge net<mailto:snort-users () lists sourceforge net>
Subject: [Snort-users] Snort spikes to 100% CPU followed by network latency

Hello,

We have been running snort in-line for over a year now with no issues in terms of latency or CPU usage.  Recently (over 
the past month) snort will all of the sudden spike CPU usage up to 100% and network latency becomes real bad, 1000+ms.

I am really not sure where to start on figuring out what is causing this.  I am starting snort so it prints the 
alerts/drops on the console and don't see any specific rule that would be causing this.

Any advise on this issue?

* Did you change your Snort version or configuration around the time you started seeing the issue?  How frequently does 
this occur?  And when it happens does it resolve itself or do you restart or what?

You can turn on PPM (config ppm ...) and enable the PPM rules (gid 134).  That may catch the problem packet which you 
can log and examine for clues.

Without any clues I'd first check for SDF and PCRE.  If you have SDF (preprocessor sensitive_data) configured you can 
try commenting that out.  If you have any pcre/O rules (PCRE override) you can try commenting those out too.

Snort OS: CentOS, 64-bit

  o"  )~   Version 2.9.6.1 GRE (Build 56)
   ''''    By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
           Copyright (C) 2014 Cisco and/or its affiliates. All rights reserved.
           Copyright (C) 1998-2013 Sourcefire, Inc., et al.
           Using libpcap version 1.0.0
           Using PCRE version: 7.8 2008-09-05
           Using ZLIB version: 1.2.3

DAQ version: 2.0.2

Thanks!


------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!