Re: Snort spikes to 100% CPU followed by network latency

[Cody Brugh <cbrugh () gmail com>]

Also note that when we see these CPU/latency spikes we have no alerts or drops that would easily tell us what is 
causing the problem. If it's not a rule what should I start turning off to try eliminate possible causes?  It's 
something that doesn't log or anything. 


> On May 28, 2014, at 5:12 PM, "Russ Combs (rucombs)" <rucombs () cisco com> wrote:
>
>
> From: Cody Brugh [cbrugh () gmail com]
> Sent: Tuesday, May 27, 2014 6:30 PM
> To: Russ Combs (rucombs)
> Cc: snort-users () lists sourceforge net
> Subject: Re: [Snort-users] Snort spikes to 100% CPU followed by network latency
>
> Russ,
>
> I am still having latency/CPU spike issues even after enabling PPM configuration.... In the below logs where does it 
> tell me which rule SIG ID so I can disable the rules that are causing me slowness?
>
> * Actually, it doesn't tell you which rule.  Only which rule tree, which isn't terribly helpful.  That's because the 
> rules are compiled into a different form to eliminate redundant checks.
>
> It does tell you that the rule trees that are triggering the PPM events are around 55 usec although you have one 
> shown which is higher.  What is your threshold set to?
>
> Also, did you analyze the packets logged with the events to see what type of traffic this is?  You may be able to 
> narrow it down and capture a whole session that triggers the problem and go from there.
>
>
> PPM: Rule-Event Pkt[20570] address=0x0x8200f80 re-enabled 05/27-18:26:27.989936 
> PPM: Rule-Event Pkt[20570] suspended (72.194.88.31:53484 -> 10.2.14.21:80).
> PPM: Rule-Event Pkt[20570] address=0x0x8200f80 used=52.7766 usecs suspended 05/27-18:26:27.989936 
> PPM: Rule-Event Pkt[30776] address=0x0x3c28fb0 re-enabled 05/27-18:26:36.195293 
> PPM: Rule-Event Pkt[36115] suspended (10.2.13.17:80 -> 164.113.217.51:41204).
> PPM: Rule-Event Pkt[36115] address=0x0x3c28fb0 used=54.8616 usecs suspended 05/27-18:26:40.813898 
> PPM: Rule-Event Pkt[47155] address=0x0x8200f80 re-enabled 05/27-18:26:48.038002 
> PPM: Rule-Event Pkt[48185] suspended (66.87.133.205:4163 -> 10.2.2.4:80).
> PPM: Rule-Event Pkt[48185] address=0x0x8200f80 used=53.992 usecs suspended 05/27-18:26:49.081371 
> PPM: Rule-Event Pkt[60578] address=0x0x3c28fb0 re-enabled 05/27-18:27:00.867186 
> PPM: Rule-Event Pkt[71509] suspended (10.2.13.48:80 -> 96.44.123.180:1046).
> PPM: Rule-Event Pkt[71509] address=0x0x3c28fb0 used=54.4075 usecs suspended 05/27-18:27:08.522285 
> PPM: Rule-Event Pkt[72989] address=0x0x8200f80 re-enabled 05/27-18:27:09.766234 
> PPM: Rule-Event Pkt[76364] suspended (10.2.13.1:35718 -> 66.135.58.62:80).
> PPM: Rule-Event Pkt[76364] address=0x0x8200f80 used=76.2302 usecs suspended 05/27-18:27:15.130825 
> PPM: Rule-Event Pkt[77634] suspended (66.8.180.174:64983 -> 10.2.13.17:80).
> PPM: Rule-Event Pkt[77634] address=0x0x820c0b0 used=53.126 usecs suspended 05/27-18:27:17.180899 
>
>
>
>
> > On Fri, May 23, 2014 at 8:09 AM, Russ Combs (rucombs) <rucombs () cisco com> wrote:
> >
> > From: Cody Brugh [cbrugh () gmail com]
> > Sent: Thursday, May 22, 2014 8:13 PM
> > To: snort-users () lists sourceforge net
> > Subject: [Snort-users] Snort spikes to 100% CPU followed by network latency
> >
> > Hello,
> >
> > We have been running snort in-line for over a year now with no issues in terms of latency or CPU usage.  Recently 
> > (over the past month) snort will all of the sudden spike CPU usage up to 100% and network latency becomes real bad, 
> > 1000+ms.
> >
> > I am really not sure where to start on figuring out what is causing this.  I am starting snort so it prints the 
> > alerts/drops on the console and don't see any specific rule that would be causing this.
> >
> > Any advise on this issue?
> >
> > * Did you change your Snort version or configuration around the time you started seeing the issue?  How frequently 
> > does this occur?  And when it happens does it resolve itself or do you restart or what?
> >
> > You can turn on PPM (config ppm ...) and enable the PPM rules (gid 134).  That may catch the problem packet which 
> > you can log and examine for clues.
> >
> > Without any clues I'd first check for SDF and PCRE.  If you have SDF (preprocessor sensitive_data) configured you 
> > can try commenting that out.  If you have any pcre/O rules (PCRE override) you can try commenting those out too.
> >
> > Snort OS: CentOS, 64-bit
> >
> >   o"  )~   Version 2.9.6.1 GRE (Build 56) 
> >    ''''    By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
> >            Copyright (C) 2014 Cisco and/or its affiliates. All rights reserved.
> >            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
> >            Using libpcap version 1.0.0
> >            Using PCRE version: 7.8 2008-09-05
> >            Using ZLIB version: 1.2.3
> >
> > DAQ version: 2.0.2
> >
> > Thanks!
------------------------------------------------------------------------------
Time is money. Stop wasting it! Get your web API in 5 minutes.
www.restlet.com/download
http://p.sf.net/sfu/restlet

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!