Re: Performance monitoring issues

[Lee Saunders <lee.saunders () zynstra com>]

Coming back to this I've temporarily removed all the ET blacklist rules, 
and tuned as far as I could the preprocessor configuration according to 
recommendations to reduce the pattern match values with little success.

The packet volume is low for now but will be expanded to monitoring much 
greater volumes so need to ensure it is correct now. It seems there is a 
lot of rebuilding going on but not really understanding why and how to 
reduce.

Attached is the preprocessor perfmonitor output (snort_pp_profile) - the 
s5TcpProcessRebuilt seems large, and snort_rules_profile (there are some 
high matching rules, and clearly more tuning, but does not explain the 
high pattern match as far as I can see. I've also included the output 
from perfmonitor in snort.stats, the % match is higher than seeing 
previously, normally in range 80 to 190 or so, but this had detect 
anomalies on so not sure if that would be a cause for a large increase.

Although the drops here were for the first period after snort restart, I 
see drops at random periods even with this volume of traffic when snort 
has been running for a while and as such is in a steady state, this is 
the concern leading to deeper analysis.

On 06/09/13 14:20, Lee Saunders wrote:
> Attached document for the preprocessor, this has the below and various 
> other runs included.
>
> As stated doing a performance monitoring run for specific rules and 
> will add those in due course. I have the blacklist rules activated and 
> these are swamping the results so will be removing these to test 
> impact of other rules, and then look to add these back in but running 
> as IDS so do not believe the reputation preprocessor will help from 
> what I've been reading.
>
> However, the two initial questions I have is:
>
> 1) From perfmon preprocessor output, I was seeing around 10% pattern 
> matching, but filtered out a lot of background noise (safe upload 
> packets) and now seeing patmatch_percent values routinely of well over 
> 100% (200 or 300% in some periods)> I don't understand how this is 
> possible - a little in terms of fragmentation and reconstruction from 
> other preprocessors but was not expecting values of this order - is it 
> normal, if not where should I look to mitigate.
>
> 2) As I said the rtn eval looks very high from what I've seen looking 
> around other discussions, is it something to worry about? Indeed is it 
> reporting correctly, it looks likes rule eval has two child modules 
> rule tree eval (with % of parent as 99.66% and rtn eval % of parent as 
> 82.75 which does not make a whole lot of sense as considerably > 100% 
> of caller)
>
> On 06/09/13 13:59, Joel Esler wrote:
> > Not sure what is causing the lines to wrap like that, but it's fairly
> > impossible to read, if you want to attach that as a plaintext word
> > document that would be beneficial, that says, do some rule profiling
> > as well, that way we can see what rules are causing problems, if any.
> >
> > On Fri, Sep 6, 2013 at 6:45 AM, Lee Saunders 
> > <lee.saunders () zynstra com> wrote:
> > > I've been starting performance monitoring on my setup, as there is
> > > unexpected drops appearing at one single installation (virtualized
> > > configuration so traffic profile pretty similar to other installation).
> > >
> > > The bandwidth is pre-filtered so is relatively small, certainly small
> > > enough to that drops are not currently expected. The first perculiarity
> > > is looking at perfmon I see patmatch-percent numbers in the range 70 -
> > > 300%, very high and not what I'd expect.
> > >
> > > Looking then at profile monitoring, I can't find much on how to read 
> > > and
> > > act on the values. I'm currently running a test for the rules 
> > > profiling,
> > > but a preprocessor profile based on total ticks sorting has raised a
> > > couple of queries. The output of a short run is repeated below. 
> > > However,
> > > the rtn eval value, when looked at against other outputs on the web
> > > looks very high, at 622810 ms, but not clear this represents or how to
> > > improve it. There is also the implication from the output that this and
> > > rule tree eval are siblings of rule eval, but the % of parent then does
> > > not add up, with a value of around 170% - is there a known problem with
> > > how these are reported and a red herring?
> > >
> > > I'm at the outset of the tuning exercise, but its proving difficult to
> > > find resources which outline how to interpret these values and how to
> > > mitigate them. I'm assuming the top offending detect is influenced
> > > primarily by the rule definitions hence the rules profiling I'm
> > > currently doing, but some insight into minimizing the subtasks would be
> > > useful, and if any of the other preprocessors can influence.
> > >
> > > timestamp: 1378398556
> > > Preprocessor Profile Statistics (all)
> > > ==========================================================
> > >    Num            Preprocessor Layer     Checks      Exits Microsecs
> > > Avg/Check Pct of Caller Pct of Total
> > >    ===            ============ =====     ======      ===== =========
> > > ========= ============= ============
> > >     1                   detect     0      10553 10553
> > > 873142      82.74         83.64        83.64
> > >      1               rule eval     1      10427 10427
> > > 752623      72.18         86.20        72.09
> > >       1         rule tree eval     2      21935 21935
> > > 750099      34.20         99.66        71.85
> > >        1               content     3       4836 4836
> > > 1449       0.30          0.19         0.14
> > >        2                  pcre     3         19 19
> > > 1438      75.71          0.19         0.14
> > >        3                 flags     3       8970 8970
> > > 656       0.07          0.09         0.06
> > >        4             byte_test     3       2929 2929
> > > 517       0.18          0.07         0.05
> > >        5              dsize_eq     3       3987 3987
> > > 402       0.10          0.05         0.04
> > >        6                  flow     3       2021 2021
> > > 172       0.09          0.02         0.02
> > >        7  preproc_rule_options     3       1256 1256
> > > 88       0.07          0.01         0.01
> > >        8            uricontent     3        168 168
> > > 64       0.38          0.01         0.01
> > >        9                 itype     3        318 318
> > > 40       0.13          0.01         0.00
> > >       10              flowbits     3        212 212
> > > 32       0.15          0.00         0.00
> > >       11                 icode     3        285 285
> > > 14       0.05          0.00         0.00
> > >       12             file_data     3        123 123
> > > 5       0.04          0.00         0.00
> > >       13          byte_extract     3          6 6
> > > 3       0.52          0.00         0.00
> > >       14              isdataat     3          8 8
> > > 1       0.20          0.00         0.00
> > >       15                window     3          4 4
> > > 0       0.15          0.00         0.00
> > >       2               rtn eval     2    1039167 1039167
> > > 622810       0.60         82.75        59.66
> > >      2                    mpse     1      10212 10212
> > > 87608       8.58         10.03         8.39
> > >     2                       s5     0       9321 9321
> > > 86594       9.29          8.29         8.29
> > >      1                   s5tcp     1       8062 6651
> > > 64012       7.94         73.92         6.1310
> > >       1             s5TcpState     2       6651 6651
> > > 46814       7.04         73.13         4.48
> > >        1             s5TcpData     3       1606 1606
> > > 5624       3.50         12.01         0.54
> > >         1       s5TcpPktInsert     4       1406 1406
> > > 4829       3.43         85.86         0.46
> > >        2            s5TcpFlush     3       1037 1037
> > > 2863       2.76          6.12         0.27
> > >         1  s5TcpProcessRebuilt     4        892 892
> > > 50297      56.39       1756.44         4.82
> > >         2     s5TcpBuildPacket     4        892 892
> > > 665       0.75         23.25         0.06
> > >       2           s5TcpNewSess     2        197 197
> > > 1075       5.46          1.68         0.10
> > >     3                      ssl     0       2090 2090
> > > 20935      10.02          2.01         2.01
> > >     4                   decode     0       9853 9853
> > > 20088       2.04          1.92         1.92
> > >     5           sensitive_data     0        162 162
> > > 14964      92.37          1.43         1.43
> > >     6                   eventq     0      20562 20562
> > > 6610       0.32          0.63         0.63
> > >     7                     smtp     0       3748 3748
> > > 5387       1.44          0.52         0.52
> > >     8              httpinspect     0       3782 3782
> > > 5162       1.36          0.49         0.49
> > >     9               DceRpcMain     0       2865 2865
> > > 3518       1.23          0.34         0.34
> > >      1           DceRpcSession     1       2865 2865
> > > 2745       0.96         78.01         0.26
> > >       1       DceRpcNewSession     2       2865 2865
> > > 1584       0.55         57.72         0.15
> > >    10                  perfmon     0      10721 10721
> > > 3119       0.29          0.30         0.30
> > >    11                      ssh     0       2433 2116
> > > 1805       0.74          0.17         0.17
> > >    12                      pop     0       3707 3707
> > > 1434       0.39          0.14         0.14
> > >    13                     imap     0       3707 3707
> > > 1254       0.34          0.12         0.12
> > >    14                      sip     0       3692 3692
> > > 842       0.23          0.08         0.08
> > >    15                   modbus     0       3707 3707
> > > 692       0.19          0.07         0.07
> > >    16                     dnp3     0       1259 1259
> > > 572       0.46          0.05         0.05
> > >    17              backorifice     0       1259 1259
> > > 437       0.35          0.04         0.04
> > >    18                      dns     0        820 820
> > > 164       0.20          0.02         0.02
> > >    total                 total     0       9835       9835 1043957
> > > 106.15          0.00         0.00
> > >
> > > ------------------------------------------------------------------------------ 
> > >
> > > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> > > Discover the easy way to master current and previous Microsoft 
> > > technologies
> > > and advance your career. Get an incredible 1,500+ hours of step-by-step
> > > tutorial videos with LearnDevNow. Subscribe today and save!
> > > http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk 
> > >
> > > _______________________________________________
> > > Snort-users mailing list
> > > Snort-users () lists sourceforge net
> > > Go to this URL to change user options or unsubscribe:
> > > https://lists.sourceforge.net/lists/listinfo/snort-users
> > > Snort-users list archive:
> > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
> > >
> > > Please visit http://blog.snort.org to stay current on all the latest 
> > > Snort news!
>
>
>
> ------------------------------------------------------------------------------
> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
> Discover the easy way to master current and previous Microsoft technologies
> and advance your career. Get an incredible 1,500+ hours of step-by-step
> tutorial videos with LearnDevNow. Subscribe today and save!
> http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk
>
>
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>
> Please visit http://blog.snort.org to stay current on all the latest Snort news!

Attachment: snort_pp_profile
Description:

Attachment: snort_rules_profile
Description:

Attachment: snort.stats
Description:

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!