Snort mailing list archives

Previous By Date Next
Previous By Thread Next

Re: Clarification upon stats

From: Todd Wease <twease () sourcefire com>
Date: Wed, 31 Jul 2013 11:43:30 -0400
Hello Reinoud,

On Wed, Jul 31, 2013 at 4:38 AM, Reinoud Koornstra <sockstat () hotmail com>wrote:


Hi Todd,

Thinking about this, there might be more adjustments needed.
For each rule, meaning OTN, ticks are being maintained as well.
Would these ticks need to be adjusted as well?
I mean only while busy processing (can also mean only reading) an OTN,
we'll get to an RTN by calling getRtnFromOtn.



ruleOTNEvalPerfStats is the cumulation of the all of the time spent
evaluating rules, particularly the rule options.  When a leaf node is
reached, it means there is a match on all of the rule options of an OTN and
then the RTN is evaluated.  detection_option_node_update_otn_stats()
updates the OTN times based on the time it took to evaluate rule options
associated with the OTN.  The fix we are going to go with is to not include
the RTN time so the ticks won't need to be adjusted:

Index: src/detection-plugins/detection_options.c
===================================================================
RCS file:
/usr/cvsroot/sfeng/ims/sfsnort/snort/src/detection-plugins/detection_options.c,v
retrieving revision 1.88
diff -p -u -r1.88 detection_options.c
--- src/detection-plugins/detection_options.c   3 Jul 2013 12:57:58
-0000       1.88
+++ src/detection-plugins/detection_options.c   30 Jul 2013 19:15:27 -0000
@@ -949,6 +949,7 @@ int detection_option_node_evaluate(detec
                     PatternMatchData *pmd = (PatternMatchData
*)eval_data->pmd;
                     int pattern_size = 0;
                     int check_ports = 1;
+                    int eval_rtn_result;
 #ifdef TARGET_BASED
                     unsigned int svc_idx;
 #endif
@@ -983,7 +984,12 @@ int detection_option_node_evaluate(detec
                         }
                     }
 #endif
-                    if (fpEvalRTN(getRuntimeRtnFromOtn(otn), eval_data->p,
check_ports))
+                    // Don't include RTN time
+                    NODE_PROFILE_TMPEND(node);
+                    eval_rtn_result = fpEvalRTN(getRuntimeRtnFromOtn(otn),
eval_data->p, check_ports);
+                    NODE_PROFILE_TMPSTART(node);
+
+                    if (eval_rtn_result)
                     {
                         if ( !otn->detection_filter ||
                              !detection_filter_test(




I noticed you use this function multiple times in profiler.c and assuming
a pointer to a RTN is returned.
Are you sure that every time an valid pointer will be returned?
I mean in the function detection_option_node_evaluate in case of a leaf
node it contains an OTN.
In which case you call getRuntimeRtnFromOtn(otn) which calls getRtnfromOtn
and you check wheter the pointer isn't NULL.



There may not be an RTN if you are using multiple policies via "config
binding".  Looks like there is one call in profiler.c in
ResetRuleProfiling() that should check for a NULL RTN.  This function would
only be called when reading pcaps from the command line and using the
--pcap-reset option.  I'll bug this as well.




Thinking about it.......
Should ruleOTNEvalPerfStats be changed to ruleNODEEvalPerfStats ?
Because only when the node is a leaf node it contains an OTN.
Actually this is how i read it, because only then you cast the pointer to
the option_data of the node to a pointer to the OTN.



I think the name is fine - it's the time spent evaluating rules,
specifically the rule options.  Even if a leaf node isn't reached, meaning
a match on all of the rule options, there was still time spent evaluating
the OTN.



Thanks,

Reinoud.


 ------------------------------
From: sockstat () hotmail com
To: twease () sourcefire com
Date: Wed, 31 Jul 2013 01:08:58 +0000
CC: snort-devel () lists sourceforge net

Subject: Re: [Snort-devel] Clarification upon stats

Hi Todd,

Thanks for your reply.
May I propose a patch to fi the OTN stats?

--- src/profiler.c.orig 2013-07-30 18:03:56.595012379 -0700
+++ src/profiler.c      2013-07-30 18:05:35.779014374 -0700
@@ -85,7 +85,7 @@


 /* Externs
********************************************************************/
-extern PreprocStats mpsePerfStats, rulePerfStats, ncrulePerfStats;
+extern PreprocStats mpsePerfStats, rulePerfStats, ncrulePerfStats,
ruleOTNEvalPerfStats, ruleRTNEvalPerfStats;


 void getTicksPerMicrosec(void)
@@ -926,6 +926,8 @@
     mpsePerfStats.ticks -= rulePerfStats.ticks;
     /* And adjust the rules to include the NC rules */
     rulePerfStats.ticks += ncrulePerfStats.ticks;
+   /* Also adjust the semi otn stats to not include the rtn stats */
+    ruleOTNEvalPerfStats.ticks -= ruleRTNEvalPerfStats.ticks;

     for (layer=0;layer<=max_layers;layer++)
     {


Kind Regards,

Reinoud.

------------------------------
Date: Tue, 30 Jul 2013 09:30:33 -0400
Subject: Re: [Snort-devel] Clarification upon stats
From: twease () sourcefire com
To: sockstat () hotmail com
CC: snort-devel () lists sourceforge net


 Hello Reinoud,

On Wed, Jul 24, 2013 at 6:01 AM, sockstat <sockstat () hotmail com> wrote:

 Hi Everyone,

I wanted to ask about the performance counters that snort maintains. In
the function detection_option_tree_evaluate you start maintaining
ruleOTNEvalPerfStats using a macro. But this actually kicks in for any type
of node while only the leaf nodes seem to be an OTN. It also adds up when a
node type is Flow or pcre etc so maybe it should better be called
ruleNODEEvalPerfStats? In detection_option_tree_evaluate you call
detection_option_node_evaluate which in case of a leaf node gets the RTN
from the OTN and fpEvalRTN is called where you maintain
ruleRTNEvalPerfStats. So in the end screen where you print the stats out it
seems to me that the microseconds for rule tree eval contain the rtn eval
microseconds, because fpEvalRTN is called while counting the clock ticks
for ruleOTNEvalPerfStats. In profile.c you print the amount of microseconds
out. So rule tree eval is not just the ticks spent in
detection_option_tree_evaluate, It has the clock ticks spent in fpEvalRTN
included, while rtn eval is printed out sperately.


Yes, it looks like the RTN stats are counted in the OTN stats.  I'll
create a bug to see if we can get this fixed.



Same thing holds for the mpse stats when __process_queue calls
rule_tree_match, then the ticks spent in detection_oprion_tree_evaluate and
fpEvalRTN are included in the ticks spent in mpseSearch. Hence it seems
there is some overlap in the stats, because I don't see the ticks spent in
fpEvalRTN subtracted from the amount of ticks spent in
detection_option_tree_evaluate for example or did I miss this?


In rule_tree_match(), rulePerfStats are started and in
ShowPreprocProfiles() in profile.c it is subtracted from mpsePerfStats.



Also in case of a core 2duo CPU the tsc isn't stable and hence the
calculation of how many clock ticks occur per microsecond might not be
accurate?
Thanks, Reinoud.



------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!



------------------------------------------------------------------------------
Get your SQL database under version control now! Version control is
standard for application code, but databases havent caught up. So what
steps can you take to put your SQL databases under version control? Why
should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk

_______________________________________________ Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-develPlease visit
http://blog.snort.org for the latest news about Snort!


------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent 
caught up. So what steps can you take to put your SQL databases under 
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!
Previous By Date Next
Previous By Thread Next

Current thread: