Re: snort.stats key-value mapping (Solved)

[Karolis <karolis.cepulis () gmail com>]

Hi Juan,

On the snort.stats change date in dpkg.log found this: "upgrade
securityonion-snort 2.9.5.6-0ubuntu0securityonion1
2.9.7.0-0ubuntu0securityonion4"

Statistics prior upgrade:
#time,pkt_drop_percent,wire_mbits_per_sec.realtime,alerts_per_second,kpackets_wire_per_sec.realtime,avg_bytes_per_wire_packet,patmatch_percent,syns_per_second,synacks_per_second,new_sessions_per_second,deleted_sessions_per_second,total_sessions,max_sessions,stream_flushes_per_second,stream_faults,stream_timeouts,frag_creates_per_second,frag_completes_per_second,frag_inserts_per_second,frag_deletes_per_second,frag_autofrees_per_second,frag_flushes_per_second,current_frags,max_frags,frag_timeouts,frag_faults,iCPUs,usr[0],sys[0],idle[0],wire_mbits_per_sec.realtime,ipfrag_mbits_per_sec.realtime,ipreass_mbits_per_sec.realtime,rebuilt_mbits_per_sec.realtime,mbits_per_sec.realtime,avg_bytes_per_wire_packet,avg_bytes_per_ipfrag_packet,avg_bytes_per_ipreass_packet,avg_bytes_per_rebuilt_packet,avg_bytes_per_packet,kpackets_wire_per_sec.realtime,kpackets_ipfrag_per_sec.realtime,kpackets_ipreass_per_sec.realtime,kpackets_rebuilt_per_sec.realtime,kpackets_per_sec.realtime,pkt_stats.pkts_recv,pkt_stats.pkts_drop,total_blocked_packets,new_udp_sessions_per_second,deleted_udp_sessions_per_second,total_udp_sessions,max_udp_sessions,max_tcp_sessions_interval,curr_tcp_sessions_initializing,curr_tcp_sessions_established,curr_tcp_sessions_closing,tcp_sessions_midstream_per_second,tcp_sessions_closed_per_second,tcp_sessions_timedout_per_second,tcp_sessions_pruned_per_second,tcp_sessions_dropped_async_per_second,current_attribute_hosts,attribute_table_reloads,mpls_mbits_per_sec.realtime,avg_bytes_per_mpls_packet,kpackets_per_sec_mpls.realtime,total_tcp_filtered_packets,total_udp_filtered_packets,ip4::trim,ip4::tos,ip4::df,ip4::rf,ip4::ttl,ip4::opts,icmp4::echo,ip6::ttl,ip6::opts,icmp6::echo,tcp::syn_opt,tcp::opt,tcp::pad,tcp::rsv,tcp::ns,tcp::urg,tcp::urp,tcp::trim,tcp::ecn_pkt,tcp::ecn_ssn,tcp::ts_ecr,tcp::ts_nop,tcp::ips_data,tcp::block,total_injected_packets,frag3_mem_in_use,stream5_mem_in_use,total_alerts_per_second

Statistics  after upgrade:
#time,pkt_drop_percent,wire_mbits_per_sec.realtime,alerts_per_second,kpackets_wire_per_sec.realtime,avg_bytes_per_wire_packet,patmatch_percent,syns_per_second,synacks_per_second,new_sessions_per_second,deleted_sessions_per_second,total_sessions,max_sessions,stream_flushes_per_second,stream_faults,stream_timeouts,frag_creates_per_second,frag_completes_per_second,frag_inserts_per_second,frag_deletes_per_second,frag_autofrees_per_second,frag_flushes_per_second,current_frags,max_frags,frag_timeouts,frag_faults,iCPUs,usr[0],sys[0],idle[0],wire_mbits_per_sec.realtime,ipfrag_mbits_per_sec.realtime,ipreass_mbits_per_sec.realtime,rebuilt_mbits_per_sec.realtime,mbits_per_sec.realtime,avg_bytes_per_wire_packet,avg_bytes_per_ipfrag_packet,avg_bytes_per_ipreass_packet,avg_bytes_per_rebuilt_packet,avg_bytes_per_packet,kpackets_wire_per_sec.realtime,kpackets_ipfrag_per_sec.realtime,kpackets_ipreass_per_sec.realtime,kpackets_rebuilt_per_sec.realtime,kpackets_per_sec.realtime,pkt_stats.pkts_recv,pkt_stats.pkts_drop,total_blocked_verdicts,new_udp_sessions_per_second,deleted_udp_sessions_per_second,total_udp_sessions,max_udp_sessions,max_tcp_sessions_interval,curr_tcp_sessions_initializing,curr_tcp_sessions_established,curr_tcp_sessions_closing,tcp_sessions_midstream_per_second,tcp_sessions_closed_per_second,tcp_sessions_timedout_per_second,tcp_sessions_pruned_per_second,tcp_sessions_dropped_async_per_second,current_attribute_hosts,attribute_table_reloads,mpls_mbits_per_sec.realtime,avg_bytes_per_mpls_packet,kpackets_per_sec_mpls.realtime,total_tcp_filtered_packets,total_udp_filtered_packets,num_normalizations,ip4::trim,ip4::tos,ip4::df,ip4::rf,ip4::ttl,ip4::opts,icmp4::echo,ip6::ttl,ip6::opts,icmp6::echo,tcp::syn_opt,tcp::opt,tcp::pad,tcp::rsv,tcp::ns,tcp::urp,tcp::ecn_pkt,tcp::ecn_ssn,tcp::ts_ecr,tcp::ts_nop,tcp::ips_data,tcp::block,tcp::req_urg,tcp::req_pay,tcp::req_urp,tcp::trim_syn,tcp::trim_rst,tcp::trim_win,tcp::trim_mss,would_ip4::trim,would_ip4::tos,would_ip4::df,would_ip4::rf,would_ip4::ttl,would_ip4::opts,would_icmp4::echo,would_ip6::ttl,would_ip6::opts,would_icmp6::echo,would_tcp::syn_opt,would_tcp::opt,would_tcp::pad,would_tcp::rsv,would_tcp::ns,would_tcp::urp,would_tcp::ecn_pkt,would_tcp::ecn_ssn,would_tcp::ts_ecr,would_tcp::ts_nop,would_tcp::ips_data,would_tcp::block,would_tcp::req_urg,would_tcp::req_pay,would_tcp::req_urp,would_tcp::trim_syn,would_tcp::trim_rst,would_tcp::trim_win,would_tcp::trim_mss,total_injected_packets,frag3_mem_in_use,stream5_mem_in_use,total_alerts_per_second

Karolis

On Tue, Apr 14, 2015 at 10:22 AM, Juan Jesus Prieto <jjprieto () redborder org>
wrote:

>  Hi Karolis,
>
>   What version of snort are you testing? I would like to check the source
> code for the perfmonitor preprocessor.
>
> Regrads.
>
>
> El lun, 13-04-2015 a las 20:29 +0300, Karolis escribió:
>
> Hi Juan,
>
>
>
>  I have found  the root cause of the problem. Snort all the time
> outputted key-value pairs correctly.  It seems that snort upgrade changed
> the number of statistics monitored. I have formed the array exactly as you
> are "head'ed" the keys and "tail'ed" the latest values whats why they do
> not correlate anymore. I will modify the script so it reads keys from the
> end of the file to avoid such problems in the future.
>
>
>
>  Karolis
>
>
>
>
>  On Mon, Apr 13, 2015 at 10:59 AM, Juan Jesus Prieto <
> jjprieto () redborder org> wrote:
>
>  Hi Karolis,
>
>   Could you attach a stat file content example? every key should be
> accompanied with their corresponding value, one on one.
>
> Regards.
>
>
>
> El jue, 09-04-2015 a las 19:46 +0300, Karolis escribió:
>
> Hi Juan,
>
>
> Thanks for reply. I have got same associative array but can I rely on it?
> As I mentioned there are 96 keys and 131 values in the snort.stats file.
> How do you know that first 96 keys correspond to the first 96 values
> on one to one relationship and only the last values misses keys?
> Can it be what there are gaps in key value pairs eg. key 10 correspond to
> value 12?
>
>
> Karolis
>
>
>
>
>
> On Mon, Apr 6, 2015 at 11:14 AM, Juan Jesus Prieto <jjprieto () redborder org>
> wrote:
>
> Hi Karolis,
>
>   The manual is out-of-date at this point. I use scripting for dinamically
> map this pairs. For example:
>
> # declare -A v; \
> keys=( $(head /var/log/snort/snort.stats -n2 | tail -n1 | sed 's/^#//' | tr ',' ' ') ); \
> count=0; \
> for n in $(tail /var/log/snort/snort.stats -n1 | tr ',' ' '); do \
>    v[${keys[$count]}]=$n; \
>    count=$(($count+1)); \
>    done; \
> echo "stream5_mem_in_use: ${v['stream5_mem_in_use']}"; \
> echo "curr_tcp_sessions_established: ${v['curr_tcp_sessions_established']}"
> stream5_mem_in_use: 13950060
> curr_tcp_sessions_established: 5195
>
>
>
> This small script will map into a hash (named 'v') all pairs key/value and
> present last values from stats file (stream5_mem_in_use and
> curr_tcp_sessions_established in this example).
>
> Another option is to use my snmp passthrou agent:
>
> https://github.com/redBorder/rb_snmp_pass
>
> You will need to adapt it for your case.
>
>
> El mar, 31-03-2015 a las 10:03 +0300, Karolis escribió:
>
> Hi,
>
> I am trying to map perfmonitor preprocessors statistics keys to values.
>
>
> config:
> preprocessor perfmonitor: time 300 file
> /nsm/sensor_data/"sensor-name"/snort.stats pktcnt 10000
>
>
>
> snort manual   <http://manual.snort.org/node88.html>states "There are
> over 100 individual statistics included. A header line is output at startup
> and rollover that labels each column." although only 75 keys are listed.
>
>
> snort.stats file has 96 keys  and 131 values.
>
>
> How can I correctly map keys to values?
>
>
> Karolis
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
> by Intel and developed in partnership with Slashdot Media, is your hub for all
> things parallel software development, from weekly thought leadership blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to 
> this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users 
> list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org 
> to stay current on all the latest Snort news!
>
>
>
>
>
> ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live
> exercises
> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
> event?utm_
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>
> Please visit http://blog.snort.org to stay current on all the latest
> Snort news!
>
>
>
> ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live 
> exerciseshttp://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
> _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to 
> this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users 
> list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org 
> to stay current on all the latest Snort news!
>
>
>
>
>
> ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live
> exercises
> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
> event?utm_
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>
> Please visit http://blog.snort.org to stay current on all the latest
> Snort news!
>
>
>
>  ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live 
> exerciseshttp://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
> _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to 
> this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users 
> list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org 
> to stay current on all the latest Snort news!
>
>
>
>
> ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live
> exercises
> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
> event?utm_
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe:
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive:
> http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users
>
> Please visit http://blog.snort.org to stay current on all the latest
> Snort news!
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!