Snort mailing list archives

Previous By Date Next
Previous By Thread Next

Re: Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

From: "Hui Cao (huica)" <huica () cisco com>
Date: Thu, 4 Jun 2015 15:23:57 +0000
Try 

Assume snort pid is 1222

gdb /path/to/snort 1222

Best,
Hui.
On 6/4/15, 10:37 AM, "elof () sentor se" <elof () sentor se> wrote:



An update:

On a sensor where snort crashed with signal 6 three times, I downgraded
daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem
to 
be in 'snort' or 'daq'.

With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.


This make me believe that there's something wrong in snort 2.9.7.3 and
not in daq 2.0.5.



On this sensor I have now done the opposite, upgraded daq to 2.0.5 and
downgraded snort to 2.9.7.2 to see if I get any more signal 6.

On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
2.0.5 
without chroot and uid/gid change, i.e. running as root, in order to
create a core file, if the problem happen again.
(if it doesn't happen on this sensor, I guess the problem lies somewhere
in the chrooting code in snort. I know it has been updated between
2.9.7.2 
and 2.9.7.3)



Russ C also wrote:

Elof - since this is happening frequently, you could try attaching the
debugger to one of your Snort processes and wait wait for segfault.


I know too little about debugging. :-/ Can you give me instructions or
point me to a guide that describes the steps I should take?



/Elof


On Thu, 4 Jun 2015, elof () sentor se wrote:



Five different sensors have now had bus errors (signal 10), segmentation
faults (signal 11) and even signal 6 (SIGABRT).

My snort config uses both chroot and dropping user privileges, so even
if
I start out as root with ulimit unlimited, this doesn't seem to be in
effect
after the chroot/uid-change.

So currently I have no core-file to debug. :-/

Anyone know how to set the ulimits for a chrooted and uid/gid-changed
process in FreeBSD?

/Elof


On Thu, 4 Jun 2015, elof () sentor se wrote:



Hi Hui!

Yes, the dynamic engine/preproc files are updated as well.

Last night the problem reocurred, so this seem to be reproduceable.
Good.
Then there's a good chance this problem can be sorted out.


A few minutes ago a signal 10 happened on another sensor (running
FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort
2.9.7.3 and not in the hardware nor in FreeBSD.


I will compile a debug-snort and try to generate core files.
I'll let you know the outcome next week.

/Elof


On Wed, 3 Jun 2015, Hui cao wrote:


Hi Elof,

Are snort and snort dynamic preprocessors are in sync?

If so, can you help us get a backtrace from the crush? You need
1)  build snort with ./configure --enable-debug
2)  allowing core dump (ulimit -c unlimited)
3) run the snort
4) use "gdb snort core_file " and them type "bt" in the gdb command
line

Best,
Hui.


On 06/03/2015 05:51 AM, elof () sentor se wrote:

Hi all!

This is just a report to inform that after I updated snort and DAQ
to the
latest versions, one of my sensors started throwing signal 10 (bus
error)
and signal 11 (segmentation fault).

# uptime
11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
# dmesg | grep snort
pid 1183 (snort), uid 100: exited on signal 11
pid 16920 (snort), uid 100: exited on signal 11
pid 17502 (snort), uid 100: exited on signal 11
pid 18862 (snort), uid 100: exited on signal 11
pid 20223 (snort), uid 100: exited on signal 11
pid 20927 (snort), uid 100: exited on signal 11
pid 1193 (snort), uid 100: exited on signal 11
pid 2447 (snort), uid 100: exited on signal 11
pid 3811 (snort), uid 100: exited on signal 10
pid 7881 (snort), uid 100: exited on signal 11
pid 9252 (snort), uid 100: exited on signal 10
pid 25593 (snort), uid 100: exited on signal 11
pid 26627 (snort), uid 100: exited on signal 11
pid 56658 (snort), uid 100: exited on signal 11
pid 57237 (snort), uid 100: exited on signal 10
pid 58595 (snort), uid 100: exited on signal 11
pid 68639 (snort), uid 100: exited on signal 11
pid 70008 (snort), uid 100: exited on signal 11
pid 71361 (snort), uid 100: exited on signal 10
pid 72725 (snort), uid 100: exited on signal 11

20 crashes in a day...
A reboot didn't help.

This sensor has never behaved like this during its lifetime (1 year).




FreeBSD 9.3 amd64

     ,,_     -*> Snort! <*-
    o"  )~   Version 2.9.7.3 (Build 217)
     ''''    By Martin Roesch & The Snort Team:
http://www.snort.org/contact#team
             Copyright (C) 2014-2015 Cisco and/or its affiliates.
All rights
reserved.
             Copyright (C) 1998-2013 Sourcefire, Inc., et al.
             Using libpcap version 1.4.0
             Using PCRE version: 8.37 2015-04-28
             Using ZLIB version: 1.2.8

daq-2.0.5



Bus errors are quite unusual in general, so I'll keep looking at
this,
trying to see if it is e.g. paging errors.
It doesn't look like it though:
# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap   4194300        0  4194300     0%

The machine doesn't seem to be overheated either:
System Temp:      30 degrees C
Peripheral Temp: 40 degrees C
CPU Temp: Low


If you need me to do something special to debug this further, let me
know.


PS. It is only one sensor, out of 20, that behaves like this. So
perhaps
it is something in the mirrored traffic that make DAQ or snort point
at
illegal memory addresses and crash.
Or this particular machine is having hardware issues. However, it is
strange that those hw-issues should suddenly start right after I
updated
the software on the machine...

When I write this, the current snort process has been alive for 5
hours.
It's going to be interesting to see if the traffic tonight will
cause it
to crash many times again.

/Elof


----------------------------------------------------------------------
--------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!




-----------------------------------------------------------------------
-------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!




------------------------------------------------------------------------
------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!




-------------------------------------------------------------------------
-----
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!




------------------------------------------------------------------------------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!
Previous By Date Next
Previous By Thread Next

Current thread: