Snort mailing list archives

Previous By Date Next
Previous By Thread Next

Re: Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

From: elof () sentor se
Date: Thu, 4 Jun 2015 18:10:28 +0200 (CEST)

So I just had a signal 6...
I assume I can't attach files to the mailing list, so here it is, directly in the mailbody. :-) 





gdb /usr/local/bin/snort 11057

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Attaching to program: /usr/local/bin/snort, process 11057
Reading symbols from /usr/local/lib/libdnet.so.1...done.
Loaded symbols for /usr/local/lib/libdnet.so.1
Reading symbols from /usr/local/lib/libpcre.so.1...done.
Loaded symbols for /usr/local/lib/libpcre.so.1
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libcrypto.so.6...done.
Loaded symbols for /lib/libcrypto.so.6
Reading symbols from /lib/libpcap.so.8...done.
Loaded symbols for /lib/libpcap.so.8
Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
Loaded symbols for /usr/local/lib/libsfbpf.so.0
Reading symbols from /lib/libz.so.6...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /usr/lib/liblzma.so.5...done.
Loaded symbols for /usr/lib/liblzma.so.5
Reading symbols from /lib/libthr.so.3...done.
[New Thread 815a59400 (LWP 100459/snort)]
[New Thread 802407400 (LWP 100375/snort)]
Loaded symbols for /lib/libthr.so.3
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
Reading symbols from /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
Loaded symbols for /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
[Switching to Thread 815a59400 (LWP 100459/snort)]
0x0000000801faa40c in nanosleep () from /lib/libc.so.7
(gdb) set logging file gdb-snort.txt
(gdb) set logging on
Copying output to gdb-snort.txt.
(gdb) continue
Continuing.



<...it has just been a few minutes when I receive a SIGABRT>



Program received signal SIGABRT, Aborted.
[Switching to Thread 802407400 (LWP 100375/snort)]
0x0000000801f2364c in thr_kill () from /lib/libc.so.7

(gdb) backtrace full
#0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
No symbol table info available.
#2  0x0000000801fab315 in __assert () from /lib/libc.so.7
No symbol table info available.
#3  0x0000000805068395 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
No symbol table info available.
#4  0x0000000805068781 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
No symbol table info available.
#5  0x000000080506afd0 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
No symbol table info available.
#6  0x000000080506b85b in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
No symbol table info available.
#7  0x000000080506c150 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
No symbol table info available.
#8  0x000000080506cb27 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
No symbol table info available.
#9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00, policy_id=0, policy=0x802faa000) at detect.c:136
        scb = (SessionControlBlock *) 0x8a1aad2f0
        ppn = (PreprocEvalFuncNode *) 0x8033ff0a0
        pps_enabled_foo = 1123336
        alerts_processed = true
#10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
        retval = 0
        policy_id = 0
        policy = (SnortPolicy *) 0x802faa000
        pktcnt = 0
        snort_ticks_start = 34413886976
        snort_ticks_end = 34413888664
#11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50, st=0x80811cfa8, bytes=3644, p=0x8033a4900, 
sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
        tmp_do_detect = 1
        tmp_do_detect_content = 1
        snort_ticks_start = 37073416192
        snort_ticks_end = 37069258752
        start_seq = 846966387
        stop_seq = 1940818286
        footprint = 3644
        bytes_processed = 3644
        flushed_bytes = 3644
        pkth = {ts = {tv_sec = 100375, tv_usec = 0}, caplen = 0, pktlen = 0, ingress_index = -1, egress_index = -1, 
ingress_group = -1, egress_group = -1, flags = 0, opaque = 8, priv_ptr = 0x8a1800000, flow_id = 535241216, 
address_space_id = 0}
        enc_flags = 2147483648
        snort_ticks_start = 51544732022
        snort_ticks_end = 113187
#12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50, st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4, 
dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
No locals.
#13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, 
dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4559
        bytes = 3644
#14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50, st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, 
dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
        fm = (FlushMgr *) 0x80811cfb4
#15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0, tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
        p = (Packet *) 0x8033a4900
        flushed = 1926
        tmp_pcap_hdr = {ts = {tv_sec = 1433431165, tv_usec = 321125}, caplen = 94, pktlen = 94, ingress_index = 
5004089, egress_index = 0, ingress_group = 38246208, egress_group = 8, flags = 4294960320, opaque = 32767, priv_ptr = 
0x4b4c81, flow_id = 0, address_space_id = 0}
#16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0, freeApplicationData=1) at snort_stream_tcp.c:5115
        tcpssn = (TcpSession *) 0x80811ce50
#17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
No locals.
#18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0, s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) 
at snort_stream_tcp.c:5648
        sscc = {old_mem_in_use = 15788887, client_ip = {family = 2, bits = 32, ip = {u6_addr8 = "\nm\027L", '\0' <repeats 11 times>, u6_addr16 = 
{27914, 19479, 0, 0, 0, 0, 0, 0}, u6_addr32 = {1276603658, 0, 0, 0}}}, server_ip = {family = 2, bits = 32, ip = {u6_addr8 = "\nm\026\024", '\0' 
<repeats 11 times>,
      u6_addr16 = {27914, 5142, 0, 0, 0, 0, 0, 0}, u6_addr32 = {337014026, 0, 0, 0}}}, client_port = 39946, server_port 
= 6400, lw_session_state = 200, lw_session_flags = 4284679, app_proto_id = 0}
        tdb = {seq = 1940818286, ack = 2349672268, win = 64032, end_seq = 1940818325, ts = 0}
        rc = 0
        status = 4512282
        snort_ticks_start = 34397587520
        snort_ticks_end = 140737488348864
#19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at spp_stream6.c:751
        key = {ip_l = {0, 0, 4216431, 0}, ip_h = {0, 2, 362856224, 8}, port_l = 59136, port_h = 65535, vlan_tag = 32767, protocol = 0 
'\0', pad = 0 '\0', mplsLabel = 5328944, addressSpaceId = 0, addressSpaceIdPad1 = 0}
        scb = (SessionControlBlock *) 0x8a1aad2f0
        snort_ticks_start = 140737488348960
        snort_ticks_end = 34722594592
#20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0, policy=0x802faa000) at detect.c:136
        scb = (SessionControlBlock *) 0x8a1aad2f0
        ppn = (PreprocEvalFuncNode *) 0x815b61340
        pps_enabled_foo = 1123336
        alerts_processed = true
#21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
        retval = 0
        policy_id = 0
        policy = (SnortPolicy *) 0x802faa000
        pktcnt = 0
        snort_ticks_start = 0
        snort_ticks_end = 6059431713369489410
#22 0x00000000004351b3 in ProcessPacket (p=0xee2a40, pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at 
snort.c:1873
        verdict = DAQ_VERDICT_PASS
        __func__ = "ProcessPacket"
#23 0x0000000000434ccd in PacketCallback (user=0x0, pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
        inject = 0
        verdict = DAQ_VERDICT_PASS
        snort_ticks_start = 34894979584
        snort_ticks_end = 34367935488
#24 0x000000000052fe34 in pcap_process_loop ()
No symbol table info available.
#25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
No symbol table info available.
#26 0x000000000053025f in pcap_daq_acquire ()
No symbol table info available.
#27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40 <PacketCallback>, user=0x0) at sfdaq.c:541
        err = 0
#28 0x0000000000437616 in PacketLoop () at snort.c:3268
        error = 0
        pkts_to_read = 0
#29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at snort.c:921
        tmp_ptr = 0x0
        intf = 0x8024c4540 "mon0"
        daqInit = 1
#30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at snort.c:817
No locals.
rax            0x0      0
rbx            0x7fffffffddec   140737488346604
rcx            0x801fc8fbc      34393067452
rdx            0x0      0
rsi            0x6      6
rdi            0x18817  100375
rbp            0x7fffffffde60   0x7fffffffde60
rsp            0x7fffffffddd8   0x7fffffffddd8
r8             0x0      0
r9             0xfffffe0032ea54a8       -2198169037656
r10            0x59     89
r11            0x202    514
r12            0x80811ce50      34495123024
r13            0x8033a51b8      34413892024
r14            0x82251deaa      34935529130
r15            0x1ba24  113188
rip            0x801f2364c      0x801f2364c <thr_kill+12>
eflags         0x206    518
cs             0x43     67
ss             0x3b     59
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
0x801f2364c <thr_kill+12>:        jb     0x801f2364f <thr_kill+15>
0x801f2364e <thr_kill+14>: retq 0x801f2364f <thr_kill+15>: mov 0x2d6bea(%rip),%rcx # 0x8021fa240 <__nsdefaultsrc+5696> 
0x801f23656 <thr_kill+22>:        jmpq   *%rcx
0x801f23658 <thr_kill+24>: nop 0x801f23659 <thr_kill+25>: nop 0x801f2365a <thr_kill+26>: nop 0x801f2365b <thr_kill+27>: nop 0x801f2365c <thr_kill+28>: nop 0x801f2365d <thr_kill+29>: nop 0x801f2365e <thr_kill+30>: nop 0x801f2365f <thr_kill+31>: nop 0x801f23660 <thr_self>: mov $0x1b0,%rax 
0x801f23667 <thr_self+7>: mov    %rcx,%r10
0x801f2366a <thr_self+10>: syscall 0x801f2366c <thr_self+12>: jb 0x801f2366f <thr_self+15> 

Thread 2 (Thread 802407400 (LWP 100375/snort)):
#0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
#1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
#2  0x0000000801fab315 in __assert () from /lib/libc.so.7
#3  0x0000000805068395 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
#4  0x0000000805068781 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
#5  0x000000080506afd0 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
#6  0x000000080506b85b in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
#7  0x000000080506c150 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
#8  0x000000080506cb27 in ?? () from /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
#9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00, policy_id=0, policy=0x802faa000) at detect.c:136
#10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
#11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50, st=0x80811cfa8, bytes=3644, p=0x8033a4900, 
sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
#12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50, st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4, 
dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
#13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, 
dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4559
#14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50, st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, 
dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
#15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0, tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
#16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0, freeApplicationData=1) at snort_stream_tcp.c:5115
#17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
#18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0, s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) 
at snort_stream_tcp.c:5648
#19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at spp_stream6.c:751
#20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0, policy=0x802faa000) at detect.c:136
#21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
#22 0x00000000004351b3 in ProcessPacket (p=0xee2a40, pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at 
snort.c:1873
#23 0x0000000000434ccd in PacketCallback (user=0x0, pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
#24 0x000000000052fe34 in pcap_process_loop ()
#25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
#26 0x000000000053025f in pcap_daq_acquire ()
#27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40 <PacketCallback>, user=0x0) at sfdaq.c:541
#28 0x0000000000437616 in PacketLoop () at snort.c:3268
#29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at snort.c:921
#30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at snort.c:817

Thread 1 (Thread 815a59400 (LWP 100459/snort)):
#0  0x0000000801faa40c in nanosleep () from /lib/libc.so.7
#1  0x0000000801f15a58 in sleep () from /lib/libc.so.7
#2  0x0000000801ca8078 in sleep () from /lib/libthr.so.3
#3  0x000000000043b215 in ReloadConfigThread (data=0x0) at snort.c:5695
#4  0x0000000801ca5dc4 in pthread_getprio () from /lib/libthr.so.3
#5  0x0000000000000000 in ?? ()
#0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
The program is running.  Quit anyway (and detach it)? (y or n) Detaching from program: /usr/local/bin/snort, process 
11057




As gdb detached from snort, I got the signal 6 in my syslog:
2015-06-04 17:51:53 +02:00 foobar kernel: pid 11057 (snort), uid 100: exited on signal 6





So, this time we got a signal 6 but during this sensor's 14 hour uptime we've seen:
pid 1199 (snort), uid 100: exited on signal 10
pid 4503 (snort), uid 100: exited on signal 10
pid 5908 (snort), uid 100: exited on signal 10
pid 11057 (snort), uid 100: exited on signal 6





I hope this gdb was helpful.
Let me know if it should be run again.





This was all performed on a sensor running:

   ,,_     -*> Snort! <*-
  o"  )~   Version 2.9.7.3 (Build 217)
   ''''    By Martin Roesch & The Snort Team: http://www.snort.org/contact#team
           Copyright (C) 2014-2015 Cisco and/or its affiliates. All rights reserved.
           Copyright (C) 1998-2013 Sourcefire, Inc., et al.
           Using libpcap version 1.4.0
           Using PCRE version: 8.37 2015-04-28
           Using ZLIB version: 1.2.8


daq-2.0.5

FreeBSD 9.3-RELEASE-p13


/Elof





On Thu, 4 Jun 2015, Hui Cao (huica) wrote:


That¹s cool. All looks good to me. No need to do more things...

Best,
Hui

On 6/4/15, 11:35 AM, "elof () sentor se" <elof () sentor se> wrote:



Hi Hui.

That much I know. It is the debugging steps I'm curious about.

(I think you forgot one important first command: continue )


Is this a good start:

gdb /path/to/snort 1222
(gdb) set logging file gdb-snort.txt
(gdb) set logging on
(gdb) continue

<wait for it to crash>

(gdb) backtrace full
(gdb) info registers
(gdb) x/16i $pc
(gdb) thread apply all backtrace
(gdb) quit

Email the report.


Should I prepare more stuff before the 'continue'?
Like "handle SIG33 pass nostop noprint" or something?

/Elof


On Thu, 4 Jun 2015, Hui Cao (huica) wrote:


Try

Assume snort pid is 1222

gdb /path/to/snort 1222

Best,
Hui.
On 6/4/15, 10:37 AM, "elof () sentor se" <elof () sentor se> wrote:



An update:

On a sensor where snort crashed with signal 6 three times, I downgraded
daq to 2.0.4_1 and rebooted the machine to rule out if the problem seem
to
be in 'snort' or 'daq'.

With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.


This make me believe that there's something wrong in snort 2.9.7.3 and
not in daq 2.0.5.



On this sensor I have now done the opposite, upgraded daq to 2.0.5 and
downgraded snort to 2.9.7.2 to see if I get any more signal 6.

On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
2.0.5
without chroot and uid/gid change, i.e. running as root, in order to
create a core file, if the problem happen again.
(if it doesn't happen on this sensor, I guess the problem lies
somewhere
in the chrooting code in snort. I know it has been updated between
2.9.7.2
and 2.9.7.3)



Russ C also wrote:

Elof - since this is happening frequently, you could try attaching the
debugger to one of your Snort processes and wait wait for segfault.


I know too little about debugging. :-/ Can you give me instructions or
point me to a guide that describes the steps I should take?



/Elof


On Thu, 4 Jun 2015, elof () sentor se wrote:



Five different sensors have now had bus errors (signal 10),
segmentation
faults (signal 11) and even signal 6 (SIGABRT).

My snort config uses both chroot and dropping user privileges, so even
if
I start out as root with ulimit unlimited, this doesn't seem to be in
effect
after the chroot/uid-change.

So currently I have no core-file to debug. :-/

Anyone know how to set the ulimits for a chrooted and uid/gid-changed
process in FreeBSD?

/Elof


On Thu, 4 Jun 2015, elof () sentor se wrote:



Hi Hui!

Yes, the dynamic engine/preproc files are updated as well.

Last night the problem reocurred, so this seem to be reproduceable.
Good.
Then there's a good chance this problem can be sorted out.


A few minutes ago a signal 10 happened on another sensor (running
FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in Snort
2.9.7.3 and not in the hardware nor in FreeBSD.


I will compile a debug-snort and try to generate core files.
I'll let you know the outcome next week.

/Elof


On Wed, 3 Jun 2015, Hui cao wrote:


Hi Elof,

Are snort and snort dynamic preprocessors are in sync?

If so, can you help us get a backtrace from the crush? You need
1)  build snort with ./configure --enable-debug
2)  allowing core dump (ulimit -c unlimited)
3) run the snort
4) use "gdb snort core_file " and them type "bt" in the gdb command
line

Best,
Hui.


On 06/03/2015 05:51 AM, elof () sentor se wrote:

Hi all!

This is just a report to inform that after I updated snort and DAQ
to the
latest versions, one of my sensors started throwing signal 10 (bus
error)
and signal 11 (segmentation fault).

# uptime
11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
# dmesg | grep snort
pid 1183 (snort), uid 100: exited on signal 11
pid 16920 (snort), uid 100: exited on signal 11
pid 17502 (snort), uid 100: exited on signal 11
pid 18862 (snort), uid 100: exited on signal 11
pid 20223 (snort), uid 100: exited on signal 11
pid 20927 (snort), uid 100: exited on signal 11
pid 1193 (snort), uid 100: exited on signal 11
pid 2447 (snort), uid 100: exited on signal 11
pid 3811 (snort), uid 100: exited on signal 10
pid 7881 (snort), uid 100: exited on signal 11
pid 9252 (snort), uid 100: exited on signal 10
pid 25593 (snort), uid 100: exited on signal 11
pid 26627 (snort), uid 100: exited on signal 11
pid 56658 (snort), uid 100: exited on signal 11
pid 57237 (snort), uid 100: exited on signal 10
pid 58595 (snort), uid 100: exited on signal 11
pid 68639 (snort), uid 100: exited on signal 11
pid 70008 (snort), uid 100: exited on signal 11
pid 71361 (snort), uid 100: exited on signal 10
pid 72725 (snort), uid 100: exited on signal 11

20 crashes in a day...
A reboot didn't help.

This sensor has never behaved like this during its lifetime (1
year).




FreeBSD 9.3 amd64

     ,,_     -*> Snort! <*-
    o"  )~   Version 2.9.7.3 (Build 217)
     ''''    By Martin Roesch & The Snort Team:
http://www.snort.org/contact#team
             Copyright (C) 2014-2015 Cisco and/or its affiliates.
All rights
reserved.
             Copyright (C) 1998-2013 Sourcefire, Inc., et al.
             Using libpcap version 1.4.0
             Using PCRE version: 8.37 2015-04-28
             Using ZLIB version: 1.2.8

daq-2.0.5



Bus errors are quite unusual in general, so I'll keep looking at
this,
trying to see if it is e.g. paging errors.
It doesn't look like it though:
# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap   4194300        0  4194300     0%

The machine doesn't seem to be overheated either:
System Temp:    30 degrees C
Peripheral Temp: 40 degrees C
CPU Temp: Low


If you need me to do something special to debug this further, let
me
know.


PS. It is only one sensor, out of 20, that behaves like this. So
perhaps
it is something in the mirrored traffic that make DAQ or snort
point
at
illegal memory addresses and crash.
Or this particular machine is having hardware issues. However, it
is
strange that those hw-issues should suddenly start right after I
updated
the software on the machine...

When I write this, the current snort process has been alive for 5
hours.
It's going to be interesting to see if the traffic tonight will
cause it
to crash many times again.

/Elof



--------------------------------------------------------------------
--
--------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!





---------------------------------------------------------------------
--
-------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!





----------------------------------------------------------------------
--
------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!





-----------------------------------------------------------------------
--
-----
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!






------------------------------------------------------------------------------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!

------------------------------------------------------------------------------

_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!
Previous By Date Next
Previous By Thread Next

Current thread: