Re: Bus errors and segmentation faults after upgrade to 2.9.7.3 and daq 2.0.5

["Hui Cao (huica)" <huica () cisco com>]

Thanks!

The issue happens on smtp preprocessor, but the so is not compiled with
debug enabled. Can you recompile it with ―enable-debug ?

Best,
Hui.

On 6/4/15, 12:10 PM, "elof () sentor se" <elof () sentor se> wrote:

> So I just had a signal 6...
>
> I assume I can't attach files to the mailing list, so here it is,
> directly 
> in the mailbody. :-)
>
>
>
>
>
> gdb /usr/local/bin/snort 11057
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
> Attaching to program: /usr/local/bin/snort, process 11057
> Reading symbols from /usr/local/lib/libdnet.so.1...done.
> Loaded symbols for /usr/local/lib/libdnet.so.1
> Reading symbols from /usr/local/lib/libpcre.so.1...done.
> Loaded symbols for /usr/local/lib/libpcre.so.1
> Reading symbols from /lib/libm.so.5...done.
> Loaded symbols for /lib/libm.so.5
> Reading symbols from /lib/libcrypto.so.6...done.
> Loaded symbols for /lib/libcrypto.so.6
> Reading symbols from /lib/libpcap.so.8...done.
> Loaded symbols for /lib/libpcap.so.8
> Reading symbols from /usr/local/lib/libsfbpf.so.0...done.
> Loaded symbols for /usr/local/lib/libsfbpf.so.0
> Reading symbols from /lib/libz.so.6...done.
> Loaded symbols for /lib/libz.so.6
> Reading symbols from /usr/lib/liblzma.so.5...done.
> Loaded symbols for /usr/lib/liblzma.so.5
> Reading symbols from /lib/libthr.so.3...done.
> [New Thread 815a59400 (LWP 100459/snort)]
> [New Thread 802407400 (LWP 100375/snort)]
> Loaded symbols for /lib/libthr.so.3
> Reading symbols from /lib/libc.so.7...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from
> /usr/local/lib/snort_dynamicengine/libsf_engine.so...done.
> Loaded symbols for /usr/local/lib/snort_dynamicengine/libsf_engine.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_dce2_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_dnp3_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_dns_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so...don
> e.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_ftptelnet_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_gtp_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_imap_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_modbus_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_pop_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so...do
> ne.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_reputation_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_sdf_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_sip_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssh_preproc.so
> Reading symbols from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so...done.
> Loaded symbols for
> /usr/local/lib/snort_dynamicpreprocessor//libsf_ssl_preproc.so
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> [Switching to Thread 815a59400 (LWP 100459/snort)]
> 0x0000000801faa40c in nanosleep () from /lib/libc.so.7
> (gdb) set logging file gdb-snort.txt
> (gdb) set logging on
> Copying output to gdb-snort.txt.
> (gdb) continue
> Continuing.
>
>
>
> <...it has just been a few minutes when I receive a SIGABRT>
>
>
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 802407400 (LWP 100375/snort)]
> 0x0000000801f2364c in thr_kill () from /lib/libc.so.7
>
> (gdb) backtrace full
> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
> No symbol table info available.
> #1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
> No symbol table info available.
> #2  0x0000000801fab315 in __assert () from /lib/libc.so.7
> No symbol table info available.
> #3  0x0000000805068395 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> No symbol table info available.
> #4  0x0000000805068781 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> No symbol table info available.
> #5  0x000000080506afd0 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> No symbol table info available.
> #6  0x000000080506b85b in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> No symbol table info available.
> #7  0x000000080506c150 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> No symbol table info available.
> #8  0x000000080506cb27 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> No symbol table info available.
> #9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
> policy_id=0, policy=0x802faa000) at detect.c:136
>       scb = (SessionControlBlock *) 0x8a1aad2f0
>       ppn = (PreprocEvalFuncNode *) 0x8033ff0a0
>       pps_enabled_foo = 1123336
>       alerts_processed = true
> #10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
>       retval = 0
>       policy_id = 0
>       policy = (SnortPolicy *) 0x802faa000
>       pktcnt = 0
>       snort_ticks_start = 34413886976
>       snort_ticks_end = 34413888664
> #11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
>       tmp_do_detect = 1
>       tmp_do_detect_content = 1
>       snort_ticks_start = 37073416192
>       snort_ticks_end = 37069258752
>       start_seq = 846966387
>       stop_seq = 1940818286
>       footprint = 3644
>       bytes_processed = 3644
>       flushed_bytes = 3644
>       pkth = {ts = {tv_sec = 100375, tv_usec = 0}, caplen = 0, pktlen = 0,
> ingress_index = -1, egress_index = -1, ingress_group = -1, egress_group =
> -1, flags = 0, opaque = 8, priv_ptr = 0x8a1800000, flow_id = 535241216,
> address_space_id = 0}
>       enc_flags = 2147483648
>       snort_ticks_start = 51544732022
>       snort_ticks_end = 113187
> #12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
> No locals.
> #13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
> p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
> dir=128) at snort_stream_tcp.c:4559
>       bytes = 3644
> #14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
> st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
> sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
>       fm = (FlushMgr *) 0x80811cfb4
> #15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
> tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
>       p = (Packet *) 0x8033a4900
>       flushed = 1926
>       tmp_pcap_hdr = {ts = {tv_sec = 1433431165, tv_usec = 321125}, caplen =
> 94, pktlen = 94, ingress_index = 5004089, egress_index = 0, ingress_group
> = 38246208, egress_group = 8, flags = 4294960320, opaque = 32767,
> priv_ptr = 0x4b4c81, flow_id = 0, address_space_id = 0}
> #16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
> freeApplicationData=1) at snort_stream_tcp.c:5115
>       tcpssn = (TcpSession *) 0x80811ce50
> #17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
> (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
> No locals.
> #18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
> s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
>       sscc = {old_mem_in_use = 15788887, client_ip = {family = 2, bits = 32,
> ip = {u6_addr8 = "\nm\027L", '\0' <repeats 11 times>, u6_addr16 = {27914,
> 19479, 0, 0, 0, 0, 0, 0}, u6_addr32 = {1276603658, 0, 0, 0}}}, server_ip
> = {family = 2, bits = 32, ip = {u6_addr8 = "\nm\026\024", '\0' <repeats
> 11 times>,
>       u6_addr16 = {27914, 5142, 0, 0, 0, 0, 0, 0}, u6_addr32 =
> {337014026, 0, 0, 0}}}, client_port = 39946, server_port = 6400,
> lw_session_state = 200, lw_session_flags = 4284679, app_proto_id = 0}
>       tdb = {seq = 1940818286, ack = 2349672268, win = 64032, end_seq =
> 1940818325, ts = 0}
>       rc = 0
>       status = 4512282
>       snort_ticks_start = 34397587520
>       snort_ticks_end = 140737488348864
> #19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
> spp_stream6.c:751
>       key = {ip_l = {0, 0, 4216431, 0}, ip_h = {0, 2, 362856224, 8}, port_l =
> 59136, port_h = 65535, vlan_tag = 32767, protocol = 0 '\0', pad = 0 '\0',
> mplsLabel = 5328944, addressSpaceId = 0, addressSpaceIdPad1 = 0}
>       scb = (SessionControlBlock *) 0x8a1aad2f0
>       snort_ticks_start = 140737488348960
>       snort_ticks_end = 34722594592
> #20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
> policy=0x802faa000) at detect.c:136
>       scb = (SessionControlBlock *) 0x8a1aad2f0
>       ppn = (PreprocEvalFuncNode *) 0x815b61340
>       pps_enabled_foo = 1123336
>       alerts_processed = true
> #21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
>       retval = 0
>       policy_id = 0
>       policy = (SnortPolicy *) 0x802faa000
>       pktcnt = 0
>       snort_ticks_start = 0
>       snort_ticks_end = 6059431713369489410
> #22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873
>       verdict = DAQ_VERDICT_PASS
>       __func__ = "ProcessPacket"
> #23 0x0000000000434ccd in PacketCallback (user=0x0,
> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
>       inject = 0
>       verdict = DAQ_VERDICT_PASS
>       snort_ticks_start = 34894979584
>       snort_ticks_end = 34367935488
> #24 0x000000000052fe34 in pcap_process_loop ()
> No symbol table info available.
> #25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
> No symbol table info available.
> #26 0x000000000053025f in pcap_daq_acquire ()
> No symbol table info available.
> #27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
> <PacketCallback>, user=0x0) at sfdaq.c:541
>       err = 0
> #28 0x0000000000437616 in PacketLoop () at snort.c:3268
>       error = 0
>       pkts_to_read = 0
> #29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
> snort.c:921
>       tmp_ptr = 0x0
>       intf = 0x8024c4540 "mon0"
>       daqInit = 1
> #30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
> snort.c:817
> No locals.
> rax            0x0     0
> rbx            0x7fffffffddec  140737488346604
> rcx            0x801fc8fbc     34393067452
> rdx            0x0     0
> rsi            0x6     6
> rdi            0x18817 100375
> rbp            0x7fffffffde60  0x7fffffffde60
> rsp            0x7fffffffddd8  0x7fffffffddd8
> r8             0x0     0
> r9             0xfffffe0032ea54a8      -2198169037656
> r10            0x59    89
> r11            0x202   514
> r12            0x80811ce50     34495123024
> r13            0x8033a51b8     34413892024
> r14            0x82251deaa     34935529130
> r15            0x1ba24 113188
> rip            0x801f2364c     0x801f2364c <thr_kill+12>
> eflags         0x206   518
> cs             0x43    67
> ss             0x3b    59
> ds             0x0     0
> es             0x0     0
> fs             0x0     0
> gs             0x0     0
> 0x801f2364c <thr_kill+12>:     jb     0x801f2364f <thr_kill+15>
> 0x801f2364e <thr_kill+14>:     retq
> 0x801f2364f <thr_kill+15>:     mov    0x2d6bea(%rip),%rcx        #
> 0x8021fa240 <__nsdefaultsrc+5696>
> 0x801f23656 <thr_kill+22>:     jmpq   *%rcx
> 0x801f23658 <thr_kill+24>:     nop
> 0x801f23659 <thr_kill+25>:     nop
> 0x801f2365a <thr_kill+26>:     nop
> 0x801f2365b <thr_kill+27>:     nop
> 0x801f2365c <thr_kill+28>:     nop
> 0x801f2365d <thr_kill+29>:     nop
> 0x801f2365e <thr_kill+30>:     nop
> 0x801f2365f <thr_kill+31>:     nop
> 0x801f23660 <thr_self>:        mov    $0x1b0,%rax
> 0x801f23667 <thr_self+7>:      mov    %rcx,%r10
> 0x801f2366a <thr_self+10>:     syscall
> 0x801f2366c <thr_self+12>:     jb     0x801f2366f <thr_self+15>
>
> Thread 2 (Thread 802407400 (LWP 100375/snort)):
> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
> #1  0x0000000801fc7c4b in abort () from /lib/libc.so.7
> #2  0x0000000801fab315 in __assert () from /lib/libc.so.7
> #3  0x0000000805068395 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> #4  0x0000000805068781 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> #5  0x000000080506afd0 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> #6  0x000000080506b85b in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> #7  0x000000080506c150 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> #8  0x000000080506cb27 in ?? () from
> /usr/local/lib/snort_dynamicpreprocessor//libsf_smtp_preproc.so
> #9  0x00000000004423f7 in DispatchPreprocessors (p=0x8033a3e00,
> policy_id=0, policy=0x802faa000) at detect.c:136
> #10 0x000000000044286d in Preprocess (p=0x8033a3e00) at detect.c:234
> #11 0x00000000004e44b8 in _flush_to_seq (tcpssn=0x80811ce50,
> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4352
> #12 0x00000000004e3e06 in flush_to_seq (tcpssn=0x80811ce50,
> st=0x80811cfa8, bytes=3644, p=0x8033a4900, sip=0x8033a51a4,
> dip=0x8033a51b8, sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4493
> #13 0x00000000004e4f8e in flush_ackd (tcpssn=0x80811ce50, st=0x80811cfa8,
> p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8, sp=39946, dp=6400,
> dir=128) at snort_stream_tcp.c:4559
> #14 0x00000000004e3bcb in flush_stream (tcpssn=0x80811ce50,
> st=0x80811cfa8, p=0x8033a4900, sip=0x8033a51a4, dip=0x8033a51b8,
> sp=39946, dp=6400, dir=128) at snort_stream_tcp.c:4588
> #15 0x00000000004e60b4 in FlushQueuedSegs (scb=0x8a1aad2f0,
> tcpssn=0x80811ce50) at snort_stream_tcp.c:5074
> #16 0x00000000004e61bd in TcpSessionCleanup (scb=0x8a1aad2f0,
> freeApplicationData=1) at snort_stream_tcp.c:5115
> #17 0x00000000004e61ea in TcpSessionCleanupWithFreeApplicationData
> (scb=0x8a1aad2f0) at snort_stream_tcp.c:5122
> #18 0x00000000004e69e1 in StreamProcessTcp (p=0xee2a40, scb=0x8a1aad2f0,
> s5TcpPolicy=0x812807000, skey=0x7fffffffe6c0) at snort_stream_tcp.c:5648
> #19 0x00000000004b5a14 in StreamProcess (p=0xee2a40, context=0x0) at
> spp_stream6.c:751
> #20 0x00000000004423f7 in DispatchPreprocessors (p=0xee2a40, policy_id=0,
> policy=0x802faa000) at detect.c:136
> #21 0x000000000044286d in Preprocess (p=0xee2a40) at detect.c:234
> #22 0x00000000004351b3 in ProcessPacket (p=0xee2a40,
> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?, ft=0x0) at snort.c:1873
> #23 0x0000000000434ccd in PacketCallback (user=0x0,
> pkthdr=0x7fffffffe8c0, pkt=0x82251deaa "\220?"\b?) at snort.c:1718
> #24 0x000000000052fe34 in pcap_process_loop ()
> #25 0x0000000801429dcd in pcap_create_interface () from /lib/libpcap.so.8
> #26 0x000000000053025f in pcap_daq_acquire ()
> #27 0x000000000045a1b4 in DAQ_Acquire (max=0, callback=0x434b40
> <PacketCallback>, user=0x0) at sfdaq.c:541
> #28 0x0000000000437616 in PacketLoop () at snort.c:3268
> #29 0x00000000004337c7 in SnortMain (argc=6, argv=0x7fffffffebc0) at
> snort.c:921
> #30 0x000000000043364f in main (argc=6, argv=0x7fffffffebc0) at
> snort.c:817
>
> Thread 1 (Thread 815a59400 (LWP 100459/snort)):
> #0  0x0000000801faa40c in nanosleep () from /lib/libc.so.7
> #1  0x0000000801f15a58 in sleep () from /lib/libc.so.7
> #2  0x0000000801ca8078 in sleep () from /lib/libthr.so.3
> #3  0x000000000043b215 in ReloadConfigThread (data=0x0) at snort.c:5695
> #4  0x0000000801ca5dc4 in pthread_getprio () from /lib/libthr.so.3
> #5  0x0000000000000000 in ?? ()
> #0  0x0000000801f2364c in thr_kill () from /lib/libc.so.7
> The program is running.  Quit anyway (and detach it)? (y or n) Detaching
> from program: /usr/local/bin/snort, process 11057
>
>
>
>
> As gdb detached from snort, I got the signal 6 in my syslog:
> 2015-06-04 17:51:53 +02:00 foobar kernel: pid 11057 (snort), uid 100:
> exited on signal 6
>
>
>
>
>
> So, this time we got a signal 6 but during this sensor's 14 hour uptime
> we've seen:
> pid 1199 (snort), uid 100: exited on signal 10
> pid 4503 (snort), uid 100: exited on signal 10
> pid 5908 (snort), uid 100: exited on signal 10
> pid 11057 (snort), uid 100: exited on signal 6
>
>
>
>
>
> I hope this gdb was helpful.
> Let me know if it should be run again.
>
>
>
>
>
> This was all performed on a sensor running:
>
>    ,,_     -*> Snort! <*-
>   o"  )~   Version 2.9.7.3 (Build 217)
>    ''''    By Martin Roesch & The Snort Team:
> http://www.snort.org/contact#team
>            Copyright (C) 2014-2015 Cisco and/or its affiliates. All
> rights reserved.
>            Copyright (C) 1998-2013 Sourcefire, Inc., et al.
>            Using libpcap version 1.4.0
>            Using PCRE version: 8.37 2015-04-28
>            Using ZLIB version: 1.2.8
>
>
> daq-2.0.5
>
> FreeBSD 9.3-RELEASE-p13
>
>
> /Elof
>
>
>
>
>
> On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
>
> > That¹s cool. All looks good to me. No need to do more things...
> >
> > Best,
> > Hui
> >
> > On 6/4/15, 11:35 AM, "elof () sentor se" <elof () sentor se> wrote:
> >
> > > Hi Hui.
> > >
> > > That much I know. It is the debugging steps I'm curious about.
> > >
> > > (I think you forgot one important first command: continue )
> > >
> > >
> > > Is this a good start:
> > >
> > > gdb /path/to/snort 1222
> > > (gdb) set logging file gdb-snort.txt
> > > (gdb) set logging on
> > > (gdb) continue
> > >
> > > <wait for it to crash>
> > >
> > > (gdb) backtrace full
> > > (gdb) info registers
> > > (gdb) x/16i $pc
> > > (gdb) thread apply all backtrace
> > > (gdb) quit
> > >
> > > Email the report.
> > >
> > >
> > > Should I prepare more stuff before the 'continue'?
> > > Like "handle SIG33 pass nostop noprint" or something?
> > >
> > > /Elof
> > >
> > >
> > > On Thu, 4 Jun 2015, Hui Cao (huica) wrote:
> > >
> > > > Try
> > > >
> > > > Assume snort pid is 1222
> > > >
> > > > gdb /path/to/snort 1222
> > > >
> > > > Best,
> > > > Hui.
> > > > On 6/4/15, 10:37 AM, "elof () sentor se" <elof () sentor se> wrote:
> > > >
> > > > > An update:
> > > > >
> > > > > On a sensor where snort crashed with signal 6 three times, I
> > > > > downgraded
> > > > > daq to 2.0.4_1 and rebooted the machine to rule out if the problem
> > > > > seem
> > > > > to
> > > > > be in 'snort' or 'daq'.
> > > > >
> > > > > With snort 2.9.7.3 and daq 2.0.4_1 I got signal 6 again.
> > > > >
> > > > >
> > > > > This make me believe that there's something wrong in snort 2.9.7.3
> > > > > and
> > > > > not in daq 2.0.5.
> > > > >
> > > > >
> > > > >
> > > > > On this sensor I have now done the opposite, upgraded daq to 2.0.5
> > > > > and
> > > > > downgraded snort to 2.9.7.2 to see if I get any more signal 6.
> > > > >
> > > > > On another sensor, I'm running 2.9.7.3 (compiled with debug) and daq
> > > > > 2.0.5
> > > > > without chroot and uid/gid change, i.e. running as root, in order to
> > > > > create a core file, if the problem happen again.
> > > > > (if it doesn't happen on this sensor, I guess the problem lies
> > > > > somewhere
> > > > > in the chrooting code in snort. I know it has been updated between
> > > > > 2.9.7.2
> > > > > and 2.9.7.3)
> > > > >
> > > > >
> > > > >
> > > > > Russ C also wrote:
> > > > > > Elof - since this is happening frequently, you could try attaching
> > > > > > the
> > > > > > debugger to one of your Snort processes and wait wait for segfault.
> > > > >
> > > > > I know too little about debugging. :-/ Can you give me instructions
> > > > > or
> > > > > point me to a guide that describes the steps I should take?
> > > > >
> > > > >
> > > > >
> > > > > /Elof
> > > > >
> > > > >
> > > > > On Thu, 4 Jun 2015, elof () sentor se wrote:
> > > > >
> > > > > > Five different sensors have now had bus errors (signal 10),
> > > > > > segmentation
> > > > > > faults (signal 11) and even signal 6 (SIGABRT).
> > > > > >
> > > > > > My snort config uses both chroot and dropping user privileges, so
> > > > > > even
> > > > > > if
> > > > > > I start out as root with ulimit unlimited, this doesn't seem to be
> > > > > > in
> > > > > > effect
> > > > > > after the chroot/uid-change.
> > > > > >
> > > > > > So currently I have no core-file to debug. :-/
> > > > > >
> > > > > > Anyone know how to set the ulimits for a chrooted and
> > > > > > uid/gid-changed
> > > > > > process in FreeBSD?
> > > > > >
> > > > > > /Elof
> > > > > >
> > > > > >
> > > > > > On Thu, 4 Jun 2015, elof () sentor se wrote:
> > > > > >
> > > > > > > Hi Hui!
> > > > > > >
> > > > > > > Yes, the dynamic engine/preproc files are updated as well.
> > > > > > >
> > > > > > > Last night the problem reocurred, so this seem to be reproduceable.
> > > > > > > Good.
> > > > > > > Then there's a good chance this problem can be sorted out.
> > > > > > >
> > > > > > >
> > > > > > > A few minutes ago a signal 10 happened on another sensor (running
> > > > > > > FreeBSD 10.1 amd64), so the problem must be in DAQ 2.0.5 or in
> > > > > > > Snort
> > > > > > > 2.9.7.3 and not in the hardware nor in FreeBSD.
> > > > > > >
> > > > > > >
> > > > > > > I will compile a debug-snort and try to generate core files.
> > > > > > > I'll let you know the outcome next week.
> > > > > > >
> > > > > > > /Elof
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 3 Jun 2015, Hui cao wrote:
> > > > > > >
> > > > > > > > Hi Elof,
> > > > > > > >
> > > > > > > > Are snort and snort dynamic preprocessors are in sync?
> > > > > > > >
> > > > > > > > If so, can you help us get a backtrace from the crush? You need
> > > > > > > > 1)  build snort with ./configure --enable-debug
> > > > > > > > 2)  allowing core dump (ulimit -c unlimited)
> > > > > > > > 3) run the snort
> > > > > > > > 4) use "gdb snort core_file " and them type "bt" in the gdb
> > > > > > > > command
> > > > > > > > line
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Hui.
> > > > > > > >
> > > > > > > >
> > > > > > > > On 06/03/2015 05:51 AM, elof () sentor se wrote:
> > > > > > > > > Hi all!
> > > > > > > > >
> > > > > > > > > This is just a report to inform that after I updated snort and
> > > > > > > > > DAQ
> > > > > > > > > to the
> > > > > > > > > latest versions, one of my sensors started throwing signal 10
> > > > > > > > > (bus
> > > > > > > > > error)
> > > > > > > > > and signal 11 (segmentation fault).
> > > > > > > > >
> > > > > > > > > # uptime
> > > > > > > > > 11:32AM  up 1 day,  9:48, 1 user, load averages: 0.36, 0.37, 0.38
> > > > > > > > > # dmesg | grep snort
> > > > > > > > > pid 1183 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 16920 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 17502 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 18862 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 20223 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 20927 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 1193 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 2447 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 3811 (snort), uid 100: exited on signal 10
> > > > > > > > > pid 7881 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 9252 (snort), uid 100: exited on signal 10
> > > > > > > > > pid 25593 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 26627 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 56658 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 57237 (snort), uid 100: exited on signal 10
> > > > > > > > > pid 58595 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 68639 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 70008 (snort), uid 100: exited on signal 11
> > > > > > > > > pid 71361 (snort), uid 100: exited on signal 10
> > > > > > > > > pid 72725 (snort), uid 100: exited on signal 11
> > > > > > > > >
> > > > > > > > > 20 crashes in a day...
> > > > > > > > > A reboot didn't help.
> > > > > > > > >
> > > > > > > > > This sensor has never behaved like this during its lifetime (1
> > > > > > > > > year).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > FreeBSD 9.3 amd64
> > > > > > > > >
> > > > > > > > >      ,,_     -*> Snort! <*-
> > > > > > > > >     o"  )~   Version 2.9.7.3 (Build 217)
> > > > > > > > >      ''''    By Martin Roesch & The Snort Team:
> > > > > > > > > http://www.snort.org/contact#team
> > > > > > > > >              Copyright (C) 2014-2015 Cisco and/or its affiliates.
> > > > > > > > > All rights
> > > > > > > > > reserved.
> > > > > > > > >              Copyright (C) 1998-2013 Sourcefire, Inc., et al.
> > > > > > > > >              Using libpcap version 1.4.0
> > > > > > > > >              Using PCRE version: 8.37 2015-04-28
> > > > > > > > >              Using ZLIB version: 1.2.8
> > > > > > > > >
> > > > > > > > > daq-2.0.5
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Bus errors are quite unusual in general, so I'll keep looking at
> > > > > > > > > this,
> > > > > > > > > trying to see if it is e.g. paging errors.
> > > > > > > > > It doesn't look like it though:
> > > > > > > > > # swapinfo
> > > > > > > > > Device          1K-blocks     Used    Avail Capacity
> > > > > > > > > /dev/mirror/swap   4194300        0  4194300     0%
> > > > > > > > >
> > > > > > > > > The machine doesn't seem to be overheated either:
> > > > > > > > > System Temp:  30 degrees C
> > > > > > > > > Peripheral Temp: 40 degrees C
> > > > > > > > > CPU Temp: Low
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > If you need me to do something special to debug this further, let
> > > > > > > > > me
> > > > > > > > > know.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > PS. It is only one sensor, out of 20, that behaves like this. So
> > > > > > > > > perhaps
> > > > > > > > > it is something in the mirrored traffic that make DAQ or snort
> > > > > > > > > point
> > > > > > > > > at
> > > > > > > > > illegal memory addresses and crash.
> > > > > > > > > Or this particular machine is having hardware issues. However, it
> > > > > > > > > is
> > > > > > > > > strange that those hw-issues should suddenly start right after I
> > > > > > > > > updated
> > > > > > > > > the software on the machine...
> > > > > > > > >
> > > > > > > > > When I write this, the current snort process has been alive for 5
> > > > > > > > > hours.
> > > > > > > > > It's going to be interesting to see if the traffic tonight will
> > > > > > > > > cause it
> > > > > > > > > to crash many times again.
> > > > > > > > >
> > > > > > > > > /Elof
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ------------------------------------------------------------------
> > > > > > > > > --
> > > > > > > > > --
> > > > > > > > > --------
> > > > > > > > > _______________________________________________
> > > > > > > > > Snort-devel mailing list
> > > > > > > > > Snort-devel () lists sourceforge net
> > > > > > > > > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > > > > > > > > Archive:
> > > > > > > > >
> > > > > > > > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-deve
> > > > > > > > > l
> > > > > > > > >
> > > > > > > > > Please visit http://blog.snort.org for the latest news about
> > > > > > > > > Snort!
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > -------------------------------------------------------------------
> > > > > > > > --
> > > > > > > > --
> > > > > > > > -------
> > > > > > > > _______________________________________________
> > > > > > > > Snort-devel mailing list
> > > > > > > > Snort-devel () lists sourceforge net
> > > > > > > > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > > > > > > > Archive:
> > > > > > > >
> > > > > > > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > > > > > > >
> > > > > > > > Please visit http://blog.snort.org for the latest news about
> > > > > > > > Snort!
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --------------------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > ------
> > > > > > > _______________________________________________
> > > > > > > Snort-devel mailing list
> > > > > > > Snort-devel () lists sourceforge net
> > > > > > > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > > > > > > Archive:
> > > > > > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > > > > > >
> > > > > > > Please visit http://blog.snort.org for the latest news about Snort!
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > -----
> > > > > > _______________________________________________
> > > > > > Snort-devel mailing list
> > > > > > Snort-devel () lists sourceforge net
> > > > > > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > > > > > Archive:
> > > > > > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> > > > > >
> > > > > > Please visit http://blog.snort.org for the latest news about Snort!
> >
> >
> >
> > -------------------------------------------------------------------------
> > -----
> > _______________________________________________
> > Snort-devel mailing list
> > Snort-devel () lists sourceforge net
> > https://lists.sourceforge.net/lists/listinfo/snort-devel
> > Archive:
> > http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel
> >
> > Please visit http://blog.snort.org for the latest news about Snort!

------------------------------------------------------------------------------
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!