Re: Problems with Snort 2.9.11.0 - Snort is terminating unexpectedly

[Anna <Anna () sonru com>]

OS: Centos 7.4

I did not compile it, I installed Snort from yum rpm -  
https://snort.org/downloads/snort/snort-2.9.11-1.centos7.x86_64.rpm

gdb really not working as cannot find the debuginfo for snort  and daq pkg

gdb /sbin/snort 1375

Reading symbols from /usr/sbin/snort-plain...(no debugging symbols found)...done.
Attaching to program: /sbin/snort, process 1375
Reading symbols from /lib64/libnghttp2.so.14...Reading symbols from /lib64/libnghttp2.so.14...(no debugging symbols 
found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnghttp2.so.14
Reading symbols from /lib64/libdnet.1...Missing separate debuginfo for /lib64/libdnet.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/cd/00c325aa44135552d31222ba244cb8f07fb761.debug
Reading symbols from /lib64/libdnet.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdnet.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libpcre.so.1.2.0.debug...done.
done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libnsl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libm-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from 
/usr/lib/debug/usr/lib64/libcrypto.so.1.0.2k.debug...done.
done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libsfbpf.so.0...Reading symbols from /lib64/libsfbpf.so.0...(no debugging symbols 
found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsfbpf.so.0
Reading symbols from /lib64/libpcap.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libpcap.so.1.5.3.debug...done.
done.
Loaded symbols for /lib64/libpcap.so.1
Reading symbols from /lib64/libz.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done.
done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libpthread.so.0...Reading symbols from 
/usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
done.
[New LWP 1377]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libnss_files-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/snort-2.9.11_dynamicengine/libsf_engine.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicengine/libsf_engine.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dce2_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dce2_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dnp3_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dns_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dns_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ftptelnet_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ftptelnet_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_gtp_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_gtp_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_imap_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_imap_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_modbus_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_modbus_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_pop_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_pop_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_reputation_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_reputation_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sdf_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sdf_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sip_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sip_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_smtp_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_smtp_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssh_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssh_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssl_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssl_preproc.so
0x00007ff26b57da3d in poll () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
Missing separate debuginfos, use: debuginfo-install snort-2.9.11-1.x86_64 daq-2.0.6-1.x86_64 
libnghttp2-1.21.1-1.el7.x86_64

when I run debuginfo-install

Could not find debuginfo for main pkg: 1:snort-2.9.11-1.x86_64
Could not find debuginfo pkg for dependency package libnghttp2-1.21.1-1.el7.x86_64
Could not find debuginfo pkg for dependency package daq-2.0.6-1.x86_64

(Is there a separate repo for debuginfo for snort?)

I also notice that when the process restarts and hangs there are different parameters running (I am running systemd 
start with parameters /sbin/snort -q -u snort -g snort -c /etc/snort/snort.conf -i eth0 -  while the restarted process 
is using those /usr/sbin/snort -A fast -b -d -D -i eth0 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort)


Thank you

Anna


> On 17 Oct 2017, at 14:43, Carter Waxman (cwaxman) <cwaxman () cisco com> wrote:
>
> Can you provide the OS and C runtime / build chain you are using? Also, can you attach gdb to the stopped process and 
> send the backtrace?
>  
> Thanks,
> Carter
>  
> From: Snort-users <snort-users-bounces () lists snort org <mailto:snort-users-bounces () lists snort org>> on behalf 
> of Anna <Anna () sonru com <mailto:Anna () sonru com>>
> Date: Tuesday, October 17, 2017 at 7:21 AM
> To: "snort-users () lists snort org <mailto:snort-users () lists snort org>" <snort-users () lists snort org 
> <mailto:snort-users () lists snort org>>
> Subject: [Snort-users] Problems with Snort 2.9.11.0 - Snort is terminating unexpectedly
>  
> Hello,
>  
> I have another problem now. I upgraded the 2.9.9.0 to 2.9.11 and the problem with snort using too much memory went 
> away. It got replaced with Snort working for  few hours and terminating while the top is showing that snort is still 
> running.
>  
> Snort was upgraded and started on 13th of Oct at 9:31 am
>  
> Oct 13 09:31:46 DEV_SERVER systemd: Started Snort NIDS Daemon.
> Oct 13 09:31:46 DEV_SERVER systemd: Starting Snort NIDS Daemon...
> Oct 13 09:31:56 DEV_SERVER kernel: device eth0 entered promiscuous mode
>  
> then on 14th of Oct at 3:21am it stopped and tried to restart (probably hanged that is why OS is still showing snort 
> as running)
>  
> Oct 14 03:21:01 DEV_SERVER systemd: Stopping SYSV: snort is a lightweight network intrusion detection tool that 
> currently detects more than 1100 host and network vulnerabilities, portscans, backdoors, and more....
> Oct 14 03:21:02 DEV_SERVER snort: *** Caught Term-Signal
> Oct 14 03:21:02 DEV_SERVER kernel: device eth0 left promiscuous mode
> Oct 14 03:21:03 DEV_SERVER snortd: Stopping snort: [  OK  ]
> Oct 14 03:21:03 DEV_SERVER systemd: Starting SYSV: snort is a lightweight network intrusion detection tool that 
> currently detects more than 1100 host and network vulnerabilities, portscans, backdoors, and more....
> Oct 14 03:21:03 DEV_SERVER snort[12509]: Running in IDS mode
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]:        --== Initializing Snort ==--
> Oct 14 03:21:03 DEV_SERVER snort[12509]: Initializing Output Plugins!
> Oct 14 03:21:03 DEV_SERVER snort[12509]: Initializing Preprocessors!
> Oct 14 03:21:03 DEV_SERVER snort[12509]: Initializing Plug-ins!
> Oct 14 03:21:03 DEV_SERVER snort[12509]: Parsing Rules file "/etc/snort/snort.conf"
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'HTTP_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 80:81 443 8000 8008 8080:8081 8085 8088 ]
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'SHELLCODE_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 0:79 81:65535 ]
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'ORACLE_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 1024:65535 ]
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'SSH_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 22 ]
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'FTP_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 21 2100 3535 ]
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'SIP_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 5060:5061 5600 ]
> Oct 14 03:21:03 DEV_SERVER snort[12509]:
> Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'FILE_DATA_PORTS' defined :
> Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 80:81 110 143 443 8000 8008 8080:8081 8085 8088 ] 
>  
>  
> This is happening to all servers I upgraded so this is not isolated incident. 
>  
>  
> How to put the snort activity logs in debug and redirect them from messages (not the u2 logs that are going to 
> barnyard)? Also when the Snort is about the terminate those logs are created snort.log.1507951274 (there is not -l 
> flag in the start command or any other output files in snort.conf) they contain the detected traffic logs 
> ^@^D<8a>Óï^@PKr^W? ô<82>\<80>^X^O¦^MÉ^@^@^A^A^H
> ^Y)×<9e><94>Íg­HEAD http://52.125.242.20:80/PMA/ <http://52.125.242.20/PMA/> HTTP/1.1^M
> Connection: Keep-Alive^M
> Keep-Alive: 300^M
> User-Agent: Mozilla/5.0 Jorgee^M
> Host: 52.125.242.20^M
>  
> Alerts log is updated with the alerts from Snort that should go to barnyard2/Snorby
>  
> 10/17-11:12:09.240501  [**] [120:3:1] (http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE [**] 
> [Classification: Unknown Traffic] [Priority: 3] {TCP} 10.0.4.224:80 -> 10.0.4.138:46108
>  
> But Snort is not running according to systemd since Oct 14 03:21:02 DEV_SERVER snort[32171]: *** Caught Term-Signal
>  
> When I try to start snort with systemd without killing the one that is “running” according to OS, they are two 
> processes running then.
>  
> Anybody can shed any light in what is going on with my upgraded Snort?
>  
> Thank you
>  
> Anna

_______________________________________________
Snort-users mailing list
Snort-users () lists snort org
Go to this URL to change user options or unsubscribe:
https://lists.snort.org/mailman/listinfo/snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!