Re: Problems with Snort 2.9.11.0 - Snort is terminating unexpectedly

["Cynthia Leonard \(cyleonar\) via Snort-users" <snort-users () lists snort org>]

Hi  Anna,
Good to know that your problem is resolved.
 here were changes to make Snort reload when certain changes to config are made and SIGHUP is issued  instead of 
getting restarted like it used to before.
Do you think there are some config changes you are making and issuing reload?

Regards
Cynthia


From: Snort-users [mailto:snort-users-bounces () lists snort org] On Behalf Of Anna
Sent: Tuesday, October 24, 2017 3:08 PM
To: Carter Waxman (cwaxman) <cwaxman () cisco com>
Cc: snort-users () lists snort org
Subject: Re: [Snort-users] Problems with Snort 2.9.11.0 - Snort is terminating unexpectedly

Ok, I solved the problem -

My restart (snort[32171]: *** Caught Term-Signal) times are corresponding with daily syslogs logrotate

/etc/logrotate.d/snort has diffrent restart postrotate - while there was no problem with that for some reason in 
2.9.9.0 ==> I started my testing server with 2.9.9.0 just to see if the service was restarted (it was not, while the 
logrotate for syslogs run at 3:31am on 24th of Oct, Snort is running continuously from 23rd of Oct)

The setup is the same for 2.9.9.0 snort (same snort.service, same snort.conf)

[root@SYS_TEST-02 ~]# systemctl status snort
● snort.service - Snort NIDS Daemon
   Loaded: loaded (/usr/lib/systemd/system/snort.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-10-23 14:05:42 UTC; 19h ago
 Main PID: 26359 (snort)
   CGroup: /system.slice/snort.service
           └─26359 /sbin/snort -q -u snort -g snort -c /etc/snort/snort.conf -i eth0

Oct 23 14:05:42 SYS_TEST-02 systemd[1]: Started Snort NIDS Daemon.
Oct 23 14:05:42 SYS_TEST-02 systemd[1]: Starting Snort NIDS Daemon...


 2.9.11 needs to be amended when someone is running off systemd in CentOS (do not know if any other distribution is 
also failing/hung)

So if anybody is running systemctl start/stop/restart snort (with your own snort.service) needs to amend in the 
/etc/logrotate.d/snort

 postrotate
        #/etc/init.d/snortd restart 1>/dev/null || true
         /usr/bin/systemctl restart snort 1>/dev/null || true
    endscript

I tested it in Monday/Tuesday and snort restarted successfully 5h ago.

Can anyone tell me why this issue is present in 2.9.11 but not affecting 2.9.9.0?


Thank you

Anna


On 17 Oct 2017, at 15:56, Anna <Anna () sonru com<mailto:Anna () sonru com>> wrote:

OS: Centos 7.4

I did not compile it, I installed Snort from yum rpm -  
https://snort.org/downloads/snort/snort-2.9.11-1.centos7.x86_64.rpm

gdb really not working as cannot find the debuginfo for snort  and daq pkg

gdb /sbin/snort 1375

Reading symbols from /usr/sbin/snort-plain...(no debugging symbols found)...done.
Attaching to program: /sbin/snort, process 1375
Reading symbols from /lib64/libnghttp2.so.14...Reading symbols from /lib64/libnghttp2.so.14...(no debugging symbols 
found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnghttp2.so.14
Reading symbols from /lib64/libdnet.1...Missing separate debuginfo for /lib64/libdnet.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/cd/00c325aa44135552d31222ba244cb8f07fb761.debug
Reading symbols from /lib64/libdnet.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdnet.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libpcre.so.1.2.0.debug...done.
done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libnsl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libm-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from 
/usr/lib/debug/usr/lib64/libcrypto.so.1.0.2k.debug...done.
done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libsfbpf.so.0...Reading symbols from /lib64/libsfbpf.so.0...(no debugging symbols 
found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsfbpf.so.0
Reading symbols from /lib64/libpcap.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libpcap.so.1.5.3.debug...done.
done.
Loaded symbols for /lib64/libpcap.so.1
Reading symbols from /lib64/libz.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libz.so.1.2.7.debug...done.
done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libpthread.so.0...Reading symbols from 
/usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
done.
[New LWP 1377]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...Reading symbols from 
/usr/lib/debug/usr/lib64/libnss_files-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/snort-2.9.11_dynamicengine/libsf_engine.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicengine/libsf_engine.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dce2_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dce2_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dnp3_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dnp3_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dns_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_dns_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ftptelnet_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ftptelnet_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_gtp_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_gtp_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_imap_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_imap_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_modbus_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_modbus_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_pop_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_pop_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_reputation_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_reputation_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sdf_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sdf_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sip_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_sip_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_smtp_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_smtp_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssh_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssh_preproc.so
Reading symbols from /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssl_preproc.so...done.
Loaded symbols for /usr/lib64/snort-2.9.11_dynamicpreprocessor//libsf_ssl_preproc.so
0x00007ff26b57da3d in poll () at ../sysdeps/unix/syscall-template.S:81
81        T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
Missing separate debuginfos, use: debuginfo-install snort-2.9.11-1.x86_64 daq-2.0.6-1.x86_64 
libnghttp2-1.21.1-1.el7.x86_64

when I run debuginfo-install

Could not find debuginfo for main pkg: 1:snort-2.9.11-1.x86_64
Could not find debuginfo pkg for dependency package libnghttp2-1.21.1-1.el7.x86_64
Could not find debuginfo pkg for dependency package daq-2.0.6-1.x86_64

(Is there a separate repo for debuginfo for snort?)

I also notice that when the process restarts and hangs there are different parameters running (I am running systemd 
start with parameters /sbin/snort -q -u snort -g snort -c /etc/snort/snort.conf -i eth0 -  while the restarted process 
is using those /usr/sbin/snort -A fast -b -d -D -i eth0 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort)


Thank you

Anna


On 17 Oct 2017, at 14:43, Carter Waxman (cwaxman) <cwaxman () cisco com<mailto:cwaxman () cisco com>> wrote:

Can you provide the OS and C runtime / build chain you are using? Also, can you attach gdb to the stopped process and 
send the backtrace?

Thanks,
Carter

From: Snort-users <snort-users-bounces () lists snort org<mailto:snort-users-bounces () lists snort org>> on behalf of 
Anna <Anna () sonru com<mailto:Anna () sonru com>>
Date: Tuesday, October 17, 2017 at 7:21 AM
To: "snort-users () lists snort org<mailto:snort-users () lists snort org>" <snort-users () lists snort 
org<mailto:snort-users () lists snort org>>
Subject: [Snort-users] Problems with Snort 2.9.11.0 - Snort is terminating unexpectedly

Hello,

I have another problem now. I upgraded the 2.9.9.0 to 2.9.11 and the problem with snort using too much memory went 
away. It got replaced with Snort working for  few hours and terminating while the top is showing that snort is still 
running.

Snort was upgraded and started on 13th of Oct at 9:31 am

Oct 13 09:31:46 DEV_SERVER systemd: Started Snort NIDS Daemon.
Oct 13 09:31:46 DEV_SERVER systemd: Starting Snort NIDS Daemon...
Oct 13 09:31:56 DEV_SERVER kernel: device eth0 entered promiscuous mode

then on 14th of Oct at 3:21am it stopped and tried to restart (probably hanged that is why OS is still showing snort as 
running)

Oct 14 03:21:01 DEV_SERVER systemd: Stopping SYSV: snort is a lightweight network intrusion detection tool that 
currently detects more than 1100 host and network vulnerabilities, portscans, backdoors, and more....
Oct 14 03:21:02 DEV_SERVER snort: *** Caught Term-Signal
Oct 14 03:21:02 DEV_SERVER kernel: device eth0 left promiscuous mode
Oct 14 03:21:03 DEV_SERVER snortd: Stopping snort: [  OK  ]
Oct 14 03:21:03 DEV_SERVER systemd: Starting SYSV: snort is a lightweight network intrusion detection tool that 
currently detects more than 1100 host and network vulnerabilities, portscans, backdoors, and more....
Oct 14 03:21:03 DEV_SERVER snort[12509]: Running in IDS mode
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]:        --== Initializing Snort ==--
Oct 14 03:21:03 DEV_SERVER snort[12509]: Initializing Output Plugins!
Oct 14 03:21:03 DEV_SERVER snort[12509]: Initializing Preprocessors!
Oct 14 03:21:03 DEV_SERVER snort[12509]: Initializing Plug-ins!
Oct 14 03:21:03 DEV_SERVER snort[12509]: Parsing Rules file "/etc/snort/snort.conf"
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'HTTP_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 80:81 443 8000 8008 8080:8081 8085 8088 ]
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'SHELLCODE_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 0:79 81:65535 ]
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'ORACLE_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 1024:65535 ]
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'SSH_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 22 ]
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'FTP_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 21 2100 3535 ]
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'SIP_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 5060:5061 5600 ]
Oct 14 03:21:03 DEV_SERVER snort[12509]:
Oct 14 03:21:03 DEV_SERVER snort[12509]: PortVar 'FILE_DATA_PORTS' defined :
Oct 14 03:21:03 DEV_SERVER snort[12509]: [ 80:81 110 143 443 8000 8008 8080:8081 8085 8088 ]


This is happening to all servers I upgraded so this is not isolated incident.


How to put the snort activity logs in debug and redirect them from messages (not the u2 logs that are going to 
barnyard)? Also when the Snort is about the terminate those logs are created snort.log.1507951274 (there is not -l flag 
in the start command or any other output files in snort.conf) they contain the detected traffic logs
^@^D<8a>Óï^@PKr^W? ô<82>\<80>^X^O¦^MÉ^@^@^A^A^H
^Y)×<9e><94>Íg­HEAD http://52.125.242.20:80/PMA/<http://52.125.242.20/PMA/> HTTP/1.1^M
Connection: Keep-Alive^M
Keep-Alive: 300^M
User-Agent: Mozilla/5.0 Jorgee^M
Host: 52.125.242.20^M

Alerts log is updated with the alerts from Snort that should go to barnyard2/Snorby

10/17-11:12:09.240501  [**] [120:3:1] (http_inspect) NO CONTENT-LENGTH OR TRANSFER-ENCODING IN HTTP RESPONSE [**] 
[Classification: Unknown Traffic] [Priority: 3] {TCP} 10.0.4.224:80 -> 10.0.4.138:46108

But Snort is not running according to systemd since Oct 14 03:21:02 DEV_SERVER snort[32171]: *** Caught Term-Signal

When I try to start snort with systemd without killing the one that is “running” according to OS, they are two 
processes running then.

Anybody can shed any light in what is going on with my upgraded Snort?

Thank you

Anna




_______________________________________________
Snort-users mailing list
Snort-users () lists snort org<mailto:Snort-users () lists snort org>
Go to this URL to change user options or unsubscribe:
https://lists.snort.org/mailman/listinfo/snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!

_______________________________________________
Snort-users mailing list
Snort-users () lists snort org
Go to this URL to change user options or unsubscribe:
https://lists.snort.org/mailman/listinfo/snort-users

Please visit http://blog.snort.org to stay current on all the latest Snort news!

Please follow these rules: https://snort.org/faq/what-is-the-mailing-list-etiquette