Re: FW: memory corruption in 2.8.6

["Safwat Fahmy" <safwat.fahmy () safemedia com>]

Russ:

 

Thank you for the  response: please see below my answers:

 

The conf file is attached:

 

Thank you

Safwat

 

 

From: Russ Combs [mailto:rcombs () sourcefire com] 
Sent: Thursday, April 29, 2010 6:58 AM
To: Safwat Fahmy
Cc: Snort-users () lists sourceforge net; Lawrence R. Hughes, Sr.
Subject: Re: FW: [Snort-users] memory corruption in 2.8.6

 

Some ideas:

 

1.      Are you sure you are running the snort you installed?  Probably a
typo but the command line below does not match the "glibc detected"
path.[Safwat Fahmy]  I do not understand why using the same startup line
without the D option ( for demon) look for glibc. If I use the same command
line WITHOUT the D option but with the "&" for background snort works like a
charm..the demon mode is causing this problem. I am running as root.

 

[Safwat Fahmy] 

Anyway this is the ldd results on snort/ it might help:

/mnt/smlog/snort286inline/bin# ldd snort

        linux-vdso.so.1 =>  (0x00007fff225fe000)

        libdnet.1 => /usr/lib64/../lib64/libdnet.1 (0x00002b4e8872b000)

        libmysqlclient.so.16 => /usr/lib64/mysql/libmysqlclient.so.16
(0x00002b4e

        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4e88bb8000)

        libstdc++.so.6 => /usr/lib64/../lib64/libstdc++.so.6
(0x00002b4e88dd2000)

        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00002b4e890cd000)

        libz.so.1 => /lib64/libz.so.1 (0x00002b4e89301000)

        libpcre.so.0 => /usr/lib64/../lib64/libpcre.so.0
(0x00002b4e89516000)

        libnsl.so.1 => /lib64/libnsl.so.1 (0x00002b4e89739000)

        libm.so.6 => /lib64/libm.so.6 (0x00002b4e8994f000)

        libdl.so.2 => /lib64/libdl.so.2 (0x00002b4e89bcf000)

        libc.so.6 => /lib64/libc.so.6 (0x00002b4e89dd2000)

        /lib64/ld-linux-x86-64.so.2 (0x00002b4e88512000)

        libgcc_s.so.1 => /usr/lib64/../lib64/libgcc_s.so.1
(0x00002b4e8a107000)

 

2.  Did you set ulimit -c unlimited for the user running Snort.  [Safwat
Fahmy] yes (Do you have a set_uid in your conf?)[Safwat Fahmy]  no

 

2.      Your core may be named core.<pid>.[Safwat Fahmy]  we operate in a
read-only kernel.this is why I did define a path for the cor.pid but its not
any place

 

4.  If all else fails, you can try posting your conf.[Safwat Fahmy]
attached in the email

On Wed, Apr 28, 2010 at 6:05 PM, Safwat Fahmy <safwat.fahmy () safemedia com>
wrote:

Thank you Russ:

 

I used  --enable-debug with --enable-corefiles and I even added a path for
thje corefiles. I configeure, make clea, make, make install with no errs.

 

I started snortinline with /snort286inline/bin/snort -QDc
/mnt/smlog/snort286inline/etc/snort.conf -l /mnt/smlog/logs br0.

 

Snort very diligently gave the error:

 

Initializing Inline mode

building cached socket reset packets

*** glibc detected *** /mnt/smlog/snort286inline/bin/snort: malloc(): memory
corruption: 0x000000000149ece0 ***

======= Backtrace: =========

/lib64/libc.so.6[0x2ba5baf8a1cc]

/lib64/libc.so.6[0x2ba5baf8c3bf]

/lib64/libc.so.6(__libc_malloc+0x73)[0x2ba5baf8d8fc]

/lib64/libc.so.6(open_memstream+0x1a)[0x2ba5baf84376]

/lib64/libc.so.6(__vsyslog_chk+0x81)[0x2ba5bafd8bf8]

/lib64/libc.so.6(syslog+0x90)[0x2ba5bafd9225]

/mnt/smlog/snort286inline/bin/snort[0x43814a]

/mnt/smlog/snort286inline/bin/snort[0x43547b]

/mnt/smlog/snort286inline/bin/snort[0x42ef39]

/mnt/smlog/snort286inline/bin/snort[0x42ef17]

/lib64/libc.so.6(__libc_start_main+0xe3)[0x2ba5baf3eaf3]

/mnt/smlog/snort286inline/bin/snort[0x404cb9]

 

It seems that debug and corefiles are broken also

 

Any other ideas???

 

Thank you

Safwat

 

From: Russ Combs [mailto:rcombs () sourcefire com] 
Sent: Wednesday, April 28, 2010 5:23 PM
To: Safwat Fahmy
Cc: Snort-users () lists sourceforge net; Lawrence R. Hughes, Sr.
Subject: Re: FW: [Snort-users] memory corruption in 2.8.6

 

Ah, my bad.  You need to add --enable-debug with --enable-corefiles.

The backtrace you get should then include source symbols.

Thanks
Russ

On Wed, Apr 28, 2010 at 5:10 PM, Safwat Fahmy <safwat.fahmy () safemedia com>
wrote:

Russ:

 

Although I defined a path for the corefiles and reconfigured, make and make
install with no errors: I did not get a corefileor backtrace although snort
crashed.

 

 I have no documentation for corfiles or backtrace  for snort any where?

 

This is the only information I can provide: the terminal output with the
error, configuration,  startup command line, and my config file

 

Snort configure , and snort.cnf are included as attachments.

_-_---_____

Startup command line: 

snort286inline/bin/snort -QDc /mnt/smlog/snort286inline/etc/snort.conf -l
/mnt/smlog/logs br0

_____-__-_-

Terminal output:

 

Initializing Inline mode

building cached socket reset packets

*** glibc detected *** /mnt/smlog/snort286inline/bin/snort: malloc(): memory
corruption: 0x000000000143ece0 ***

======= Backtrace: =========

/lib64/libc.so.6[0x2af38c89b1cc]

/lib64/libc.so.6[0x2af38c89d3bf]

/lib64/libc.so.6(__libc_malloc+0x73)[0x2af38c89e8fc]

/lib64/libc.so.6(open_memstream+0x1a)[0x2af38c895376]

/lib64/libc.so.6(__vsyslog_chk+0x81)[0x2af38c8e9bf8]

/lib64/libc.so.6(syslog+0x90)[0x2af38c8ea225]

/mnt/smlog/snort286inline/bin/snort[0x428ff5]

/mnt/smlog/snort286inline/bin/snort[0x4247f8]

/mnt/smlog/snort286inline/bin/snort[0x425c94]

/lib64/libc.so.6(__libc_start_main+0xe3)[0x2af38c84faf3]

/mnt/smlog/snort286inline/bin/snort[0x4048a9]

======= Memory map: ========

00400000-00506000 r-xp 00000000 16:01 9666566
/mnt/smlog/snort286inline/bin/snort

00705000-00708000 rw-p 00105000 16:01 9666566
/mnt/smlog/snort286inline/bin/snort

00708000-0144b000 rw-p 00708000 00:00 0
[heap]

2af38af72000-2af38af8a000 r-xp 00000000 07:00 1449
/lib64/ld-2.6.so <http://ld-2.6.so/> 

2af38af8a000-2af38af8c000 rw-p 2af38af8a000 00:00 0

2af38b189000-2af38b18a000 r--p 00017000 07:00 1449
/lib64/ld-2.6.so <http://ld-2.6.so/> 

2af38b18a000-2af38b18b000 rw-p 00018000 07:00 1449
/lib64/ld-2.6.so <http://ld-2.6.so/> 

2af38b18b000-2af38b197000 r-xp 00000000 07:00 5567
/usr/lib64/libdnet.1.0.1

2af38b197000-2af38b396000 ---p 0000c000 07:00 5567
/usr/lib64/libdnet.1.0.1

2af38b396000-2af38b398000 rw-p 0000b000 07:00 5567
/usr/lib64/libdnet.1.0.1

2af38b398000-2af38b39b000 rw-p 2af38b398000 00:00 0

2af38b39f000-2af38b3e6000 r-xp 00000000 07:00 6493
/usr/lib64/mysql/libmysqlclient.so.16.0.0

2af38b3e6000-2af38b5e6000 ---p 00047000 07:00 6493
/usr/lib64/mysql/libmysqlclient.so.16.0.0

2af38b5e6000-2af38b614000 rw-p 00047000 07:00 6493
/usr/lib64/mysql/libmysqlclient.so.16.0.0

2af38b614000-2af38b618000 rw-p 2af38b614000 00:00 0

2af38b618000-2af38b62c000 r-xp 00000000 07:00 1387
/lib64/libpthread-2.6.so <http://libpthread-2.6.so/> 

2af38b62c000-2af38b82b000 ---p 00014000 07:00 1387
/lib64/libpthread-2.6.so <http://libpthread-2.6.so/> 

2af38b82b000-2af38b82c000 r--p 00013000 07:00 1387
/lib64/libpthread-2.6.so <http://libpthread-2.6.so/> 

2af38b82c000-2af38b82d000 rw-p 00014000 07:00 1387
/lib64/libpthread-2.6.so <http://libpthread-2.6.so/> 

2af38b82d000-2af38b832000 rw-p 2af38b82d000 00:00 0

2af38b832000-2af38b913000 r-xp 00000000 07:00 5538
/usr/lib64/libstdc++.so.6.0.9

2af38b913000-2af38bb12000 ---p 000e1000 07:00 5538
/usr/lib64/libstdc++.so.6.0.9

2af38bb12000-2af38bb19000 r--p 000e0000 07:00 5538
/usr/lib64/libstdc++.so.6.0.9

2af38bb19000-2af38bb1b000 rw-p 000e7000 07:00 5538
/usr/lib64/libstdc++.so.6.0.9

2af38bb1b000-2af38bb2d000 rw-p 2af38bb1b000 00:00 0

2af38bb2d000-2af38bb32000 r-xp 00000000 07:00 1495
/lib64/libcrypt-2.6.so <http://libcrypt-2.6.so/> 

2af38bb32000-2af38bd31000 ---p 00005000 07:00 1495
/lib64/libcrypt-2.6.so <http://libcrypt-2.6.so/> 

2af38bd31000-2af38bd32000 r--p 00004000 07:00 1495
/lib64/libcrypt-2.6.so <http://libcrypt-2.6.so/> 

2af38bd32000-2af38bd33000 rw-p 00005000 07:00 1495
/lib64/libcrypt-2.6.so <http://libcrypt-2.6.so/> 

2af38bd33000-2af38bd61000 rw-p 2af38bd33000 00:00 0

2af38bd61000-2af38bd75000 r-xp 00000000 07:00 1439
/lib64/libz.so.1.2.3

2af38bd75000-2af38bf74000 ---p 00014000 07:00 1439
/lib64/libz.so.1.2.3

2af38bf74000-2af38bf75000 rw-p 00013000 07:00 1439
/lib64/libz.so.1.2.3

2af38bf75000-2af38bf76000 rw-p 2af38bf75000 00:00 0

2af38bf76000-2af38bf98000 r-xp 00000000 07:00 5519
/usr/lib64/libpcre.so.0.0.1

2af38bf98000-2af38c198000 ---p 00022000 07:00 5519
/usr/lib64/libpcre.so.0.0.1

2af38c198000-2af38c199000 rw-p 00022000 07:00 5519
/usr/lib64/libpcre.so.0.0.1

2af38c199000-2af38c1ac000 r-xp 00000000 07:00 1409
/lib64/libnsl-2.6.so <http://libnsl-2.6.so/> 

2af38c1ac000-2af38c3ab000 ---p 00013000 07:00 1409
/lib64/libnsl-2.6.so <http://libnsl-2.6.so/> 

2af38c3ab000-2af38c3ac000 r--p 00012000 07:00 1409
/lib64/libnsl-2.6.so <http://libnsl-2.6.so/> 

2af38c3ac000-2af38c3ad000 rw-p 00013000 07:00 1409
/lib64/libnsl-2.6.so <http://libnsl-2.6.so/> 

2af38c3ad000-2af38c3af000 rw-p 2af38c3ad000 00:00 0

2af38c3af000-2af38c42d000 r-xp 00000000 07:00 1407
/lib64/libm-2.6.so <http://libm-2.6.so/> 

2af38c42d000-2af38c62c000 ---p 0007e000 07:00 1407
/lib64/libm-2.6.so <http://libm-2.6.so/> 

2af38c62c000-2af38c62d000 r--p 0007d000 07:00 1407
/lib64/libm-2.6.so <http://libm-2.6.so/> 

2af38c62d000-2af38c62e000 rw-p 0007e000 07:00 1407
/lib64/libm-2.6.so <http://libm-2.6.so/> 

2af38c62e000-2af38c62f000 rw-p 2af38c62e000 00:00 0

2af38c62f000-2af38c631000 r-xp 00000000 07:00 1393
/lib64/libdl-2.6.so <http://libdl-2.6.so/> 

2af38c631000-2af38c830000 ---p 00002000 07:00 1393
/lib64/libdl-2.6.so <http://libdl-2.6.so/> 

2af38c830000-2af38c831000 r--p 00001000 07:00 1393
/lib64/libdl-2.6.so <http://libdl-2.6.so/> 

2af38c831000-2af38c832000 rw-p 00002000 07:00 1393
/lib64/libdl-2.6.so <http://libdl-2.6.so/> 

2af38c832000-2af38c95e000 r-xp 00000000 07:00 1433
/lib64/libc-2.6.so <http://libc-2.6.so/> 

2af38c95e000-2af38cb5d000 ---p 0012c000 07:00 1433
/lib64/libc-2.6.so <http://libc-2.6.so/> 

2af38cb5d000-2af38cb61000 r--p 0012b000 07:00 1433
/lib64/libc-2.6.so <http://libc-2.6.so/> 

2af38cb61000-2af38cb62000 rw-p 0012f000 07:00 1433
/lib64/libc-2.6.so <http://libc-2.6.so/> 

2af38cb62000-2af38cb67000 rw-p 2af38cb62000 00:00 0

2af38cb67000-2af38cb73000 r-xp 00000000 07:00 5339
/usr/lib64/libgcc_s.so.1

2af38cb73000-2af38cd73000 ---p 0000c000 07:00 5339
/usr/lib64/libgcc_s.so.1

2af38cd73000-2af38cd74000 rw-p 0000c000 07:00 5339
/usr/lib64/libgcc_s.so.1

2af38cd74000-2af38cd76000 rw-p 2af38cd74000 00:00 0

2af38cd76000-2af38cd7f000 r-xp 00000000 07:00 1426
/lib64/libnss_files-2.6.so <http://libnss_files-2.6.so/> 

2af38cd7f000-2af38cf7f000 ---p 00009000 07:00 1426
/lib64/libnss_files-2.6.so <http://libnss_files-2.6.so/> 

2af38cf7f000-2af38cf80000 r--p 00009000 07:00 1426
/lib64/libnss_files-2.6.so <http://libnss_files-2.6.so/> 

2af38cf80000-2af38cf81000 rw-p 0000a000 07:00 1426
/lib64/libnss_files-2.6.so <http://libnss_files-2.6.so/> 

2af38cf81000-2af38db8f000 rw-p 2af38cf81000 00:00 0

2af390000000-2af390021000 rw-p 2af390000000 00:00 0

2af390021000-2af394000000 ---p 2af390021000 00:00 0

7fff1fb23000-7fff1fb38000 rw-p 7ffffffea000 00:00 0
[stack]

7fff1fbfe000-7fff1fc00000 r-xp 7fff1fbfe000 00:00 0
[vdso]

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]

./sips286inline.sh: line 2:  6222 Aborted
/mnt/smlog/snort286inline/bin/snort -QDc
/mnt/smlog/snort286inline/etc/snort.conf -l /mnt/smlog/logs br0

 

From: Safwat Fahmy [mailto:safwat.fahmy () safemedia com] 
Sent: Wednesday, April 28, 2010 4:14 PM
To: 'Russ Combs'
Cc: Snort-users () lists sourceforge net


Subject: Re: [Snort-users] memory corruption in 2.8.6

Importance: High

 

Russ:

 

should I define a path for the corefiles in snort configure??  We are
working off an embedded target which we do not compile on?? 

Thanks

 

From: Russ Combs [mailto:rcombs () sourcefire com] 
Sent: Wednesday, April 28, 2010 4:11 PM
To: Safwat Fahmy
Cc: jesler () sourcefire com; Snort-users () lists sourceforge net
Subject: Re: [Snort-users] memory corruption in 2.8.6

 

If you configure with --enable-corefiles you will get a core file when the
program crashes.  You may need to set `ulimit -c unlimited`.  You can then
open the core in a debugger to see the stack.  If you are using gdb, you can
do `gdb -c <corefile>` and then 'bt' at the command prompt.

On Wed, Apr 28, 2010 at 3:19 PM, Safwat Fahmy <safwat.fahmy () safemedia com>
wrote:

Russ

Where the backtrace file will be generated??

Thanks

 

 

 

From: Russ Combs [mailto:rcombs () sourcefire com] 
Sent: Wednesday, April 28, 2010 1:34 PM


To: Safwat Fahmy
Cc: jesler () sourcefire com; Snort-users () lists sourceforge net
Subject: Re: [Snort-users] memory corruption in 2.8.6

 

I'm unable to reproduce it.  Can reconfigure with --enable-corefiles and
send a backtrace please?

On Wed, Apr 28, 2010 at 1:27 PM, Safwat Fahmy <safwat.fahmy () safemedia com>
wrote:

Thank you Russ

 

Yes we are working with libnet 1.0.2a

 

Just a reminder 2.8.6 work perfectly in a sniffer mode. The problem occurs
only in inline mode running in the background. If I use the -Qvc the sig
error will not happen

Thanks

Safwat

 

From: Russ Combs [mailto:rcombs () sourcefire com] 
Sent: Wednesday, April 28, 2010 1:22 PM
To: Safwat Fahmy
Cc: jesler () sourcefire com; Snort-users () lists sourceforge net
Subject: Re: [Snort-users] memory corruption in 2.8.6

 

Might this be a libnet issue?  Are you sure you are linking with the correct
version for your platform?

On Wed, Apr 28, 2010 at 12:46 PM, Safwat Fahmy <safwat.fahmy () safemedia com>
wrote:

Running snort 2.8.6 with the flowing command line:

 

/snort286inline/bin/snort -QDc /mnt/smlog/snort286inline/etc/snort.conf -l
/mnt/smlog/logs br0

 

Result in the following error:

 

initializing Inline mode

building cached socket reset packets

** glibc detected *** /mnt/smlog/snort286inline/bin/snort: malloc(): memory
corruption: 0x000000000143ece0 ***

====== Backtrace: =========

 

 

This is the config options:

re --enable-build-dynamic-examples --enable-ipv6 --enable-gre
--enable-timestats --enable-perfprofiling --enable-inline
--enable-sourcefire --enable-aruba --enable-react --enable-flexresp2
--with-libpcap-libraries=/usr/lib64 --with-libpcre-libraries=/usr/lib64
--with-libipq-includes=/usr/include --with-libipq-libraries=/usr/lib
--with-libnet-includes=/usr/include --with-libnet-libraries=/usr/lib64
--with-dnet-libraries=/usr/lib64 --with-mysql=/usr/share/mysql
--with-mysql-includes=/usr/include/mysql
--with-mysql-libraries=/usr/lib64/Mysql

 

ip_queue and iptables_ filter were modprobe + iptables  -I FORWARD -j QUEUE

 

Can you help with this

 

Many thanks

Safwat

 

 


----------------------------------------------------------------------------
--

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users

Attachment: snort.conf
Description:

------------------------------------------------------------------------------

_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users