CID duplication issues with ACID, snort and multiple sensors

[W Fenwick <wfenwick () entrenet com>]

Hi List,

We have spent the better part of a day doing in-depth debugging on this
problem and have had to resort to an email to the list.We have a
snort/ACID
combination which has been running for several months without much of a
hitch. Recently we added a second sensor with Snort 1.8.3 and it began
logging to the central Postgres database also. Everything has been
working
well, until yesterday.

First of all, we upgraded ADODB to v1.72 on Tuesday from ADODB 1.20.

The fellow who usually does the operational monitoring was away
Wednesday,
so I took over for the day and noticed that archival was partially
failing.

Added 59 alert(s) to the Alert cache

Ignored 414 duplicate alert(s)

Successful ARCHIVE-move - 84 alert(s)

This seemed odd. I wasnt sure what was going on. I later started a new
session and recieved this error on pointing at the home URL for my ACID
install.

It was complaining

sensor #5: event.cid = 2391, acid_event.cid = 0 Database ERROR:Database
ERROR:ERROR: Cannot
insert a duplicate key into unique index acid_event_pkey

INSERT INTO acid_event (sid,cid,signature,timestamp,
ip_src,ip_dst,ip_proto, sig_name,
sig_priority, sig_class_id) SELECT event.sid as sid, event.cid as cid,
signature, timestamp,
ip_src, ip_dst, ip_proto, sig_name, sig_priority, sig_class_id FROM
event
INNER JOIN signature
ON (signature = signature.sig_id) LEFT JOIN iphdr ON
(event.sid=iphdr.sid
AND
event.cid=iphdr.cid) WHERE (sig_name LIKE 'spp_%') AND (event.sid = 5
AND
event.cid > 0)

INSERT INTO acid_event (sid,cid,signature,timestamp,
                              ip_src,ip_dst,ip_proto,
                              sig_name, sig_priority, sig_class_id)
      SELECT event.sid as sid, event.cid as cid, signature, timestamp,
             ip_src, ip_dst, ip_proto, sig_name, sig_priority,
sig_class_id
      FROM event
       INNER JOIN signature ON (signature = signature.sig_id)
      LEFT JOIN iphdr ON (event.sid=iphdr.sid AND event.cid=iphdr.cid)
      WHERE  (sig_name LIKE 'spp_%')  AND
      (event.sid = 5 AND event.cid > 0)


Obviously we thought this might be an issue so we rolled back without
success. A search of the web didn't reveal much. Something has been
introduced by that or another element that we are unabled to diagnose
and cure manually.

So then I thought that maybe the alert cache was confused, so I
attempted to
rebuild it. It failed to rebuild it. So I turned on debug mode in ACID
and
in PHP (log sql) and had a look at how
the caching gets built.

<logs omitted for brevity - if you can help I'll send them along>

So then I started digging with psql into the database to see how the
dupes
could be invading the database.
I'm a UNIX security/firewall/IDS guy, not a database guy, so bear with
me.
I checked out the incoming alert database and
although there are duplicate cid numbers in the alert table, they are
each
from different Sensors (sensor 1 and sensor 5). I confirmed
in the schema that the SID and CID are together uniquifiers for the
event
and acid_event table.

So I am trying to figure out what might have happened. Our process here
is
that we try to keep our snortnew database cleared;
we triage and archive them to our archive database once they are looked
at. If they are an incident we email them out to
ourselves and archive-move them. This is within the norm of what others
do
I think, and I haven't been able to turn up anything
that would cause this to be problematic.

I can see it being how we added the second sensor. It may have been a
bit
haphazard; you can see that because our sensors are
numbered 1 and 5 :) However, the SQL indicates the problem is not with
sensor 5 (most recent) but sensor 1!

hcsnortnew=# select * from event where cid=2391;
 sid | cid  | signature |       timestamp
-----+------+-----------+------------------------
   5 | 2391 |        31 | 2002-03-14 10:34:23-05

So then a bit later I did a

hcsnortnew=# SELECT MAX(cid) FROM event WHERE sid=1;
  max
--------
 733646
(1 row)

hcsnortnew=# SELECT MAX(cid) FROM event WHERE sid=5;
 max
------
 2396
(1 row)

Hmmm... That's pretty close to 2391! And my incoming database is being
updated, so I tried to rebuild the cache again.

sensor #5: event.cid = 2396, acid_event.cid = 0 Database ERROR:Database
ERROR:ERROR: Cannot
insert a duplicate key into unique index acid_event_pkey

INSERT INTO acid_event (sid,cid,signature,timestamp,
ip_src,ip_dst,ip_proto, sig_name,
sig_priority, sig_class_id) SELECT event.sid as sid, event.cid as cid,
signature, timestamp,
ip_src, ip_dst, ip_proto, sig_name, sig_priority, sig_class_id FROM
event
INNER JOIN signature
ON (signature = signature.sig_id) LEFT JOIN iphdr ON
(event.sid=iphdr.sid
AND
event.cid=iphdr.cid) WHERE (sig_name LIKE 'spp_%') AND (event.sid = 5
AND
event.cid > 0)

INSERT INTO acid_event (sid,cid,signature,timestamp,
                              ip_src,ip_dst,ip_proto,
                              sig_name, sig_priority, sig_class_id)
      SELECT event.sid as sid, event.cid as cid, signature, timestamp,
             ip_src, ip_dst, ip_proto, sig_name, sig_priority,
sig_class_id
      FROM event
       INNER JOIN signature ON (signature = signature.sig_id)
      LEFT JOIN iphdr ON (event.sid=iphdr.sid AND event.cid=iphdr.cid)
      WHERE  (sig_name LIKE 'spp_%')  AND
      (event.sid = 5 AND event.cid > 0)

How about that... The error is on the same CID as the current highest
CID
number for sensor #5. I looked at the code, which indicated that BOTH
SID and CID are uniquifiers for the event database but I couldn't find
the
same information about the acid_event database.

So for now I am stuck on this. Sorry to bother you all about it but it
seems
like there is some sort of exceptional condition that's not
handled by ACID when it creates its cache from alerts that are
constructed
from multiple sensors. However, knowing that many people run lots of
sensors
pointed at a single database, I hardly think it's the code - it's gotta
be
something I did.

I need to get unstuck to determine what I did to create this exceptional

condition.

Any help is great. Thanks,

Wynn


_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users