Re: Announce: FLoP-1.0 --- Fast Logging Project for snort

[Dirk Geschke <Dirk () geschke-online de>]

Hi Bamm,

> I hope you don't think I was attacking flop. I actually want 
> to take a close look at what you've done and also talk to you
> about supporting sguil (http://sguil.sf.net) and its DB schema. 

I did not assume that you wanted to attack the project but there
were some points which needed some comments.

> My only intent was to start a discussion of the main disadvantage 
> of using the unix socket plugin with snort: that if for any reason,
> the process reading the unix socket dies, any alerts that snort 
> writes while the proc is down, is gone for ever. In short, by using 
> the unix socket, you've added another potential point of failure to 
> your IDS. Please understand, that I believe that risk is fairly low
> (at least for a well written/tested program), but there are those 
> that may not be willing to accept that risk, just like there are those 
> that won't entertain the idea of an inline IDS (versus passive monitoring) 
> for the sole reason a inline IDS brings another point of failure into the 
> network. 

With any solution you introduce some possible problems. Think either of
a hanging snort waiting for the output plugins and therefore simply
missing some packets to sniff. Or think of a filled up filesystem where
no process is able to write any byte which could have just more and 
worser side effects. With unix domain sockets you can add an extra
layer of security. I just mentioned it: The output-plugin realizes
if a process is reading on the socket or not. So if writing to the
socket fails you can start another output plugin or simply write
the data in a file.

The problem here is: You need a mechanism how you want to handle this
special situation. And much worser: How to watch for the process which
waits for data in this special file...

> Now that that is said, can you answer a couple of questions (why RTFM when
> I have your ear ;) ).

> From the documentation I did read, it looks like if servsock (the process 
> that recieves data from the sockserv) dies or becomes unreachable for an 
> extended period of time, then sockserv (the proc that reads the unix sock) 
> will die. I saw the experimental 'drop' feature, but I am wondering if you 
> had any plans to instead start writing the events to disk and then be able 
> to read those alerts back into sockserv and send them off once servsock 
> becomes available again? 

Okay, this can lead to problems in two sitution. Either when sockserv is
started or if an alert arises. If there is no servsock process listening
the program tries MaxTry times to connect to servsock. Between two tries
the process waits WaitTime seconds. This values are set by default to
10, which is quite small but can be adjusted with the command line
options -M and -W.

The reason why this values are so small by default is to make it easier
to test it. And the main reason why I don't use an infinite loop is to
have the possibility to inform someone of the problem.(I simply assume
that there is a job in the background surveying the processes. How else
will you notice that there is a problem? But in principle you can choose
high values for this parameters.)

The drop feature was introduced as a "stability" factor. Think of a lot
of buffered alerts and no servsock. Then the memory usage will increase
until malloc() fails. The result would be a dying sockserv and all
alerts are lost. With drop you can get a short information of wich
alerts are logged. This contains of the signature, the IP addresses
and ports (if ther are any) involved in the process together with the
timestamp. This values are either send to a list of recipients and if
this is not possible the alerts are written to syslog or stdout. (Think
of a network problem: No E-Mail can be sent so we write to stdout or 
syslog.) So effectively you only loose some network headers and the
payload.

And finally: Yes, first I thought on a temporarily buffering of alerts
by writing them to disk. But then you have the same problems we already
discussed with barnyard. Additionally there is a further problem: When
and how will you reload these alerts? Will you stop reading from the
unix socket until all data is read in from disk? Or should we create
a third thread, mixing up with the other by storing the alerts again
in memory? Think of a disturbed network where you can't reach the
cental server. All alerts will fill up memory, where dumped to disk
and reread until you have to redump them again? And what is with the
alerts ariving from snort inbetween?

So there are a lot of problems with this approach.

If you assume that sockserv will not die due to an unknown behaviour
like power shortage or something else. Then let us start a small
calculation: The alert information together with the payload of a
full saturated (MTU) packet is less than 3 kB of memory (to be more
precisely 1360 Byte + 1514 Bytes). Now memory is quite cheap so I 
assume you may have 300 MB (it's easier to calculate) of memory
available solely for sockserv. Then you can buffer up to 100,000 
alerts until you are running into problems. And this is for the case
that all alerts use the maximum MTU size of Ethernet...

A problem could arise if you have to restart or replace sockserv.
But I guess you won't do this on a heavy attack...

> Also, any plans to do much of the same thing with servsock (like 
> if the DB somehow become unavailable for an extended length of time)? 
> I can think of a couple of circimstances that may cause either the DB 
> and/or servsock to become 'unavailable' (such as a server reboot, DB 
> cleaning, etc). This also adds more points of failure, where if one 
> component dies, events could be lost to /dev/null.

Hmm, this is a little bit more complicated. I don't know how I can
identify problems with the database connection within servsock. But
the good thing is: Good databases will never die ;-)
In fact databases will rarely die and my project assumes that the
database is running on the same host as servsock. Otherwise we can't
feed the database via an unix socket... So a reboot is no issue. DB
cleaning could be: but this can be done online. Then maybe servsock 
will work a little bit slower...

> Anyway, the project definately looks cool and I am glad see there 
> is a real alternative to unified out and barnyard out there. 

Fine, give it a try and give me a note how it worked...

Finally one note to sguil: I just took a look at the project site
and TCL/TK is not really my world. But if I understand it the right
way than it seems to be a more complex system than FLoP...
Which database design are you using? I think it is the same one as
of snort/ACID? Did you ever think about the possibility to store the
whole pcap data in the database? This should make it unnecessary to
store them on disk on a separate way.

Best regards

Dirk





-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
Does SourceForge.net help you be more productive?  Does it
help you create better code?  SHARE THE LOVE, and help us help
YOU!  Click Here: http://sourceforge.net/donate/
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users