To discuss: FLoP and missing database

[Dirk Geschke <Dirk () geschke-online de>]

Hi all,

I received some concerns about the problem what would 
happen if the database is not available. 

Maybe someone has some good ideas how this should
be handled. (Note: As far as I can oversee it, neither
the database output plugin of snot nor barnyard did
really count on this problem. The database is assumed 
to run every time and forever. BTW: FLoP does the same 
up to now...)

The actual problem is:

There is no check for the availability of the database.

So: What happens if the database is not available?

Up to FLoP-1.0:
---------------
It is assumed that the database is available. If it fails,
all errors are ignored. An error message is printed but
that's all. 

So if noone implements checks for this messages all
works and nothing will be logged. Not a good solution.

FLoP-1.0.1: (ready, but not released yet -> needs testing)
-----------
There is a check for the availability of the database
if a remote sensor (sockserv) connects.
A handshake is implemented, if the database is not
available an error code is sent to the sockserv
process.
On an error: Depending on the return code the sockserv
process will retry a connection later on (recoverable
error) or will exit (unrecoverable database error).

(Further we check the endianess. These must now be
identical on server and remote sensor. This is the
first step towards a mixed environment but is not
related to the database...)

Version FLoP-1.xxx: (not started yet)
-------------------
Real Problem: What happens if the database is dying
during a session (after the initialization)?

Up to now (Version 1.0.1) we check only for the database
on a connect of a remote sensor. If the database fails
after this time we are at the old problem...

Possible solution:

+ We check for the database availabilty or failure
  of any INSERT statement (this is ignored yet).

+ If a INSERT fails we close the connection to the 
  remote sensor (aka sockserv). So no further alerts
  will be accepted in this session.

+ The sockserv process will try to reconnect. This 
  may fail for two reasons: 

  1. There are still some old alerts which have to be
     INSERTed first (yes this is checked!)

  2. The database is not available.

+ Depending on the problem (see above) sockserv will
  try several times to reconnect.

So far all seems to be easy. Now we have to think
about the alerts already spooled to servsock which
may be still in memory but not INSERTed in the
database. Here we could implement:

+ Open a file: database.sensorname (maybe should be done
  on startup?)
+ Write all INSERT statements of the failing alert to
  this file.
+ Write all INSERT statements of the remaining alerts
  to this file.
+ Close this file and do an exit.
+ Any further connects of sockserv processes are handeld
  as in version FLoP-1.0.1 (see above)

The advantage of this method:

+ We get an ASCII file of statements which can be sourced
  manually into the database.

and the disadvantage:

+ The cid is counted for each alert and is part of the
  INSERT statement (the cid is part of the combined 
  primary index). If the database is available again,
  the sockserv process re-connects and inserts alerts
  these cid values will overlap.

+ So: We have to source this file *before* the correspondent
  sockserv is starting to insert new alerts. 

+ Big Question: how and when?

+ Ok, we can check the file manually and then remove
  it if no important alert is found.

Another solution would be in writing the data binary
to a file. This data can be read in if the client
connects and the database is available. Thus we
first read in the data from the file, remove this
file and then start the normal, threaded, process.

Advantage:

+ we don't have to care about the cid
+ the data are inserted in right order

Disadvantage:

+ it's more difficult to implement.

Does anyone have a better idea, some comments 
or simply a good argument what should be implemented?

Best regards

Dirk Geschke
---
http://www.geschke-online.de/FLoP/



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users