Snort mailing list archives
RE: Question about best hardware
From: "Kreimendahl, Chad J" <Chad.Kreimendahl () umb com>
Date: Tue, 9 Mar 2004 10:27:36 -0600
I have yet to try MaxDB, so as far as the 100k events problem happening with this, I can't confirm. We currently have many many millions of records in our archive. Though performance isn't much of a problem on our DB with this many, we chose to move off old data into an archive (actually copies off realtime, so the archive is our all-time db)... And keep the recent stuff in a 'real-time' database that users can access and perform complex queries through a somewhat useful web interface. All of the performance problems we've ever run into have related directly to the joining of many tables [>4]. Our resolution to this problem is a table design that requires less joins, and overall doesn't use more space. We're currently working on (and could use help with) a new output plugin that works with this new DB design. Meeting time... -----Original Message----- From: Hutchinson, Andrew [mailto:andrew.hutchinson () Vanderbilt Edu] Sent: Monday, March 08, 2004 4:21 PM To: Jason Haar Cc: snort-users () lists sourceforge net Subject: RE: [Snort-users] Question about best hardware snortdb=# select count(*) from event; count -------- 514109 (1 row) In my "catch-all" database (I keep a catch-all db for forensic reasons, so I can go back and look at every little alert if necessary, and a "day-to-day" db, for common exploits), I currently have about half a million records, and generally archive when I reach around 4 million alerts. Performance is not terrible (it takes 30 seconds or so to find events of interest when the db approaches 4 millions recs), but I'm using Postgresql and not using ACID (I wanted some different capabilities, multiple user access levels, etc. so I made muy own interface). MySQL is generally faster than Postgresql though, and should be able to handle WAY more than 100K records w/o any problem. As I mentioned, I am currently using Postgresql. However, MySQL4 and MaxDB have sparked anew an interest in MySQL, so I've been playing around with it a bit lately. Perhaps a good way to run down the problem would be to turn on slow-query logging (with the --log-slow-queries option at MySQL startup), and then run those same queries interactively against the db with the EXPLAIN keyword to see what's causing them to be slow. I'd do this, but again I don't currently use MySQL or ACID... HTH, Andrew
On Sat, 2004-03-06 at 07:07, Kreimendahl, Chad J wrote:Hardware won't be your problem. Once you get around 100k events in the snortdb on MySQL you'll run into major performance problems that almost no amount of hardware seems to solve.You're dead right there. 100K does appear to be the limit for me too. Is this a MySQL-specific issue? How does Postgresql or Oracle handle DBs over 100K? Has anyone tried to figure out the problem? There are apparently people using MySQL with terrabytes of data (nothing to do with snort),
so why is 100K of snort records such a big deal? Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users
Thanks, Josh Berry, CISSP CTO, VP of Product Development LinkNet-Solutions 469-831-8543 josh.berry () linknet-solutions com ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=ick _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=ort-users ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users
Current thread:
- Re: Question about best hardware, (continued)
- Re: Question about best hardware Sigurd Urdahl (Mar 12)
- RE: Question about best hardware SN ORT (Mar 08)
- RE: Question about best hardware Michael Miller (Mar 08)
- RE: Question about best hardware Josh Berry (Mar 10)
- RE: Question about best hardware AJ Butcher, Information Systems and Computing (Mar 11)
- RE: Question about best hardware Josh Berry (Mar 10)
- RE: Question about best hardware Hutchinson, Andrew (Mar 08)
- Re: Question about best hardware Jason Haar (Mar 08)
- pcre.h error . David Alonso De La Vega Tapage (Mar 08)
- creating tables .. David Alonso De La Vega Tapage (Mar 09)
- Re: creating tables .. David Alonso De La Vega Tapage (Mar 09)
- Re: Question about best hardware Jason Haar (Mar 08)