RE: Question about best hardware

["Kreimendahl, Chad J" <Chad.Kreimendahl () umb com>]

I have yet to try MaxDB, so as far as the 100k events problem happening
with this, I can't confirm.  We currently have many many millions of
records in our archive.  Though performance isn't much of a problem on
our DB with this many, we chose to move off old data into an archive
(actually copies off realtime, so the archive is our all-time db)... And
keep the  recent stuff in a 'real-time' database that users can access
and perform complex queries through a somewhat useful web interface.

All of the performance problems we've ever run into have related
directly to the joining of many tables [>4].  Our resolution to this
problem is a table design that requires less joins, and overall doesn't
use more space.  We're currently working on (and could use help with) a
new output plugin that works with this new DB design.

Meeting time...

-----Original Message-----
From: Hutchinson, Andrew [mailto:andrew.hutchinson () Vanderbilt Edu] 
Sent: Monday, March 08, 2004 4:21 PM
To: Jason Haar
Cc: snort-users () lists sourceforge net
Subject: RE: [Snort-users] Question about best hardware

snortdb=# select count(*) from event;
 count  
--------
 514109
(1 row)

In my "catch-all" database (I keep a catch-all db for forensic reasons,
so I can go back and look at every little alert if necessary, and a
"day-to-day" db, for common exploits), I currently have about half a
million records, and generally archive when I reach around 4 million
alerts.  Performance is not terrible (it takes 30 seconds or so to find
events of interest when the db approaches 4 millions recs), but I'm
using Postgresql and not using ACID (I wanted some different
capabilities, multiple user access levels, etc. so I made muy own
interface).  MySQL is generally faster than Postgresql though, and
should be able to handle WAY more than 100K records w/o any problem.

As I mentioned, I am currently using Postgresql.  However, MySQL4 and
MaxDB have sparked anew an interest in MySQL, so I've been playing
around with it a bit lately.  Perhaps a good way to run down the problem
would be to turn on slow-query logging (with the --log-slow-queries
option at MySQL startup), and then run those same queries interactively
against the db with the EXPLAIN keyword to see what's causing them to be
slow.  I'd do this, but again I don't currently use MySQL or ACID...

HTH,

Andrew

> On Sat, 2004-03-06 at 07:07, Kreimendahl, Chad J wrote:
> > Hardware won't be your problem.  Once you get around 100k events in 
> > the snortdb on MySQL you'll run into major performance problems that 
> > almost no amount of hardware seems to solve.
>
> You're dead right there. 100K does appear to be the limit for me too.
>
> Is this a MySQL-specific issue? How does Postgresql or Oracle handle 
> DBs over 100K?
>
> Has anyone tried to figure out the problem? There are apparently 
> people using MySQL with terrabytes of data (nothing to do with snort),

> so why is 100K of snort records such a big deal?
>
> Cheers
>
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +64 3 9635 377 Fax: +64 3 9635 417
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of 
> GenToo technologies. Learn everything from fundamentals to system 
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> Snort-users mailing list
> Snort-users () lists sourceforge net
> Go to this URL to change user options or unsubscribe: 
> https://lists.sourceforge.net/lists/listinfo/snort-users
> Snort-users list archive: 
> http://www.geocrawler.com/redir-sf.php3?list=snort-users


Thanks,
Josh Berry, CISSP
CTO, VP of Product Development
LinkNet-Solutions
469-831-8543
josh.berry () linknet-solutions com



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=ick
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=ort-users



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Snort-users mailing list
Snort-users () lists sourceforge net
Go to this URL to change user options or unsubscribe:
https://lists.sourceforge.net/lists/listinfo/snort-users
Snort-users list archive:
http://www.geocrawler.com/redir-sf.php3?list=snort-users