Snort mailing list archives
Announce: StreamDB
From: Martin Holste <mcholste () gmail com>
Date: Tue, 1 Feb 2011 14:26:14 -0600
I wanted to let these communities know about a new open-source project called StreamDB (http://code.google.com/p/streamdb/) I've just published that's proven to be extremely helpful for my analysts. It is a fast and simple tool for quickly viewing traffic related to IDS alerts (or any IP-based event) which specializes in ultra-fast retrievals from very large data sets. It can hook into Snorby as it is OpenFPC compatible. It is also very effective for PCRE searching traffic from a given source or destination IP address. Streams are rotated out based on configured retention size in a ring-buffer fashion. From the project home page: StreamDB is a high-performance framework for storing network streams. The current version uses Vortex IDS to read the streams from a file or network interface and saves them to an indexed DB and data file. Web code provides an URL-based query interface. There is also a command-line interface which includes the ability to read piped queries from STDIN. In addition to almost instant retrieval by IP address, StreamDB also allows PCRE searches and file type searches on streams if an IP address is provided as an initial filter. The system can handle recording gigabit line-speed networks and can retrieve arbitrary streams from terabytes of data in milliseconds. It is designed to be a complimentary tool to intrusion detection systems to aid security analysts. Here are some query examples: http://streamdb/?srcip=10.0.0.1 http://streamdb/?srcip=10.0.0.1&dstip=1.1.1.1&dstport!80 http://streamdb/?srcip=10.0.0.1&dstip=1.1.1.1&dstport=80&start=2 weeks ago&end=now http://streamdb/?srcip=10.0.0.1&pcre=example.com http://streamdb/?srcip=10.0.0.1&pcre=MZ.*PE\x00\x00 http://streamdb/?srcip=10.0.0.1&sort=1&as_hex=1 http://streamdb/?srcip=10.0.0.1&raw=1 http://streamdb/?srcip=10.0.0.1&offset=1000&limit=200 http://streamdb/?srcip=10.0.0.1&filetype=executable Examples from the CLI: ./sdb --srcip 10.0.0.1 --filetype pdf --headers-only tail -f /var/log/snort/alert | ./sdb > alert_streams.txt All of these queries will return in a second or two, assuming that the IP's referred to aren't busy web servers or NAT points. IP's with many connections will benefit from more specific search filters for time and/or destination IP address. Some non-scientific benchmarks on commodity hardware searching 5 TB of data: PCRE search for a given srcip with 1000 connections completes in about one second. A similar search for a srcip with 50,000 connections will take about two minutes. A lookup for a given srcip/dstip pair will complete in less than a second, including browser render time. Example output as text/plain: Returning 2 of 2 at offset 0 from Sun Jan 30 11:56:11 2011 to Sun Jan 30 11:56:11 2011 2011-01-30 11:56:11 192.168.58.52:4099 -> 131.243.2.191:80 13s 512 bytes FIN ASCII text, with CRLF line terminators GET /bro-workshop-2009-2/slides/Installation.pdf HTTP/1.1 Host: www.bro-ids.org User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Referer: http://www.bro-ids.org/bro-workshop-2009-2/slides/ X-Do-Not-Track: 1 Connection: keep-alive 2011-01-30 11:56:11 192.168.58.52:4099 <- 131.243.2.191:80 13s 778247 bytes FIN PDF document, version 1.3 200 OK Connection: Keep-Alive Date: Sun, 30 Jan 2011 17:56:11 GMT Accept-Ranges: bytes ETag: "8f724e-bde6b-47617252eebc0" Server: Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 OpenSSL/0.9.8e DAV/2 PHP/5.3.5 with Suhosin-Patch mod_python/3.3.1 Python/2.6.6 mod_wsgi/2.8 Content-Length: 777835 Content-Type: application/pdf Last-Modified: Sat, 17 Oct 2009 01:08:07 GMT Keep-Alive: timeout=5, max=100 X-HTTP-Version: 1.1 %PDF-1.3 %........... 4 0 obj << /Length 5 0 R /Filter /FlateDecode >> stream x..VM..7..W.T..z,i4....I.-.bS..!.a...nl..=. ...%O.Q.....|.Y.(..|.......). 9........l.h.'E....-....&.7]....... .}Fvr....}.x... .)...^k."U.rC.....w.N<...Z..u<..Z..e.j....4T.Hpj.........u...../g.....n.....o.......R....*.Do.9<.*]>...(...I8....ikJ_.T...:......c|..Ki..Q..>.U.MZ...*...!........jKik~7.qg.iw#.......|..............D/.\Yx..v...2<..d. O./...<...&.DDF....x..K. Qy...|..f F>.2k.v....^v..{^<..Q..v....t}.f;x.e.S..]...U.7......l..uD... <clipped for brevity> I hope that others have found it as useful as we have for rapidly investigating IDS alerts. Please use the project page or email me to let me know about any questions, issues, or suggestions. Thanks, Martin ------------------------------------------------------------------------------ Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d _______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://www.geocrawler.com/redir-sf.php3?list=snort-users
Current thread:
- Announce: StreamDB Martin Holste (Feb 01)
- Re: Announce: StreamDB Joel Esler (Feb 02)