IDS mailing list archives
RE: Views and Correlation in Intrusion Detection
From: David Markle <davidmarkle () comcast net>
Date: Thu, 26 Jun 2003 12:42:31 -0400
Well, since we are speaking in idealistic terms, retrofitting a system of this magnitude in a truly global enterprise would be very difficult and the scaling issues would definitely come into play. My thoughts on this are that the global model must have a hierarchical tiered approach where the event aggregation (central collection PLUS event linking/counting/suppression) and correlation takes place further down the tree prior to it boiling up to the top level monitor. This is the only way to distribute processing and deal with scalability. So you're absolutely correct, no one system will handle the load (i.e. correlation for the mainframe). I agree with your vendor standardization comments. They are generally NOT willing to spend the development $$ on something that does not produce revenue first (no offense vendors, but its a revenue based world ;) ). Therefore, as we are seeing with the several products out there (Arc-site, etc....), log agent listeners are developed just for this "vendor" specific purpose (aggregation and normalization). There are a whole lot of smart people out there and the problems can be resolved. The scalability issue can be resolved via the hierarchical tiered approach, add levels of duplicate alert suppression, bandwidth throttling, and queuing and the issue is pretty much resolved. (remember ...we're being idealistic here ...) I think the greatest growth potential in the near future is the intelligence behind the correlation. Aggregating and normalizing the logs is not a huge problem. Companies with $$ with be able to hire the mathematicians, and statisticians to create the needed logic behind the concept. Hopefully, our Open Source community has a few of those types out there to help push the freeware movement. as always, my $.02 David Markle -----Original Message----- From: Sekurity Wizard [mailto:s.wizard () boundariez com] Sent: Wednesday, June 25, 2003 11:03 PM To: DAVID MARKLE; Blake Matheny Cc: focus-ids () securityfocus com Subject: RE: Views and Correlation in Intrusion Detection David, Your are all absolutely correct - correlation is the gold medal...right now everyone in the industry is praying for bronze at best. The one glimmer of hope I see are products out there, and I don't remember the company name right now, that aggregate hundreds of gigabits of logs per hour and try to make sense of it all. The question them becomes one of scalability...assuming we take for granted someone CAN write an engine that processes this sort of data in a sane manner. Scalability, in the form of the type of environment I work at is insanely large. We have umpteen numbers of DS3's, countless T1's and thousands of pipes to and from segments we aren't even *aware of*...not to count the couple of hundred (close to 1,000) firewalls that are out there. Now, let's say we put a couple of these boxes (~50Mbit/sec each) to the test in my environment. There STILL NEEDS TO BE A CENTRAL PROCESSOR...otherwise, we're left with the distributed view - which we don't want, right? Is it realistic to think there is such a scalable system that can process hundreds of gigabits of data per second, aggregate it all, normalize it, and correlate too? I dare say not at this point...unless we come up with some sort of standard, "XML for security devices" that makes the processing and data crunching easier....but the problem there is I don't see Checkpoing, Cisco, Enterasys, and ISS (and others) getting together on this any time soon.... So scalability is our main opponent as I see it...because at the end of the day - the only attack that counts is the 1 in 100,000,000 that sent that single UDP packet that triggered a shutdown of the entire network due to SQL Server port floods...right? Sleep well... :) -----Original Message----- From: DAVID MARKLE [mailto:davidmarkle () comcast net] Sent: Tuesday, June 17, 2003 1:49 PM To: Blake Matheny Cc: focus-ids () securityfocus com; davidmarkle () comcast net Subject: Re: Views and Correlation in Intrusion Detection Blake, I agree with your sentiments regarding correlation and have more to add. The point of correlation is the value it adds to mostly autonomous, unreviewed, and meaningless data. (The folks that disagree with this line must have economically independent budgets with staffing consisting of superstar (I applaud you)). Who reviews the firewall logs? I don't. We have over 500 global firewalls. The point here is (as you stated) AUTOMATION. But it does not stop there. That data has to be normalized and applied towards something. The correlation piece adds that middleware "something". An IDS alert is ONLY relevant if the firewall permits the traffic through. To further the comment, and attack signature tripped for (known attack) xyz, is ONLY relevant when the attacked host is vulnerable to xyz. This is the ultimate job of correlation. If the above surrounding conditions are true, the severity of the attack becomes increased to critical, otherwise it is informational only. There are also netops statistics that should be considered security related (and monitored). Baseline your bandwidth, averaged over 12 months. Normal increases in business offerings are roughly 5 percent per month. Since there was no change control this past weekend (to relate), why did you see a spike in bandwidth by 17 percent ???? Why is tcp 2148 increasing on your global perimeter over the past 3 days? These are relevant questions. Without the collection and aggregation of the appropriate data, we run the operations in the dark. With regards to the state of correlation, I still think its an infancy issue. Historically, I believe that the industry (tech folks) has been extremely focused on growth development and deployment of the technology (firewalls, IDS-(H/N), etc.). Firewalls have been around for awhile and have matured to a point of plateau (mostly). IDS is now in "the growth phase" (with heuristic, anomaly, signature, blah, blah, blah), and all that hype. I really think that the industry had recently realizes that we are now overwhelmed with too much data. Now everyone is scrambling to catch up ..... David Markle davidmarkle () comcast net davidmarkle () elephantfoot org ----- Original Message ----- From: Blake Matheny <bmatheny () mkfifo net> Date: Tuesday, June 17, 2003 1:32 pm Subject: Views and Correlation in Intrusion Detection
Two areas that I have recently been doing research in, are views and their connection to correlation techniques. In terms of systems, given some event, the information we get about the occurrence of such an event comes to us in the form of either a primary or a secondary view. Information about secondary views typically come to us from applications such as firewalls and ID systems. Primary information usually is received from the application actually processing this data for use. For instance, an ID sensor may produce an alert about some traffic. However, this is a secondary view of the event and needs to be correlated with other, relevant information. So of course firewall logs might be checked, to see if traffic actually passed that corresponds to the event in question. This is also a secondary view, so a third place is checked, the applications logs. There are really several issues here. First of all, a tremendous amount of time is being spent, trying to correlate all the relevant information. This is something that _can_ be automated. Second, the applications logs may not be trustworthy. Third, and to me, most importantly, is the fact that this is such a 'basic' thing that people using ID systems have to do, and there is no piece of software yet that does this. So something we have been working on, is a system to deal with this basic type of scenario. This will entail data transformations into an intermediarylanguage, an event description language, offline state analysis and several other components (there is more information at http://www.nongnu.org/babe/).If you spend some time thinking about everything involved to do this in a scalable fashion, it's an enormous task (I said basic, not simple). What I am finding frustrating, is that much of the base research has not yet even been done. Much of the research that has been done, is either too primitive or too impractical to be implemented. Is this due to the infancy and immaturity of the field, do people not see this as being feasible and therefor aren'tspending the research time, or is this simply too far down the line? In any case, feedback welcome. Thanks. Cheers, -Blake -- Blake Matheny "... one of the main causes of the fall of the bmatheny () mkfifo net Roman Empire was that, lacking zero, they had http://www.mkfifo.net no way to indicate successful termination of http://ovmj.org/GNUnet/ their C programs." --Robert Firth ------------------------------------------------------------------- ------------ Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas, the world's premier technical IT security event! 10 tracks, 15 training sessions, 1,800 delegates from 30 nations including all of the top experts, from CSO's to "underground" security specialists. See for yourself what the buzz is about! Early-bird registration ends July 3. This event will sell out. www.blackhat.com--------------------------------------------------- ----------------------------
------------------------------------------------------------------------ ------- Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas, the world's premier technical IT security event! 10 tracks, 15 training sessions, 1,800 delegates from 30 nations including all of the top experts, from CSO's to "underground" security specialists. See for yourself what the buzz is about! Early-bird registration ends July 3. This event will sell out. www.blackhat.com ------------------------------------------------------------------------ ------- ------------------------------------------------------------------------------- Attend the Black Hat Briefings & Training, July 28 - 31 in Las Vegas, the world's premier technical IT security event! 10 tracks, 15 training sessions, 1,800 delegates from 30 nations including all of the top experts, from CSO's to "underground" security specialists. See for yourself what the buzz is about! Early-bird registration ends July 3. This event will sell out. www.blackhat.com -------------------------------------------------------------------------------
Current thread:
- Re: Views and Correlation in Intrusion Detection, (continued)
- Re: Views and Correlation in Intrusion Detection Randy Taylor (Jun 23)
- RE: Views and Correlation in Intrusion Detection adam.w.hogan (Jun 25)
- RE: Views and Correlation in Intrusion Detection Schmehl, Paul L (Jun 25)
- RE: Views and Correlation in Intrusion Detection Scott M. Algatt (Jun 25)
- RE: Views and Correlation in Intrusion Detection Chmielarski TOM-ATC090 (Jun 25)
- Re: Views and Correlation in Intrusion Detection Mike Coliton (Jun 26)
- RE: Views and Correlation in Intrusion Detection Sakaba (Jun 26)
- RE: Views and Correlation in Intrusion Detection Kohlenberg, Toby (Jun 26)
- RE: Views and Correlation in Intrusion Detection Chmielarski TOM-ATC090 (Jun 26)
- RE: Views and Correlation in Intrusion Detection Sekurity Wizard (Jun 26)
- RE: Views and Correlation in Intrusion Detection David Markle (Jun 27)
- RE: Views and Correlation in Intrusion Detection Ron Gula (Jun 26)
- RE: Views and Correlation in Intrusion Detection Paul Schmehl (Jun 27)
- RE: Views and Correlation in Intrusion Detection Richard Ginski (Jun 27)