IDS mailing list archives
IPS test criteria (was IDS\IPS that can handle one Gig)
From: Bob Walder <bwalder () spamcop net>
Date: Tue, 07 Jun 2005 08:04:28 +0200
Guys, I don't often weigh in to toot my own horn but feel I have to here. Perhaps when we open our new US security testing lab next year we will be "on the radar" a little more? ;o)) Chris - what makes ICSA particularly relevant when it comes to defining IPS test criteria? Speak to the vendors who were at their recent forum meeting where their new NIPS criteria was being discussed to find out how far adrift they are. As good as it is in other areas, to my knowledge ICSA has very little - if any - experience in house of testing IDS/IPS devices and seems to be struggling to create a useful test methodology right now. One thing ICSA certainly will NOT be focussing on as far as I can see is performance - they tend to focus mainly on management minutiae NSS, on the other hand, covers both. Whilst we do not define individual "tests" for the management minutiae as ICSA seems to do, the 30+ page evaluation which accompanies each NSS Approved certification goes into plenty of detail on how easy the device is to manage, how scalable it is, pros and cons, etc, etc. In addition, we stress performance of these devices in a number of ways. It has been suggested by one vendor that pps is the be-all-and-end-all of performance. That is not the case. It IS, however, one facet of performance (raw packet processing power) and we DO, Chris, cover this in our tests with a wide range of traffic loads and packet sizes. Detection engine performance is often independent of raw packet processing performance, however. When you give the appliance something to actually DETECT, that wonderful microsecond latency 1Gbps with 64-byte packets throughput that the hardware guys boast of can suddenly plummet to much less than 1Gbps. I agree with Paul from ISS in that it does not always require ASICS and FPGAs to get 1Gbps of throughput from an in-line device (though the hardware acceleration provides other advantages that have been discussed at length already - so let's not go there again) So, in addition, we also cover a wide range of traffic mixes which stress the detection engine too. These take care of protocol mix, average packet size, HTTP connections per second and transaction per second, and so on and so forth. Finally, we introduce REAL Web/FTP servers into the mix, using copies of real Web sites and real user browsing sessions to create as close a "real world" conditions in our labs to show how devices are likely to perform in real deployments, when such extremes of connections per second and pps are less likely to occur. Creating these tests to provide this wide range of test condition and - more importantly - produce stable and precisely repeatable conditions for every single test is not easy (i.e. If we say we are testing 20,000 conns per sec with a 5k response size and 350 byte average packet size at 1Gbps then that is exactly what is produced throughout the test). Ask anyone who has used the Spirent Avalanche test gear how easy it is to do (and with most of the other test gear out there you cannot even get close to specifying such a wide range of test criteria). But we ARE already doing it, so why not take advantage of it? Having examined our test criteria (www.nss.co.uk/ips) I would be happy to take suggestions from interested parties on how it may be improved, what tests are overkill, what test we are missing, and so on. It will be a while before you can do that with the ICSA NIPS test criteria, I fear ;o) Ed - That is a cool idea - I would definitely be interested in participating the type of forum you mention. Check out our existing test criteria at www.nss.co.uk/ips and see if you think we have a good basis for initial discussions Bob Walder The NSS Group Www.nss.co.uk On 6/6/05 5:20 pm, "Ed Gibbs" <ed () digitalconclave com> wrote:
You're absolutely right - there needs to be IPS test standards. I would like to propose putting together a forum, and defining what the IPS test standards should be - is anyone interested? I would like to see several members from each IPS vendor involved. The result is that we create a set of procedures that provide guidance, and help someone determine which IPS is best for their environment. Ed ----- Original Message ----- From: "Chris Harrington" <charrington () nitrosecurity com> To: <THolman () toplayer com>; <PPalmer () iss net>; <ed () digitalconclave com>; <prashant () juniper net>; <focus-ids () securityfocus com> Sent: Saturday, June 04, 2005 11:43 PM Subject: RE: IDS\IPS that can handle one GigLet's have another vendor weigh in :) See my comments in line.-----Original Message----- From: THolman () toplayer com [mailto:THolman () toplayer com] Sent: Friday, June 03, 2005 8:25 AM 1) Gigabit performance is irrelevant; it's the packets per second that count. Vendors cheat and claim 1Gb performance based on large packet sizes (not real world), or just add up the sizes of all their interfaces.It would be nice if there was a standardized IPS performance test with regards to packet size, traffic mix, etc. I don't see that happening unless ICSA does it for the NIPS certification. This would cut down on the shady performance numbers that Tim refers to.2) In PC architecture, the PCI bus is the bottleneck, not the processor.That depends on what you are doing with the processor. If you are doing pattern matching in the CPU you could run out of CPU well before you run out of bus capacity. A PCI bus has a theoretical limit of 1.05 Gbps. A 16 lane PCI-Express bus is 80 Gbps. Several vendors are already shipping 10 Gig PCI-Express cards.3) An Intel processor has a large instruction set designed for workstation/server performance and GUI operations, and not for packet processing.I would say that the processor designers didn't have any specific tasks in mind. It is a general purpose processor.4) An ASIC has a tiny instruction set in comparison, designed for a specific task. So, a 3.2Ghz Intel processor forwarding/processing network traffic is on a par with a 133Mhz ASIC designed to do the same thing.I'm not an ASIC guy so I will take your word for it on the comparison :)5) Processors can only do one thing at once. Thus, a networking device with several processors installed in parallel (ASICs OR Intel) is far more effective than a box with a single/dual processor.More processors gives you more flexibility in what gets processed where.6) Hard disks do not slow down performance. They lower reliability as fail all the time (!). RAID would help, but I don't think any security vendor offers a RAID array as an integral part of their appliance, so cut to the chase, get the HDD off the inline unit and place on a separate management machine so we have a reliable distributed architecture that isn't put at risk by HDD failure. On the same note, dual fans and power supplies also need to be considered.Hard drives do fail, no question there. I definitely disagree with your statement about vendors not having RAID. There are definitely vendors (other than us) who have drives in RAID configuration, both 1 and 5. I am not sure taking the drive off the device makes for a more reliable distributed architecture. What if the link from the IPS to the Management machine goes down or the Syslog server dies? What if the hard drive in the Management machine fails? :) With no drive on the IPS your space to store events, system data, etc, is somewhat limited. How long before you have to start overwriting event data on the IPS? Same goes for dual fans and power supplies. There are vendors (again other than us) who have dual fans and hot swappable power supplies. Although these are generally found in the 500 mbps and up ranges. Don't forget fail open NIC's and bypass devices. Most vendors (including ASIC IPS') have them, at least as an option. If not having a hard drive is the path to reliability then why do vendors without hard drives have fail open NIC's? Because other components can and do fail as well.7) Single-processor machines can easily FORWARD 64-byte packets at 'multi-Gig' speeds. They can do this as the processor doesn't have to do anything with them. As soon as you add intensive operations to the packets in question, bearing in mind there is only a single CPU that can only do one thing at once, you introduce LATENCY plus reduce pps performance DRASTICALLY. This is where a parallel processing architecture comes into it's own and takes leaps forward over what a single-CPU box can do.You are assuming that the CPU is doing the packet processing. Many vendors are using network content accelerators and other processing cards to offload the CPU intensive operations.In conclusion: A box with one or two ASICs in is easily outperformed by a PC with the latest Intel processor, fast network cards and a good chunk of memory. However, the PC is more prone to hard disk failure, which is why you should never put one inline if uptime is critical. A box with several ASICs in will outperform ANY PC-based solution, and ANY ASIC solution that relies only on one or two processors.But at what cost in terms of price per Gigabit and flexibility? Adding new functionality to software is pretty easy......and one comment to Ed with respect to McAfee/TippingPointboth products really don't care if you have every signature and then some on.Yes they do. If you turn on every signature check with these IPS's, pps performance slows to a mediocre dribble...They do care. Look at some of the product reviews and you will see that vendor X has 2000 rules / filters / signatures but only 500 are on by default. I've witnessed a couple of ASIC IPS' that were brought to their knees when asked to store the offending packets. What about storing the TCP stream involved with an event? Customers are asking about this...Inline devices should NOT rely on REGEX signatures - by nature, string searching is very resource intensive and best left to a nice fast offline IDS running on an up-to-date PC platform, where latency is not going to be an issue...There are PC platform IPS on the market that are under 100 microseconds that do pattern matching.Hope this helps - this isn't an all out war ASIC-based vs PC-based, it's a question of architecture and suitability for the job in hand!Definitely an interesting thread. I agree that it is about suitability. --Chris Christopher Harrington, CISSP Chief Technology Officer nitrosecurity o: 603.570.3931 c: 603.969.0592 e: charrington () nitrosecurity com w: www.nitrosecurity.com Skype: chrisharrington-------------------------------------------------------------------------- Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 to learn more. --------------------------------------------------------------------------
-------------------------------------------------------------------------- Test Your IDS Is your IDS deployed correctly? Find out quickly and easily by testing it with real-world attacks from CORE IMPACT. Go to http://www.securityfocus.com/sponsor/CoreSecurity_focus-ids_040708 to learn more. --------------------------------------------------------------------------
Current thread:
- RE: IDS\IPS that can handle one Gig, (continued)
- RE: IDS\IPS that can handle one Gig Dave Hawkins (Jun 01)
- RE: IDS\IPS that can handle one Gig Palmer, Paul (ISSAtlanta) (Jun 04)
- RE: IDS\IPS that can handle one Gig THolman (Jun 04)
- Re: IDS\IPS that can handle one Gig Ed Gibbs (Jun 04)
- RE: IDS\IPS that can handle one Gig Chris Harrington (Jun 06)
- Re: IDS\IPS that can handle one Gig Nick Black (Jun 07)
- RE: IDS\IPS that can handle one Gig THolman (Jun 04)
- Re: IDS\IPS that can handle one Gig Mike Frantzen (Jun 06)
- Re: IDS\IPS that can handle one Gig Nick Black (Jun 07)
- Re: IDS\IPS that can handle one Gig Mike Frantzen (Jun 06)
- Re: IDS\IPS that can handle one Gig Ed Gibbs (Jun 06)
- IPS test criteria (was IDS\IPS that can handle one Gig) Bob Walder (Jun 07)
- RE: IDS\IPS that can handle one Gig Gary Halleen (Jun 06)
- RE: IDS\IPS that can handle one Gig Hovis, Chris (Jun 07)
- RE: IDS\IPS that can handle one Gig THolman (Jun 07)
- RE: IDS\IPS that can handle one Gig Edward Sohn (Jun 08)
- RE: IDS\IPS that can handle one Gig Barrett G . Lyon (Jun 08)
- RE: IDS\IPS that can handle one Gig Palmer, Paul (ISSAtlanta) (Jun 08)
- RE: IDS\IPS that can handle one Gig Andrew Plato (Jun 10)
- Re: RE: IDS\IPS that can handle one Gig ian . bamford (Jun 10)