Penetration Testing mailing list archives
RE: Why eEye Retina (was MBSA scanner)
From: "Robert E. Lee" <robert () dyadsecurity com>
Date: Thu, 22 Apr 2004 11:48:22 -0700
Mike, I agree with you. The skills are indeed separate, but it helps to have tools that know about the latest public problems. That being said I'm finding myself cursing at every VA tool out there. There are four things wrong with all tools that I have played with: 1) False Positives - No <insert silly tool here>, this machine is not vulnerable to the "bitch slap" DoS :). This part takes a long time to ferret out. You don't want to tell people they're broken when they're not. 2) False Negatives - Even easy things are often missed by all tools (which is why most people end up using 2-3 of them for complete results). You can only test what you know about... and hope that you wrote your test in a sane universal way. 3) Too much focus on "network" problems. Not enough "outside of the box" thinking here. Granted a lot of this work may be through manual steps the tester makes, but there should be a good way of tracking every significant activity/discovery the team makes. 4) Broken AI - It's better to have a security analyst make conclusions than a piece of software that can't possibly see the big picture. In short, using these tools for a test of 1-50 systems is useful. Once you cross the 50-100-1000+++ system mark it really becomes information deluge. These tools are being sold to non-security professionals in the hopes of helping them improve InfoSec posture. This is unfortunate because it's missing the mark badly. My team is writing/using a data correlation engine that allows for a team of testers to have the computer do the tedious work that humans make get bored with and are error prone at, while allowing for human interaction throughout. It is all based around the OSSTMM (www.osstmm.org). This will make a good team of testers a lot more efficient in their time and more complete in their analysis. This represents a goal/design shift for "VA" tools, but I think it is an important one that other companies might follow, especially as they realize that companies that can afford $17,000 for VA software are likely have an internal team of testers anyway :). Robert -----Original Message----- From: Mike Murray [mailto:mmurray () ncircle com] Sent: Wednesday, April 21, 2004 8:33 PM To: Shawn Edwards; clarke-cummings () columbus rr com Cc: pen-test () securityfocus com Subject: RE: Why eEye Retina (was MBSA scanner) Let me state up front: I work for a competitor in the VA market, so I'm going to stay far away from any discussion on products, and try to stick with a bit of philosophy. I had one comment on something that Shawn said:
I know for a fact that they have some very skilled persons doing dev there. ... Just check some of their development discoveries that's gotta count for something!
While this is definitely an argument for the fact that a company has very smart people working for it (which is definitely not in question in eEye's case), I question the validity of the argument as far as the evaluation of a network VA tool. If the ability to discover new vulnerabilities were the gold standard for a good VA tool, we'd all be buying something that Dave Aitel wrote. While it is often given as a reason that one tool is better than another, it simply doesn't follow that an aptitude for discovering new vulnerabilities in code is the same as an aptitude for discovering known vulnerabilities in running services in the real world. IMHO, the skills are related, but significantly different. In my mind the analogy is similar to that of the difference between medical research and surgery. People who practice one extremely well don't usually practice the other to the same level, even though the skills (though not necessarily the mindsets) required to perform both are somewhat similar in many cases. One just happens to be focused on discovering new techniques out in the world, and the other happens to be focused on saving lives. My $0.02. M ------------------------------------------------- Michael Murray Director of Vulnerability and Exposure Research nCircle Network Security Office: 416-533-5305 ------------------------------------------------- ------------------------------------------------------------------------ ------ Ethical Hacking at the InfoSec Institute. Mention this ad and get $545 off any course! All of our class sizes are guaranteed to be 10 students or less to facilitate one-on-one interaction with one of our expert instructors. Attend a course taught by an expert instructor with years of in-the-field pen testing experience in our state of the art hacking lab. Master the skills of an Ethical Hacker to better assess the security of your organization. Visit us at: http://www.infosecinstitute.com/courses/ethical_hacking_training.html ------------------------------------------------------------------------ ------- ------------------------------------------------------------------------------ Ethical Hacking at the InfoSec Institute. Mention this ad and get $545 off any course! All of our class sizes are guaranteed to be 10 students or less to facilitate one-on-one interaction with one of our expert instructors. Attend a course taught by an expert instructor with years of in-the-field pen testing experience in our state of the art hacking lab. Master the skills of an Ethical Hacker to better assess the security of your organization. Visit us at: http://www.infosecinstitute.com/courses/ethical_hacking_training.html -------------------------------------------------------------------------------
Current thread:
- RE: Why eEye Retina (was MBSA scanner), (continued)
- RE: Why eEye Retina (was MBSA scanner) Steve (Apr 21)
- RE: Why eEye Retina (was MBSA scanner) Bojan Zdrnja (Apr 22)
- RE: Why eEye Retina (was MBSA scanner) Steve (Apr 21)
- Re: Why eEye Retina (was MBSA scanner) Shawn Edwards (Apr 21)
- Re: Why eEye Retina (was MBSA scanner) Bobby . Clarke (Apr 22)
- RE: Why eEye Retina (was MBSA scanner) Cam Beasley, ISO (Apr 21)
- RE: Why eEye Retina (was MBSA scanner) Chris Hurley (Apr 21)
- RE: Why eEye Retina (was MBSA scanner) Lovrien, Scott (Apr 21)
- Re: Why eEye Retina (was MBSA scanner) Renaud Deraison (Apr 22)
- RE: Why eEye Retina (was MBSA scanner) Mike Murray (Apr 22)
- Re: Why eEye Retina (was MBSA scanner) Shawn Edwards (Apr 22)
- RE: Why eEye Retina (was MBSA scanner) Robert E. Lee (Apr 22)
- RE: Why eEye Retina (was MBSA scanner) Peter Benson (Apr 22)
- RE: Why eEye Retina (was MBSA scanner) Doty, Stephen (BearingPoint) (Apr 22)
- Re: Why eEye Retina (was MBSA scanner) Rainer Duffner (Apr 24)
- RE: Why eEye Retina (was MBSA scanner) Riley Hassell (Apr 22)
- Re: Why eEye Retina (was MBSA scanner) clarke-cummings () columbus rr com (Apr 23)
- RE: Why eEye Retina (was MBSA scanner) Steve Goldsby (ICS) (Apr 26)