Re: Pentesting tool - Commercial

[Ivan Arce <ivan.arce () coresecurity com>]

Hello Andre & pen-testers

Since Core was been mentioned as having lied on this list I felt compelled 
to reply and ask you to better qualify that statement so I can verify it 
and address it properly if need be.

I'd also like add my comments regarding your assessment of CORE IMPACT 
capabilities with the *obvious caveat* that I am Core's CTO. But before 
that, I'd like to ask you to clarify how did you come to your conclusions 
and if you were or are a licensed user of a current and up-to-date version 
of CORE IMPACT because I suspect you may be providing opinions that are 
based on a partial or limited view of our product. Please feel free to 
contact me directly or through any of Core's Customer Support channels so 
we can follow up on any particular feedback (or complain) you may want to 
provide.

We really appreciate constructive criticism and value the help of the 
security community to improve our product, then again, many may choose not 
to help us and that is totally fair too.

Going back to the original comments about CORE IMPACT and the 'count of 
exploits' I'd like point out just that throwing numbers without qualifying 
the measurement criteria and the relevance of the methodology is not a 
very serious assessment of a product's capabilities, its suitability for a 
given use or the value it may provide to a security professional.

Nonetheless, the last time I checked (a few minutes ago as of today) a 
fully-functional legitimately activated and up-to-date CORE IMPACT 7.5 
distribution has 665 exploits targeting 4414 unique target combinations 
(OS, SP, service, application version) distributed across 466 modules and 
that's not counting the components that detect and exploit SQL injection 
and Remote File Inclusion vulnerabilities in custom/proprietary web 
applications.

Note that, all of our exploits are fully-tested to meet quality standards 
before they ship to all our active customers and thereafter continue to be 
supported through the entire life cycle of the product (all of which is 
included in the product's purchase and licensing terms). We do not ship 
0day exploits (that is, exploits that target vulnerabilities for which 
information is not publicly available without access or confidentiality 
restrictions), we do not ship different sets of exploits for subsets of 
users in our customer base and we do not ship or support exploits (or 
other modules) that are not developed by our own team, so it is not so 
easy to a provide fair comparison just with "count numbers".

Exploit quality assurance seems to be an often overlooked topic, perhaps 
because we are the only organization (that I know of) that does full 
system tests running all the supported exploits against all the supported 
platform combinations with all the supported connection methods *daily* in 
order to have constant visibility of the quality of the "exploits" 
component of our product (there are many "non-exploit" modules that 
provide valuable penetration testing capabilities as well). This requires 
a level of investment in processes and infrastructure that is not 
insignificant and perhaps only available to a commercial organization that 
is fully committed to ensure a certain degree of product quality.

Lastly, as I said above, we maintain all the modules shipped with the 
product and provide regular product updates on a weekly basis. The number 
of module updates (which include new exploit modules as well as 
improvement and bug fixes to existing ones and other types of modules) has 
been at an average of around 22.5 per month in the past 6 months.

But of course... all of this information is provided directly to all our 
users in the product's dashboard (the first screen that you see when you 
run CORE IMPACT) so I am merely transcribing it here without much analysis 
so far.

In any case, I don't know what is the "right" criteria to use for counting 
of exploits or what's the criteria to determine their relevance, although 
I've talked about this and presented several ideas at many public venues 
in the past years and I'd welcome further discussion on that topic.

However, I do think that throwing numbers without a consistent and sound 
rationale for a comparison is usually miss-leading. In fact I believe this 
because that had been the case during the early 1990's when "number of 
vulnerability checks" was the common currency for comparisons of 
vulnerability scanning products. At the time I was part of the development 
team of one of the three emerging leaders in the vulnerability scanning 
software market and I clearly remember that every vendor and all otherwise 
biased parties had their own (self-serving) methodology and product 
marketing strategies to count "vuln. checks". In the end those comparisons 
did not prove very useful to anybody and today there aren't many 
organizations that use number of vulnerability checks as substantially 
relevant information to determine if a given scanner will help them 
fulfill their requirements and provide them the value expected from it.

So that moves the evaluation criteria discussion to a discussion about 
whether a given product may or may not provide its customer with enough 
value to justify its purchase and subsequent use and whether it 
appropriately fits what its expected from it.

I can't speak for other tools or even for CORE IMPACT because I am not a 
paying customer :) but I can certainly say that at Core we view our 
product as much more than a bunch of features and a bag of exploits and we 
make a very serious effort to understand our customer's needs and to 
combine all our capabilities to build software that gives back to our 
users the money they paid (and hopefully more!) in the form of valuable 
and usable tools to improve their security or the security of their customers.

We may or may not succeed in that endeavor but we firmly believe -across 
our entire organization- that to be the promise that commercial 
penetration testing software vendors must deliver on.

-ivan


Andre Gironda wrote:
> On Wed, Feb 27, 2008 at 1:38 PM, Trygve Aasheim <trygve () pogostick net> wrote:
> >  This doesn't mean I don't like or use Metasploit, Canvas or any
> >  other...I just want to point out that the quality of a product is not
> >  based a number, and Core Impact has proven its quality many times, and
> >  in many ways.
>
> The numbers show that Core Impact is superior to Canvas and Metasploit.
>
> Unfortunately, it also shows that Impact is missing quite a lot.  The
> point I was trying to make is that you can't use only one exploitation
> engine.
>
> However, I also fail to see the point of using an exploitation engine
> except in the case of testing IPS/IDS or similar.  In this case,
> anyone would clearly be better off using BreakingPoint Systems
> BPS-1000.
>
> Using exploits on production or IT networks is unethical.  This isn't
> the wild west.  You're overpaying by about $19K-$26K for what you need
> when you go with Core Impact.  I don't know about ya'll, but the idea
> of propagating a pseudo-worm through a corporate network seems about
> as good of an idea as asking the power company to shut off electricity
> to a hospital for "just a minute, to see what will happen".
>
> Instead of RPT, I suggest asset management combined with regular,
> good-old fashioned vulnerability scanning.  Most of the "experts" I
> know don't even understand the difference between a vulnerability and
> an exploit.  More of those people don't even understand how unreliable
> exploits usually are (let alone scanning errors in vulnerability-only
> scanners).
>
> Core already lied once on this list about how many modules vs.
> exploits vs. CVE's they support.  They could make anything up.  The
> money numbers do not lie.  Compare to Rapid7, Tenable, Lumension, or
> McAfee for yourself.
>
> If you have to raise awareness by running live exploits, try
> Metasploit.  It's free.  Management still not convinced?  Already
> covered all the Metasploit exploits?  Try Canvas, it's cheap.
> Management still not convinced?  Already covered the Canvas exploits,
> too?  Add an exploitation pack or two.  Start writing your own
> exploits.  Management still not convinced?  Already covered all of the
> Canvas exploitation packs and started writing your own in-house
> exploits specific to your architecture?  Maybe Core Impact will help;
> call them for a demo.
>
> I have no idea why people are so quick to jump to Core Impact first.
> You can't just throw money at these types of problems.  Security is a
> very careful and gradual process.
>
> Cheers,
> Andre
>
> ------------------------------------------------------------------------
> This list is sponsored by: Cenzic
>
> Need to secure your web apps NOW?
> Cenzic finds more, "real" vulnerabilities fast.
> Click to try it, buy it or download a solution FREE today!
>
> http://www.cenzic.com/downloads
> ------------------------------------------------------------------------

--
"Buy the ticket, take the ride" -HST

Ivan Arce
CTO

CORE SECURITY TECHNOLOGIES
http://www.coresecurity.com

PGP Fingerprint: C7A8 ED85 8D7B 9ADC 6836  B25D 207B E78E 2AD1 F65A


------------------------------------------------------------------------
This list is sponsored by: Cenzic

Need to secure your web apps NOW?
Cenzic finds more, "real" vulnerabilities fast.
Click to try it, buy it or download a solution FREE today!

http://www.cenzic.com/downloads
------------------------------------------------------------------------