Re: Vulnerability scanners don't work

["Adriel T. Desautels" <ad_lists () netragard com>]

Comments embedded below:


On Jan 8, 2009, at 3:49 PM, security curmudgeon wrote:

> : Good point, I don't think that I was as clear as I could have been.
> : The truth is that vulnerability scanners do contain signatures or
> : scripts that allow them to hunt for certain types of  
> vulnerabilities as
> : well as the specific known vulnerabilities.  But are you saying that
> : they can actually identify new vulnerabilities? I'm still saying  
> that
> : they can't.
>
> Absolutely, and they have. In the last 12 months, I handled a  
> disclosure
> between the pen-test shop I recently left and Real Networks for a
> vulnerability in one of their products. A consultant ran Nessus  
> against a
> client and ended up finding a traversal that allowed him to grab any  
> file
> on the remote system. Likewise with AppScan, I handled several  
> disclosures
> to vendors for a wide variety of SQLi and XSS in various products.

Ah right, that makes perfect sense. Web Application Vulnerability  
Scanners
do a very good job at identifying vulnerabilities (so long as they are  
not false
positives). This is because Web Application Scanners or the modules that
scan web applications actually attack/exploit the vulnerability. This  
is like the
difference between a penetration test and a vulnerability assessment.  
With a
penetration test there's no reason to have false positives.

That said, the last application scanner I ran (web apps) returned  
something
like 2300 vulnerabilities. Of those 2300 only 2 were legitimate and it  
missed
6 SQL Injection vulnerabilities. :(

> I'm not saying that either product found vulnerabilities the  
> consultant
> didn't or wouldn't have, but those tools were used on every network or
> application test to set a base line. In several cases, each found
> new vulnerabilities before the consultant began the manual testing.
>
> I'm a little confused here on why you are so insistant on  
> vulnerability
> scanners not being able to find a new vulnerability.

I was being too granular in my thinking and had my head stuck up my  
ass. :)

> : Lets take your www_too_long_auth.nasl script into consideration only
> : because it is the first one that I noticed.  That script just  
> connects
> : to a web service and blindly dumps a 2048 bit payload into the
> : authorization buffer. If the service stops responding then the  
> script
> : tells the scanner that the service is vulnerable, but is it? If the
> : service keeps responding then the script tells the scanner that the
> : service is not vulnerable, how accurate is that? Would you  
> consider that
> : to be positive vulnerability identification? Can we be certain  
> that the
> : scripts are finding real, exploitable conditions and not false
> : positives?
>
> How accurate? Pretty accurate that a long string may cause a DoS
> condition.
>
> Positive vulnerability identification? Absolutely not.
>
> Can we be certain..? Absolutely not.

We're on the same page here.

> I only stated that these products can find vulnerabilities, and don't
> necessarily require or only use signature based auditing. Just like a
> human doing a pen-test, the scanners have to find evidence of a
> vulnerability first. That may be in the form of a crash, error  
> message or
> something else that catches their eye. Yes, vulnerability scanners are
> primitive compared to a good pen-tester, i'm not arguing that or  
> trying to
> say that scanners can replace humans. I am saying that vulnerability
> scanners have their place in the market for many reasons, and that the
> important part is to understand how they work, what they can find  
> and the
> limitations inherant in their design.

Mostly agreed. Wouldn't you consider a script to be a dynamic signature?
It is after all checking for one thing and on thing only.  Maybe its  
just a
matter of opinion.

> : Sure they might be able to identify a problem that might be a
> : vulnerability via the ad-hoc perl -e style testing, but in my  
> opinion
> : thats not good enough. That is not a positive identification of a  
> new
> : vulnerability.  That is the identification of a theoretical
> : vulnerability which isn't technically a real vulnerability until its
> : been proved by a human, right?
>
> Sure, just like it is with a human tester. Many of whom cannot do the
> required follow-up either =) (be it time, resources, skillset, etc)

So true...

> : So is this inaccurate, or just unclear:
> :
> : "The fact is that vulnerability scanners can not detect  
> vulnerabilities unless
> : someone has first identified the vulnerability and created a  
> signature for its
> : detection."
>
> Inaccurate. Vulnerability scanners that use general fuzzing or common
> exploit scenarios *can* find/detect vulnerabilities possibly. It is  
> not
> reliable and should not be used to replace a pen-tester by any means.

I can agree with you there.

> : Perhaps I should write:
> :
> : "The fact is that vulnerability scanners can not positively identify
> : vulnerabilities."
>
> Inaccurate. =) Jumping back to my example above, the vulnerability  
> found
> in a Real product was positively identified by a scanner. It did a GET
> request to an RTSP enabled server and grabbed /etc/password due to a
> traversal vulnerability. The output of the vulnerability scanner  
> made it
> very clear there was an issue there as it displayed the captured file.
>
> What the scanner couldn't do, that the consultant did, was test that  
> he
> could grab other files and then share that information with me so I  
> could
> in turn share with the vendor.

Right, again because this time it wasn't a banner grab or something  
like that.
This time the scanner ran the attack and got a positive result. That  
result
was verified by a human and found to be true. But the scanner still  
has no
way to know if the result is in fact true. It just reports on it and  
allows the human
to validate the issue. So who is doing the discovery then? The scanner  
or the
human?

> : I think that "what's best" is a major part of the problem. Most  
> people
> : don't know the difference between a vulnerability scan and a manual
> : vulnerability assessment.  Most people think that they are both  
> the same
> : thing, same quality, etc. Thats an advantage for the vulnerability  
> scan
> : vendor, but its a disadvantage to the people who don't know "what's
> : best".
>
> Right, and this has been the battle you and many others have fought  
> for
> over a decade now. Educating customers on what they really need when  
> they
> come to you saying "test our systems" in so many words. If a pen- 
> test shop
> has a good sales goon, that is the first hurdle they have to jump;
> identifying what the customer really needs, because 95% of the time  
> they
> sure don't know.

Its so true and they get so confused by marketing hype.


> : I'd like people to be able to make well informed decisions so that  
> if
> : they use a vulnerability scanner they know what they are really  
> getting.
> : The fact of the matter is that vulnerability scanners are an  
> invaluable
> : tool with respect to maintaining the security of a network and doing
> : nightly checkups, but they are not nearly as accurate as the human
> : teams. As a result, we recommend to our customers they perform
> : vulnerability scans frequently and undergo intense manual  
> penetration
> : testing once or twice a year.
>
> Exactly. Most shops need a blended solution like this, and it is  
> much more
> viable financially than the message "only use a pen-test shop for real
> testing" sends. Like I said, when a company has 100k machines across  
> the
> org, vulnerability scanners certainly have their place.

Hey man, if they want to pay us hourly to test that many systems  
manually
i could find the people to do it  (yes I am joking).  We use  
vulnerability scanners
to perform reconnaissance against large networks prior to getting into
Real Time Dynamic Testing.  That said, we avoid them if our penetrations
are supposed to be covert. Thats a totally different subject though.

> : I might actually take consideration and write the article that  
> you've
> : suggested.
>
> You should, it definitely seems in line with previous posts about the
> quality of some pen-test teams. I'm sure you've been on an test that
> ended up finding many vulnerabilies, only to be told "what?! we had
> $company do a pen-test 3 months ago!"

Indeed, and half the time we get to see their results too. It scares me
what people will pay for sometimes.

>



        Adriel T. Desautels
        ad_lists () netragard com
        --------------------------------------

        Subscribe to our blog
        http://snosoft.blogspot.com