Re: Rooting out false positives

[Renaud Deraison <deraison () nessus org>]

On Jul 19, 2005, at 1:50, Scott Fuhriman wrote:

> Rather than rooting out false positives, it is a question of  
> understanding
> the vulnerability and how it can be exploited through other means  
> than an
> obvious direct approach.

Exactly. And it's also a matter of understanding why and how this  
alert showed up in the first place.

One of my main disappointement with the Nessus project in general is  
how some people tend to react when it finds some "new" flaws. A good  
example for this are some of the "generic" checks we write for FTP,  
HTTP and so forth (send a too long username or a lot of '%s%n', and  
see how the remote server reacts. If it crashes there might be a  
problem) : When Nessus finds something suspicious, it reports it to  
the end user who often dismiss the plugin output very quickly because  
a SecurityFocus search shows no such problem with the incriminated  
FTP server, or because the plugin itself mentions a problem in  
FooFTPd while the tested server was BarFTPd. In other words, a lot of  
end-users do not want to find new problems.

While these particular plugins are not false positive-proof, and  
while we should also sometime change the description to be more  
generic, a good pen tester using Nessus (as one of his many tools --  
don't get me started on the "Nessus Scan" == "Pen Test" subject)  
should definitely install the software and try to understand what's  
going on and why the plugin is generating such an alert. The plugin  
source code is here, if you're auditing one host it really is no big  
deal to have a look at what each plugin does and try to understand  
why it would generate such an alert.

We recently downloaded dozens of (free|share|crap)ware FTP and HTTP  
daemons, and our plugins found out many occurences of DoS,  
exploitable overflows and remote files access against what are  
extremely simple and outdated attacks (GET ../, USER + crap(4096),  
etc...). Of course, we also found some false positives that we fixed,  
as these servers tend to be extremely "laxist" (to say the least)  
when it comes to implementing RFCs.

So my point here is that in my experience not enough users (and pen- 
testers) really try to understand why some plugins are firing up -  
they just tend to ignore the non-obvious stuff, and more often than  
not what they really expect is a list of missing Microsoft patches  
and that's it. If you look at some other scanners, that's exactly  
what some do : have them scan your fully patched Windows box with  
dozens of insecurely configured non-Microsoft services (Apache on  
Win32, etc...) and you'll find yourself with an "everything is OK  
since you're not missing any patch" kind of report. While this might  
be acceptable (and even though, it's debatable) when you scan half a  
million IPs, it's absolutely not when you're doing a pen-test against  
one host. Also, as I said, most end-users do not want to find new  
flaws - there are so many problems already that they don't want to  
add anything to their list. I've received emails of users who  
complain to me because Nessus crashed such or such service (this may  
happen with any security scanner trying to do its job by the way -  
some servers are coded so that they crash when they do not receive  
the exact handshake they expected, so a simple port scan might kill  
them), and having them contact their vendor is most of the time out  
of the question (and then some vendors actually do contact me and ask  
me to change Nessus to stop connecting to "their" port or to fix my  
handshake with their server [true story!]).


That being said, I have other comments to make :

- If you want Nessus to avoid "false positives", make sure safe  
checks are set and go to Prefs->Global Settings and set the "Report  
Paranoia" to "Avoid false positives". This causes Nessus to disable  
some checks which are known to generate FPs (ie: because there's no  
way to remotely determine with certainty if service XXXX is patched  
or not). At the opposite, if you want to torture a server, disable  
safe checks, set the report paranoia to "Paranoid" (in Prefs), and  
enable "Thorough Checks" (also in Prefs->Global Settings). Your scan  
will run much slower (Thorough checks) and will generate a more  
chatty report but you may find some surprising problems ;

- Update Nessus before doing your scan. We've done a huge "false  
positive" hunt over the last months but we still receive report from  
people using Nessus 2.0.12 and its default set of plugins, and  
complain about some problem we fixed many months ago ;

- We're about to announce a public "False Positive Hunt" on  
Nessus.org in the next few days : help us find and fix five plugins  
generating false positives and win a free Nessus direct feed (ie: get  
the newest Nessus plugins for free with no delay). We've done our  
best to fix a majority of false positives (including backported  
patches from Red Hat, Debian and so on), but I'm sure we can  help us  
go further. If you are already sitting on a pile of false positives,  
contact me already and I'll start your "FP counter" (there's no high  
limit on the number of winners) ;

- Finally, please note that some vendors claim to have zero false  
positive. The way they do that is by changing the definition of what  
a false positive is[1]. While I'm the first to admit that Nessus can  
generate false positives (especially if not configured to "Avoid  
False Positives"), _every other scanner does_. The problem is that  
there are so many services/devices out there who implement RFCs in  
their own way that sometimes a scanner who attempts to do a full  
audit will be fooled. I'm not just talking about web servers not  
returning a 404 error code here, but bogus FTP servers, SSH daemons,  
and so on. Oh, and don't get me started on IPSes which detect the  
attack halfway thru and suddenly block the scanner for some time.


                                        -- Renaud


[1] Some people at nCircle (a competitor) also seems to be annoyed by  
this:     http://blog.ncircle.com/archives/2005/04/security_and_qu.htm



--
Renaud Deraison
http://www.nessus.org
http://www.tenablesecurity.com