[WEB SECURITY] SATE?

[jim at manico.net (Jim Manico)]

Great SATE reply from Vadim Okun:

> We have been releasing the real deep data. There have been delays, but there are no sinister reasons for the delays.
>
> The results of the 2nd SATE (our report and all data) will be released in June (We promised to release between 
> February and May, but we're late with the report).
>
> We released the results of the 1st SATE last summer: our report, the raw tool reports, and our analysis of the 
> reports. The data is available (below the list of cautions) from
> http://samate.nist.gov/SATE2008.html
>
> or a direct link:
> http://samate.nist.gov/SATE2008/resources/sate2008.tar.gz
>
> I will answer some specific points in Jim's email below, but first, let me describe some limitations of SATE and how 
> we are addressing them. SATE 2008 had a number of big limitations, including:
>
> 1) We analyzed a non-random subset of tool warnings
> 2) Determining correctness of tool warnings turned out to be more complicated than a binary true/false decision. 
> Also, determining relevance of a warning to security turned out more difficult than we thought.
> 3) In most cases, we did not match warnings from different tools that refer to the same weakness. When we started 
> SATE, we thought that we could match warnings by line number and weakness name or CWE id. In fact, most weaknesses 
> are more complex - see Section 3.4 of our report.
> 4) Analysis criteria were applied inconsistently.
>
> In our publicly released analysis, we used the confirmed/unconfirmed markings instead of true/false markings. We 
> describe the reasons for this in our report - Section 4.2, page 29 of
> http://samate.nist.gov/docs/NIST_Special_Publication_500-279.pdf
>
> In SATE 2009, we made some improvements, including:
>
> 1) Randomly select a subset of tool warnings for analysis
> 2) We also looked at tool warnings that were related to human findings by security experts.
> 3) Use 4 categories for analysis of correctness: true, true but insignificant (for security), false, unknown.  It is 
> an improvement, but there are still problems: for example distinguishing true from true but insignificant is often 
> hard.
>
>    
> > 1) false positive rates from these tools are overwhelming
> >      
> First, defining a false positive is tough.  Also in SATE 2008, the criteria that we used for analysis of correctness 
> were inconsistent, we did not analyze a random sample of warnings, our analysis had errors. Steve gave a good example 
> in his reply. We corrected some of these problems in 2009, but still way to go.
>
>    
> > 2) the work load to triage results from ONE of these tools were
> > man-years
> >      
> We are not the developers of the test cases, our knowledge of the test case code is very limited. Also, we used tools 
> differently from their use in practice. We analyzed tool warnings for correctness and looked for related warnings 
> from other tools, whereas developers use tools to determine what changes need to be made to software, auditors look 
> for evidence of assurance.
>
>    
> > 3) by every possible measurement, manual review was more cost effective
> >      
> As Steve said, SATE did not consider cost. In SATE 2009, we had security contractors analyze two of the test cases 
> and report the most important security weaknesses. We then looked at tool warnings that report the same (or related) 
> weakness. This will be released as part of 2009 release (The data set is too small for statistical conclusions.)
>
> A big limitation of SATE has been the lack of ground truths about what security weaknesses really are in the test 
> cases. This determination is hard for reasonably large software. We are trying to address this: manual analysis by 
> security contractors, "CVE-selected" test cases.
>
>    
> > the NIST team chose only a small percentage of the automated findings to review
> >      
> A small percentage by itself should not be a problem if the selection of tool warnings is done correctly (it was not 
> done correctly in SATE 2008).
>
> Vadim
>
> ________________________________________
> From: Jim Manico [jim at manico.net]
> Sent: Thursday, May 27, 2010 5:31 PM
> To: 'Webappsec Group'
> Subject: [WEB SECURITY] SATE?
>
> I feel that NIST made a few errors in the first 2 SATE studies.
>
> After the second round of SATE, the results were never fully released to
> the public - even when NIST agreed to do just that at the inception of
> the contest. I do not understand why SATE censored the final results - I
> feel such censorship hurts the industry.
>
> And even worse, I felt that vendor pressure encouraged NIST to not
> release the final results. If the results (the real deep data, not the
> executive summary that NIST release) were favorable to the tool vendors,
> I bet they would have welcomed the release of the real data. But
> instead, vendor pressure caused NIST to block the release of the final
> data set.
>
> The problems that the data would have revealed is:
>
> 1) false positive rates from these tools are overwhelming
> 2) the work load to triage results from ONE of these tools were man-years
> 3) by every possible measurement, manual review was more cost effective
>
> Even worse were the methods around the process of this "study". For
> example, all of the Java app's in this "study" contained poor hash
> implementations. But because the tools (none of them) could see this,
> that "finding" was completely ignored. The coverage was limited ONLY to
> injection and data flow problems that tools have a chance of finding. In
> fact, the NIST team chose only a small percentage of the automated
> findings to review, since it would have taken years to review everything
> due to the massive number of false positives. Get the problem here?
>
> I'm discouraged by SATE. I hope some of these problems are addressed in
> the third study.
>
> - Jim
>
> ----------------------------------------------------------------------------
> Join us on IRC: irc.freenode.net #webappsec
>
> Have a question? Search The Web Security Mailing List Archives:
> http://www.webappsec.org/lists/websecurity/archive/
>
> Subscribe via RSS:
> http://www.webappsec.org/rss/websecurity.rss [RSS Feed]
>
> Join WASC on LinkedIn
> http://www.linkedin.com/e/gis/83336/4B20E4374DBA
>
>