Solutions, Results, and Comments - Was [ISA Server and SQL Injection]

[Jeremiah Grossman <jeremiah () whitehatsec com>]

On Monday, February 21, 2005, at 12:06  PM, Mark Curphey wrote:

> At Foundstone we have tested literally thousands of apps and the most
> frequent and most insidious issues are rarely data validation issues 
> these
> days. They were a few years back but not today. They are issues with
> authorization systems, user management systems, business logic, data
> protection in (storage and transport), broken protocols, badly 
> implemented
> cryptography and such like. The bar has been raised.

> A few people keep asking me where I think validation should be done. I
> always say in the code. The argument sometimes comes back that that's 
> not
> always possible. While I completely disagree (every commercial company 
> I
> have worked for has a release cycle of every few weeks and consistently
> changes the code base so either I have been living under a rock in 
> major
> financial services companies or the vendors marketing thought they had 
> a
> good sales angle)
>
> The same is true of automated testing, especially pen testing. I 
> personally
> firmly believe that the most cost effective and efficient way to test
> software is to use the code. To say that code review is too expensive 
> is
> misleading and outright wrong. It's a common misconception perpetuated 
> by
> the uneducated. I wrote an article about it at www.softwaremag.com. Pen
> testing web apps is a Turing problem, the number of inputs has to 
> equal the
> number of outputs and that's computationally unfeasible in a modern 
> app. For
> low hanging fruit its great but for the type of issues being found now 
> the
> bar has been raised I don't think its appropriate. Again at work we 
> have
> driven trucks past app that have been given a clean bill of health by 
> the
> scanners. And that's pretty frequent not an edge case BTW. If you look 
> at
> code analysis tools as well they are getting good at finding bugs like
> overflows but flaws like people using poor PRNG or storing keys in 
> config
> files they are next to hopeless. Guess what, flaws are the most 
> insidious
> type of issue we find. Its not that these tools are of no value. We use
> automated tools for code review and will likely buy or build something 
> for
> web app pen tests this year. For the LHF it's a no brainer. The point 
> is any
> automated solution is a subset of the problem space. At the moment I 
> would
> say its less than 25% for the web app firewall space and about 20% for 
> the
> pen test space (gut feeling).


<the above has been snipped to highlight the key points>

Per your "balanced" approach, I find conflict within several key points 
of your assertions.

Web application software is not the same as other forms of software we 
are used to dealing with. They're rate of change, the types of attacks, 
and the scale of usage differs entirely.  As such, so do they're 
security requirements. You said above "...every commercial company I 
have worked for has a release cycle of every few weeks and consistently 
changes the code..." on this we agree. Then you went onto say "...code 
review is too expensive is misleading and outright wrong." This is 
where we disagree.


1) Source Code Reviews ARE expensive

While we could argue on how to define "expensive", we can all agree 
line-by-line analysis is time intensive and requires intelligent 
testers. As a result price quotes will go up into the several 10's of 
thousands range, if not higher depending on the scope of work. So we 
can use this as a basis for further discussion.

The first time a company has a code review performed, this is where 
they'll get the most bug for the buck.  Through an exhaustive process, 
a good code review can knock out a ton of security issues and prevent a 
lot of pain down the road. They also can find problems no scanner or 
black-box pen-tester will ever find, like those pesky backdoors and 
easter eggs.  The draw back is the cost of third-party source code 
reviews will prevent them from being repeatable. This is important to 
understand if you subscribe to security is a process principal.

If companies have a release cycle every few weeks (per your quote), 
having a third-party assessor constantly in-house is certainly going to 
extend way beyond anyones budget.  However, if your shipping shrink 
wrapped software or a newly designed website where the code won't/can't 
be changed for the next year, source code review is the way to go. Your 
maximizing ROI. Pro and Con.


2) You compare Source Code Reviews and Security Assessments (Pen-Tests) 
and its like comparing apples to oranges. Both solutions should be used 
in different circumstances and maintain different ROI models.

If your web application is frequently updated, this is where you might 
find security assessments (pen-tests) more cost effective. Security 
assessments are typically faster and less expensive than your typical 
source code review. Current assessment methodology will normally 
identify nearly all (if not all) of the easy issues and most of the 
hard ones. They'll also give you visibility into what a real world 
attacker would see. I find it rare for attackers to have access to your 
source code prior to targeting your website. Its going to be far more 
common for attackers to black-box you and go from there. Vulnerability 
Scanners and continuous assessment services help make make the process 
more repeatable. But neither Source Code Reviews or Security 
Assessments will ever claim to find everything.

The point I'm trying to make here is that customers have several 
options available to them depending on they're needs.  They have access 
to Source Code Reviews, Penetration Tests, Vulnerability Scanners, 
Application Firewalls, etc etc. Each with they're own set of pros and 
cons.


To say your way is the one and only way is the part I find misleading 
and irresponsible.



3) Scoping the Web Application Security problem.

We also assess the security of tons of web sites as well.  As such, we 
have comparable results.  You said your results show the most frequent 
and insidious issues today are "...authorization systems, user 
management systems, business logic, data protection in (storage and 
transport), broken protocols, badly implemented cryptography and such 
like". On the severity of these issues when they are identified, I 
believe our results would closely match. However, as far as the total 
number of vulnerabilities found by class, nothing tops basic SQL 
Injection and Cross-Site Scripting. The discrepancy might have 
something to do with our mutually different approached and/or 
methodologies.



4) Hypocritical discussion of "Thinly disguised marketing posts in 
public lists"

You began and ended your email harping on vendors for what you said are 
"Thinly disguised marketing posts in public lists". With a good 
moderator present, this is something many of us find annoying yet 
tolerable.  You then proceeded to lace your message with the very same 
negatives issues you were accusing the other vendors of. Mentioning 
your company and offerings numerous times. Lets be honest, you have a 
product to sell no different than anyone else. Yours just happens to be 
called consulting. And if anyone were to read closely... you mirror 
your own comment. "I think x and BTW I happen to make a product that 
does exactly this."




I'm going to skip over the application firewall topic because thats a 
whole other issue and I've written enough for now.



Regards,

Jeremiah Grossman