Security in QA is more than exploits

[Paco at cigital.com (Paco Hope)]

All,

I just read Robert's blog entry about "re-aligning training expectations for QA." (http://bit.ly/157Pc3) It has some 
useful points that both developers and so-called "security people" need to hear. I disagree with some implicit biases, 
however, and I think we need to get past some stereotypes that sneak out in the article.

Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is more non-web software than web software 
in the world, and non-web software does more important stuff than all the web software combined. The role of security 
in _software_ testing is vital, and the presence or absence of web technologies does not change that. Despite writing a 
recent book on Web Security Testing, I know my place in the universe. Quality assurance and software testing are 
disciplines far older than the web, and their mission is so much bigger than finding vulnerabilities.

Bias #2 is vulnerabilities ?ber alles. By talking about weaving vulnerabilities into security test plans, we've 
overlooked the first place where security goes into the QA process: test strategy. Look at any of the prominent folks 
in QA (Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I'm privileged to share podiums with at QA 
conferences like STAR West, STAR East, and Better Software, and you'll see that security is part of the overall 
risk-based testing strategy. Risk-based testing has been around for a really long time. Longer than the web.

Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. 
What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and 
ultimately our test plans and test cases.

Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in impact to the business. That is, you just 
toss as many as you can into your test plan and test for as much as you can. This isn't how testing is prioritized.

You don't organize testing based on which top X vulnerabilities are likely to affect your organization (as the blog 
suggests). Likelihood is one part of the puzzle. Business impact is the part that is missing. You prioritize security 
tests by risk severity?that marriage of likelihood and impact to the business. If I have a whole pile of very likely 
attacks that are all low or negligible impact, and I have a few moderately likely attacks that have high impact, I 
should prioritize my testing effort around the ones with greater impact to my business.

Bias #4 is the treatment of testers like second class citizens. In the blog article, developers are "detail oriented" 
have a "deep understanding of flows." Constrast this with QA who merely understand "what is provided to them." They 
sound impotent, as if all they can do is what they're told. Software testing, despite whatever firsthand experience the 
author may have, is a mature discipline. It is older and more formalized than "security" as a discipline. Software 
testing is older than the Internet or the web. If software testing as a discipline has adopted security too slowly, 
given security's rise to the forefront in the marketplace, that might be a legitimate criticism. But I don't approve of 
the slandering QA by implying that they just take what's given them and execute it. QA is hard and there are some 
really bright minds working in that field.

As someone who has been training in risk-based security testing for several years now, I totally agree with some 
points, but very much disagree with others. I agree that the "bug parade" (as we call it) of top X vulnerabilities to 
find is the wrong way to teach security testing. Risk management, though, has been a fundamental part of mainstream QA 
for a very long time. Likewise, risk management is the same technique that good "security people" use to prioritize 
their results. Risk management is certainly how the business is going to make decisions about which issues to remediate 
and when. Risk management is what ties this all together.

If there's something that QA needs to learn that they're not already learning, it's the weaving of "security" into the 
risk management techniques they already know how to do. If testers fall short in their ability to apply risk management 
techniques, then they are falling short against the QA yardstick, there's nothing particularly security-related in this 
observation. If they are applying mature QA practices with modern risk management, but are not adequately addressing 
the software-induced business risks facing their stakeholders, then some security training might be in order. That 
security training should be built on the foundation of modern QA practice, including risk-based testing.

So, in some ways we agree: speak the lingo of QA. But in other ways we disagree. I think the original article fails to 
give credit to the decades of substantial research and practice in QA. In other words, it's a lot more than speaking 
the language. It is standing on the shoulders of giants, not their toes.

My $0.02.

Paco