Security in QA is more than exploits

[steingra at gmail.com (Andy Steingruebl)]

On Wed, Feb 4, 2009 at 11:17 AM, Paco Hope <Paco at cigital.com> wrote:


> Before anyone talks about vulnerabilities to test for, we have to figure
> out what the business cares about and why. What could go wrong? Who cares?
> What would the impact be? Answers to those questions drive our testing
> strategy, and ultimately our test plans and test cases.


Paco,

I don't really read what Robert wrote this way.  I think what this general
"risk management" approach misses is that certain things are always going to
be defects, bugs, etc.  Sure there are differences per-business and
per-application.  All bugs aren't created equal.  But I think we lose
something when we start saying "everything is relative."  Each application,
each business, each org needs a testing plan, strategy, and a definition of
what they care about.  At the same time there are going to be common types
of tests that everyone performs.  All Robert is pointing out is that if
certain classes of vulnerabilities are important to you, then you want to
have a common testing process for them.

Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in
> impact to the business. That is, you just toss as many as you can into your
> test plan and test for as much as you can. This isn't how testing is
> prioritized.


Again, I don't think he's saying this at all.  Where I work every XSS is
absolutely critical, and we get them fixed immediately.  this might not be
the case elsewhere.  Some folks don't really worry about XSS that much.
Because I can find differences though doesn't mean that everything is
relative.  Authentication bypass, SQL Injection, these types of things tend
to rate HIGH/P1/Major for almost everyone, and I think.


> You don't organize testing based on which top X vulnerabilities are likely
> to affect your organization (as the blog suggests). Likelihood is one part
> of the puzzle. Business impact is the part that is missing. You prioritize
> security tests by risk severity?that marriage of likelihood and impact to
> the business. If I have a whole pile of very likely attacks that are all low
> or negligible impact, and I have a few moderately likely attacks that have
> high impact, I should prioritize my testing effort around the ones with
> greater impact to my business.


Again - fair enough.  But at the same time you also prioritize around effort
to test and avoid, right?

Bias #4 is the treatment of testers like second class citizens. In the blog
> article, developers are "detail oriented" have a "deep understanding of
> flows." Constrast this with QA who merely understand "what is provided to
> them." They sound impotent, as if all they can do is what they're told.
> Software testing, despite whatever firsthand experience the author may have,
> is a mature discipline. It is older and more formalized than "security" as a
> discipline. Software testing is older than the Internet or the web. If
> software testing as a discipline has adopted security too slowly, given
> security's rise to the forefront in the marketplace, that might be a
> legitimate criticism. But I don't approve of the slandering QA by implying
> that they just take what's given them and execute it. QA is hard and there
> are some really bright minds working in that field.


I don't think Robert's comments were about the general field/discipline of
QA.  His commentary was more about the types of QA organizations he has come
across.  My own experience (albeit limited as well) has found a relative
lack of highly skilled QA folks as well.  There are people responsible for
quality that are at the level you're talking about but I still bet they are
more the exception than the rule.  Most QA organizations are staffed with
people writing relatively simple tests, running through positive functional
testing, etc.  I think the point here is that you have to tailor
expectations to the organization you have.  Much in the same way that if you
have mostly junior programmers who are lucky to get their code to compile
you're probably not going to have a lot of luck training them on formal
proofs, rigorous design, etc.

-- 
Andy Steingruebl
steingra at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://krvw.com/pipermail/sc-l/attachments/20090204/9f6a2e0a/attachment.html