Security in QA is more than exploits

[steingra at gmail.com (Andy Steingruebl)]

On Wed, Feb 4, 2009 at 7:26 PM, Paco Hope <Paco at cigital.com> wrote:

> Andy also said "I think we lose something when we start saying 'everything
> is
> relative.'" I think we lose something more important if we try to impose
> abolutes: we lose the connection to the business. No business operates on
> absolutes and blind imperatives. Few, if any, profit-focused businesses
> dogmatically fix all remotely exploitable SQL injections. Every business
> looks
> pragmatically at these things. Fixing the bug might cause the release of
> the
> product to slip by 6 weeks or a major customer to buy a competitor's
> product
> this quarter instead of waiting for the release. It's always a judgment
> call
> by the business. Even if their goal and their track record is fixing 100%
> of
> sev 1 issues before release, you know that each sev 1 issue was considered
> in
terms of its cost, impact, schedule delay and so on.


The ppint here though is that repeatable processes do matter. Having a
standard of what constitutes a given severity of bug standardized in a
policy statement is a good thing.  Sure that is hard as every application is
different, but you need a starting place.  And so while my standards don't
say "XSS always equals P1" they do say "XSS that can be discovered in an
external facing application" or even slightly more generically than that.
So my bug priority matrix does talk about business impact because that is
what matters, but I still have to give real world examples to folks who
aren't expert security testers of how to handle a bug when they come across
it.  And we need to provide clear guidance in standards because every single
bug shouldn't require an ad-hoc trage process.


> It is an outstanding idea for infosec guys to provide security test cases,
> or
> the framework for them, to QA. That beats the heck out of what they usually
> do. However, a bunch of test cases for XSS, CSRF, SQL injection and so on
> will
> not map easily to requirements or to the QA workflow. At what priority do
> they
> execute? When the business (inevitably) squeezes testing to try to claw
> back a
> day or two on the slipped schedule, can any of these security tests be left
> out? Why or why not? Without hanging them into the QA workflow with clear
> traceability, QA will struggle to prioritize them correctly and maintain
> them.
> Security requirements would make that priority and maintenance
> straightforward. At this point I'm not disagreeing with you, but taking
> your

good approach and extending it a step farther.


I undertsand this, but handing security requirements to QA folks without a
set of reeatable test cases for doing them isn't going to help much, in mos
organizations.  James Whittaker doesn't work for me :) .  And if you're
developing web applications you're probably going to have some set of
standardized testing you do.  You need to have  a repository of test cases
for certain things, and I think testing for certain type of attacks is
probably a decent starting point.  Sure you want QA to own those, but if
you're worried about buffer overflows you've going to have a bunch of
standard test cases, test scenarios, test data (long input strings, inputs
with null bytes in them, etc) that you're going to reuse a bunch of times so
that each tester isn't starting from scratch when they see the security
requirments - "Application must handle input properly and not crash."

I don't think we're far off here in what we're saying, but repeatability is
key.  Leaving the interface with QA at the level of security requriements in
a functional spec isn't going to cut it.  And, you're probably going to have
some standardized set of security requirements for a whole swath of your
applications that you might not want to repeat ad-naseum in every single
product/feature spec.  This is the place for standards, policies, and
testing guidelines so that this becomes just part of the regular QA cycle.

-- 
Andy Steingruebl
steingra at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://krvw.com/pipermail/sc-l/attachments/20090205/23d139ca/attachment.html