Continuous Integration (was Re: Coverage and a recent paper by L. Suto)

["Andre Gironda" <andreg () gmail com>]

On 10/29/07, J.M. Seitz <lists () bughunter ca> wrote:
> > Developers can also improve automation through continuous
> > system tests (e.g. using Canoo WebTest, Jameleon, staf.sf.net, etc).
>
> Continuous integration is a definite plus, if you can couple a per-build
> smoke test with a nightly build robustness test (fuzzing), you are sitting
> pretty. And generally it's a great first warning that something is up. The
> real problem is in designing an automation framework that is extensible
> enough to not only allow custom fuzzers, but is able to adapt to changes in
> the test targets, network parameters, protocol/file format changes, etc.

Wow, somebody is talking my language!  Smoke test?  Nightly build?
Nobody uses those words!

Can you add intake testing with something like ct-eclipse.tigris.org,
clover, fortifysoftware sca, and test-ng all integrated?  if the
developer gets up from their seat, can it run findbugs with extra
security checks, too?  and a pony?!111?

> > Design and code review is difficult to automate, but we can "automate"!
> > processes through workflow tools
>
> Atlassian's JIRA also has some excellent workflow tools that can extend into
> any one of their products and SVN, etc. It is easily automatable through
> build tools like Tinderbox, Hudson, etc.

Seriously, yes.  You really are mirroring my words.  It almost feels
like you are mocking me (pun intended there).

> > More advanced operations/maintenance
> > security testing can be done using EFS and/or
> > WI/AppScan/Hailstorm
>
> How do you know where your weak areas of code are? Is EFS going to
> successfully dissect a binary protocol and give you metrics that are
> useable? Not really. In my experience, and I love Jared like a brother, EFS
> works very well against ASCII protocols, with a known set of seeds. If you
> turn it loose against an unknown binary protocol, it doesn't do so well.
> This is where we need to advance the whole notion of automated binary
> protocol dissection, and then couple that will a block-based (or genetic if
> you want) fuzzer that can then utilize the results that your dissection
> yielded. Look for something like this in December, by some goofy looking
> Canadian dude :)

I saw Charlie Miller talk at Toorcon 9 last weekend and he showed the
audience how to "do it" better with EFS.  By "do it", I sure hope you
know what I'm talking about ;>  If not, you can search for the full
pornographic details via Google Video...

Even giving into your point (which I wasn't arguing against in the
first place) - existing binary protocols can be Matasano'd, Sulley'd,
and Canadian-guy'd to keep those methods viable.  Protocol dissection
work is so tedious.  I wish the people who wrote the protocols just
ran them through a model-checker like SPIN (or the code through
modern-day equivalents like PREVENT or Java PathFinder).  Better yet
would be to do a very good job at design review... see: Justin Schuh
(TAOSSA), Brenda Larcom (Octotrike), the SwA Acquisition WG, NIST
SAMATE, and the SATC group at NASA.

> > Unfortunately, the largest problem we're facing with the
> > above strategies is the growing rate of code vs. the growing
> > rate of qualified experts.  Make sure to check out Felix
> > Linder (FX)'s talk from HITB 2007, available here -
> > http://conference.hackinthebox.org/hitbsecconf2007kl/materials
> > /D2T1%20-%20Felix%20Lindner%20-%20%20%09%20Attack%20Surface%20
> > of%20Modern%20Applications.pdf
>
> You got that right, in a development shop you are lucky to have one person
> in the whole company that can actually test the corner cases, use a debugger
> to track down problems (without source), demonstrate attack surface,
> yadayada. Problem is there are all kinds of jobs for security engineers,
> pentesters, app-pen testers, etc. But how often do you see a job posting
> like: "Hax0r In the QA Department Needed". That's where the demand _has_ to
> start moving towards. And please folks, pay your QA hackers well :)

Were you there for my talk at Toorcon?  They don't have the slides up
yet, but you can get a similar copy of the talk here -
https://www.owasp.org/index.php/Image:Andreg-cpt-owasp-msp.pdf

What you're saying about "lucky to have one person" is almost exactly
what I said.  I also included the opposite: you're unlucky to have one
specific person: the ignorant and lazy guy/gal who checks in all their
code at the last minute of the project, completely untested, and then
management makes you go live (and probably doesn't even fire the
jerk).

SQA is dead.  Reasons:

1) Security testing has evolved to be as important or more important
than performance testing.  I read this on a blog somewhere lately and
whoever wrote it has a good idea of the landscape going on right now.
So do some of the people at the VERIFY conference that I am attending
right now in DC (Security / Automated / Quality testing con)

2) Performance testing has evolved to be the "new" quality testing
(and very automated - look at tools like YSlow as an example)

3) Anyone who is still so good at quality testing that anyone would
skill want/need their skills is really a developer-tester.  See: Jason
Huggins (ex-ThoughtWorks developer of Selenium who now works for
Google with the title of software quality engineer)

The remaining title "SQE" is a misnomer today, just as SQA is.
Developer-testers should be involved in writing functional testing
tools, system testing tools, automated anything, unit test integration
with the above, continuous integration with the lot, and
anything-everything test-driven development (TDD) and design for
[test|operations|security].  Other SQE's and quality testers should
consider careers in performance or security testing before AOL fires
another 2k of you little buggers (or should I say,
former-bug-finders)?

dre
_______________________________________________
Dailydave mailing list
Dailydave () lists immunitysec com
http://lists.immunitysec.com/mailman/listinfo/dailydave