What do you like better Web penetration testing or static code analysis?

[arian.evans at anachronic.com (Arian J. Evans)]

The world of web software is the future and the future is a wild
open-ended place by design. I for one would like to keep it that way.

You guys that write a lot of ideological software SDL-theory books can
keep your dinosaur Multics.

About 4 years ago I shifted my focus away from static analysis to
focus entirely on black box testing, not out of sexiness or
"fun-factor", but because:


1) Software security needs to become operationalized, independent of
any discussions about SDL.

I think in many cases you need SDL and operational controls, but at a
minimum you need operational controls. I believe automated BB analysis
is one of the only ways to accomplish operationalizing software
security. I'm working with almost 1800 websites right now which
requires me to look at the problem intimately at full scale.


2) Secondly I think BB helps answer the question "What are my top
trivially-exploitable threats in my web software?" reasonably well. BB
alone can provide better answers to this question than many forms of
static analysis alone can provide.

Sorry, I know a lot of you on this list disagree with #2, but you're
wrong. Real-world compromises to web software completely validate #2.

Luckily, you don't have to chose one versus the other.

BB and static analysis fit together hand in glove, and obviously some
of us on this list are working to explore the best marriage of the
two. I think we will be able to really dial in the efficiency of
analysis efforts once we have a clearer understanding of where BB and
static overlap, and where they don't.

Just to be clear- there are absolutely many times doing BB-only
analysis where I would love to have access to source.

Static analysis can be so much faster and more effective at verifing
certain classes of defects in source (mostly syntax
injection/manipulation issues, and some plain vanilla auth issues),
especially once you already know what you are looking for. Which again
speaks to the value of hybrid solutions.

---

Finally - to add to Kevin's point - I have seen situations on where
developers respond much more positively to talking to someone who can
read and explain their code. In fact, I just spent this week on the
road having to talk to developers and look at source code for this
very reason. (Their lack of understanding of BB analysis/results.)

Developers really like to talk to other developers, and not some
hotshot pen-tester with crazy hair and earrings that is full of Way
Cool Exploits and has no real idea how software development happens.

Unfortunately, in the information-security industry, there are far
more Way Cool Exploit Dudes than there are professionals with a
background in software development. It is what it is.

Human_Hand_Driven(BB + Static + Developer Interpretation)==Great Combo Answer

---
Arian Evans



On Fri, Apr 23, 2010 at 11:08 AM, Brian Chess <brian at fortify.com> wrote:
> I like your point Matt. ?Everybody who's responded thus-far has wanted to
> turn this into a discussion about what's most effective or what has the most
> benefit, sort of like we were comparing which icky medicine to take or which
> overcooked vegetable to eat. ?Maybe they don't get any pleasure from the
> work itself.
>
> It sounds as though you need to change up your static analysis style. ?A few
> years back we ran competitions at BlackHat where we found we could identify
> and exploit vulnerabilities starting from static analysis just as quickly as
> from fuzzing. ?Here?s an overview:
>
> http://reddevnews.com/Blogs/Desmond-File/2008/08/Iron-Chef-Competition-at-Bl
> ack-Hat-Cooks-Up-Security-Goodness.aspx
>
> Interviews with Charlie Miller and Sean Fay:
> http://blog.fortify.com/blog/2009/05/02/Iron-Chef-Interviews-Part-1-Charlie-
> Miller-1-2
> http://blog.fortify.com/blog/2009/05/02/Iron-Chef-Interviews-Part-2-Sean-Fay
>
> Brian
>
> On 4/23/10 7:05 AM, "Matt Parsons" <mparsons1980 at gmail.com> wrote:
>
> > Gary,
> > I was not stating which was better for security. ?I was stating what I
> > thought was more fun. ? I feel that penetration testing is sexier. ?I find
> > penetration testing like driving a Ferrari and static code analysis like
> > driving a Ford Taurus. ? I believe with everyone else on this list that
> > software security needs to be integrated early in the development life
> > cycle. ?I have also read most of your books and agree with your findings.
> > As you would say I don't think that penetration testing is magic security
> > pixie dust but it is fun when you are doing it legally and ethically. ?My
> > two cents.
> > Matt
> >
> >
> > Matt Parsons, MSM, CISSP
> > 315-559-3588 Blackberry
> > 817-294-3789 Home office
> > "Do Good and Fear No Man"
> > Fort Worth, Texas
> > A.K.A The Keyboard Cowboy
> > mailto:mparsons1980 at gmail.com
> > http://www.parsonsisconsulting.com
> > http://www.o2-ounceopen.com/o2-power-users/
> > http://www.linkedin.com/in/parsonsconsulting
> > http://parsonsisconsulting.blogspot.com/
> > http://www.vimeo.com/8939668
> > http://twitter.com/parsonsmatt
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: sc-l-bounces at securecoding.org [mailto:sc-l-bounces at securecoding.org]
> > On Behalf Of Gary McGraw
> > Sent: Thursday, April 22, 2010 2:15 PM
> > To: Peter Neumann; Secure Code Mailing List
> > Subject: Re: [SC-L] What do you like better Web penetration testing or
> > static code analysis?
> >
> > I hereby resonate with my esteemed colleague and mentor pgn. ?But no puns
> > from me.
> >
> > gem
> >
> >
> > On 4/22/10 1:57 PM, "Peter Neumann" <neumann at csl.sri.com> wrote:
> >
> >
> >
> > Matt Parsons wrote:
> > > What do you like doing better as application security professionals, web
> > > penetration testing or static code analysis?
> >
> > McGovern, James F. (P+C Technology) wrote:
> > > Should a security professional have a preference when both have
> > > different value propositions? While there is overlap, a static analysis
> > > tool can find things that pen testing tools cannot. Likewise, a pen test
> > > can report on secure applications deployed insecurely which is not
> > > visible to static analysis.
> > >
> > > So, the best answer is I prefer both...
> >
> > Both is better than either one by itself, but I think Gary McGraw
> > would resonate with my seemingly contrary answer:
> >
> > ? BOTH penetration testing AND static code analysis are still looking
> > ? at the WRONG END of the horse AFTER it has left the DEVELOPMENT BARN.
> > ? Gary and I and many others have for a very long time been advocated
> > ? security architectures and development practices that greatly enhance
> > ? INHERENT TRUSTWORTHINESS, long before anyone has to even think about
> > ? penetration testing and static code analysis.
> >
> > ? This discussion is somewhat akin to arguments about who has the best
> > ? malware detection. ?If system developers (past-Multics) had paid any
> > ? attention to system architectures and sound system development
> > ? practices, viruses and worms would be mostly a nonproblem!
> >
> > ? Please pardon my soapbox.
> >
> > ? ? The past survives.
> > ? ? The archives
> > ? ? have lives,
> > ? ? not knives.
> > ? ? High fives!
> >
> > ? ? (I strive
> > ? ? to thrive
> > ? ? with jive.)
> >
> > PGN
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > _______________________________________________
> >
> >
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > _______________________________________________
> >
> > _______________________________________________
> > Secure Coding mailing list (SC-L) SC-L at securecoding.org
> > List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> > List charter available at - http://www.securecoding.org/list/charter.php
> > SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> > as a free, non-commercial service to the software security community.
> > Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> > _______________________________________________
>
>
> _______________________________________________
> Secure Coding mailing list (SC-L) SC-L at securecoding.org
> List information, subscriptions, etc - http://krvw.com/mailman/listinfo/sc-l
> List charter available at - http://www.securecoding.org/list/charter.php
> SC-L is hosted and moderated by KRvW Associates, LLC (http://www.KRvW.com)
> as a free, non-commercial service to the software security community.
> Follow KRvW Associates on Twitter at: http://twitter.com/KRvW_Associates
> _______________________________________________