Re: FWTK and smap/smapd

["Paul D. Robertson" <proberts () patriot net>]

On Fri, 19 Jul 2002, Marcus J. Ranum wrote:

[Mod note:  This is starting to get tangental, but I'd like to get some 
opinions from people on the list who aren't on the CC line.  I'll only 
pass through really interesting replies on this thread to the list, but 
off-list replies are fine too.  Feel free to flame me directly if you 
think this is too far off topic.]

> Joseph S D Yao wrote:
> > without
> > commentary providing what you might call the specifications or design,
> > or the social contract between the programmer and the user, there is
> > nothing against which you can hold a piece of code and say, "THIS IS
> > WRONG!"  Code is amoral; it has an inherently situational ethic; such
[snip]

> Hmmm... you've convinced me.  I hadn't looked at it from that
> angle before.

Joe's perspective is certainly interesting- and I'd like to take advantage 
of the experienced coders on this list to poke at this a little more, 
becuase I think it's directly relevent to the software we use every day, 
as well as a potential upcomming project.

Typically, where programmer "correctness" is measured at all, it's 
measured in bugs per thousand lines of code (bugs/kloc.)  In a commercial 
setting, it's mostly about dollars at some point, and in the Open Source 
community, the "many eyeballs" theory has been expounded quite a bit.  

Darren raised some interesting points about code auditing and the 
difference between looking at someone else's code and the code you chunk 
out.  Marcus' early experiences are close to what I experienced when doing 
development (and back then I commented heavily when I didn't want to 
maintain things, and almost not at all when it was "cool code" that only 
someone "good" should be messing with.)  If the code review/audit is with 
the less-than-clued it's either an educational experience for them, or a 
complete waste of time.  

Typically, I see a "code audit" as an after-the-fact look at the code, and 
a "code review" as a during the development step-by-step thing.  

Studies (not sure if I can find direct citations at the moment) have found 
that finding bugs by code review was somewhere around 15x cheaper than 
finding them during testing, and around 70x cheaper than if a customer 
found it.  

We've got code review, unit testing, code auditing, pen testing, etc.  
Which ones work most effectively for what class of projects?  Does 
programmer "correctness" matter in security programs?

In Open Source projects, is there some sense that moving the testing out to the 
customer isn't as bad as it is in commercial development, as that's their 
cost of goods?  My gut is that sometimes that adds good eyeballs that 
wouldn't have looked otherwise, but is that accurate?

Is there a good/interesting/novel way to do code review instead of code 
auditing in Open Source?  Would that be more efficient?  (I realize that 
part of the "ease" of review over audit is that you're getting chunks of 
code at a time, and communicating with the developer in a more interactive 
way.)

> I guess what was motivating my opinion was some awful early experiences
> I had at a small security company where they had folks assigned to audit
> UNIX/C code who didn't know anything about UNIX or C. That left an

Back during the dawn of time, I worked closely with some IBM contractors 
who'd eventually start on code, but only after a complete internal design 
specification (logic flow, program goals, syntax...) and external design 
specification (every single character on every single screen accounted for 
and agreed to by the user *before* any coding started.)  I'm sure that 
some of that was predicated by the IBM studies on costs of finding bugs at 
the design versus the implementation stage (45x.)  While they were 
documenting, I'd whip out a prototype, put the user in front of it and 
find out what they really wanted.  I'm sure I killed thousands of billable 
hours.

This twisted path has a destination of sorts- 

Sometimes process gets in the way, sometimes it's more efficient to the 
overall effort, and sometimes it's just plain dumb.  Often efficiency 
isn't intuitive because the efficiency of a part of the process may not 
mean efficiency over the entire process (uh oh, I feel myself being drawn 
into agreeing with some of that old lifecycle stuff I used to dismiss 
summarily...)

Back when I used to choose security products for production environments 
where I had the ultimate security responsibility, I'd go to meet with 
vendors, do some Q&A that might be relatively harsh, beat them up about 
strategy, and generally be a pain in the butt long before I even finalized 
which products I'd think of buying.  Obviously that's mostly overkill for 
a lot of products/vendors/customers- especially as security products get 
to be more of a commodity.

This gets me to why I think I tend to gravitate to source-available 
implementations.  Even if it's relatively expensive to do, I still get to 
poke at the underlying infrastructure, even if it's not the development 
group and their processes.  I've been line-by-line through postfix, the 
diffs between OpenSSH for OpenBSD and for Linux, large potitions of a few 
kernels, the parts of the fwtk that I use, apache's mod_proxy, etc.

Now, I'm *defintely* not the greatest code reviewer on the planet.  But at 
least I have some assurance that the code does what it says it does, 
isn't obviously broken, etc.

How many people actually look at the code at all?

Can someone who knows nothing about C or Unix even start to learn how to 
do code review?  If so, is there a good way to facilitate that somewhere?  
Should the efforts be put more into the "code checking" programs?

Would a "this is how it works" document for a particular snapshot be more 
or less helpful?  Would it create more help or hinderance?

I hear all the time "I'm not a coder" when I suggest someone modify some 
Open Source thing they're using to do something they want it to do.  By 
the same token, I often bug the authors instead of authoring my own bugs.  
Is that just "the way it is" or is there something better we can do?

> But you're right - what we're really talking about is checks and
> balances. And if you just give code there's, well, just code...
> I retract my previous comments on this topic!!! :) Where's the "undo"
> button?!

Hmmm, impetus to go buy a button-making machine... ;)

I used to have a button that said "Real programmers don't comment their 
code, if it was hard to write, it should damn well be hard to read!"  I 
was chunking out serious ammounts of assembler at the time, and I feel 
sorry for anyone who ended up having to add new functionality to it (there 
weren't any bugs, and I'm sticking to that story!)

I'm currently trying to talk my management into allowing me to release 
some kernel module stuff (Solaris, Linux, *BSD) I've been working on under 
either a GPL or BSD license, and all of these things play into that 
somewhat (though the simpler of the modules should be pretty easy to 
verify, and they're mostly trivial or lightweight versions of things that 
others have done a much deeper job on.)

Paul
-----------------------------------------------------------------------------
Paul D. Robertson      "My statements in this message are personal opinions
proberts () patriot net      which may have no basis whatsoever in fact."
probertson () trusecure com Director of Risk Assessment TruSecure Corporation

_______________________________________________
firewall-wizards mailing list
firewall-wizards () honor icsalabs com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards