any one a CSSLP is it worth it?

[Paco at cigital.com (Paco Hope)]

On 14 Apr 2010, at 16:24, Wall, Kevin wrote:
> I just reread your Dark Reading post and I must say I agree with it
> almost 100%. The only part where I disagree with it is where you wrote:
>
>        The multiple choice test itself is one of the problems. I
>        have discussed the idea of using multiple choice to
>        discriminate knowledgeable developers from clueless
>        developers (like the SANS test does) with many professors
>        of computer science. Not one of them thought it was possible.

This is the part of the article I disagree with most, as well. Asking whether multiple choice exams can discriminate 
between clueful and clueless developers is a valid and important question to ask.  However, I believe few professors of 
computer science could discriminate between clueful and clueless developers if "developer" and "clue" have 
industry-relevant definitions.  What passes for "development" in an academic sense and what is required for "clue" in 
an academic sense are usually defined on very different axes than the axes used in industry.

So, I think asking college professors whether standardised tests are valid in this respect is posing the important 
question to the wrong people. There are notorious disconnects between what academics and industry value. Perhaps if you 
asked the folks who hire, promote, and evaluate developers, they could give a better opinion as to whether clue and 
standardised test performance correlate. Even then, I'd prefer to see something somewhat objective, like months between 
promotions versus certifications held, as opposed to calling a bunch of CIOs or VPs of Engineering and asking how well 
they think tests work.

Having said this, I am a CSSLP and I have helped write a ton of questions for the exam. I can tell you we struggle long 
and hard to write meaningful questions that actually discriminate a practitioner who has experience from a random, 
unqualified candidate. We use follow well-established psychometric principles when designing the questions. The whole 
test creation/maintenance process is ANSI-approved and audited. Careful statistics are kept on the pass/fail rates on 
individual questions to discard questions that do not discriminate well. Over time, the question bank is maintained to 
remove questions that don't test well and to write new questions that represent changes in the landscape. Some of you 
will undoubtedly dismiss this, saying "garbage in, garbage out, regardless of how pristine the pipes are." I believe 
that's too simplistic a view.

Paco