[BSDCert] ARRL format

[Evan Leibovitch]

Richard Bejtlich wrote:

>So, if I memorize that the FCC is in charge, and the underlying
>material says the FCC is in charge, how is that bad?
>
It's not bad, it just means that you know one snippet of information. If 
the exam's objective is to test you for specific snippets of information 
and you know those snippets, then you pass. But most IT certifications 
are geared to far more than that.

>Similarly, if I
>memorize an answer that says "rpcinfo shows RPC services on a remote
>system", and that is true, how is that bad?
>  
>

It's bad because it means that instead of studying and being aware of 
the whole realm of inter-process communications (which is likely the 
intent of the certification), you don't bother to learn anything except 
what rpcinfo does, because you know that that's the only question on the 
exam of that realm.

A 75-item LPI exams requires 90 minutes to complete (I've proctored 
exams and most people need every minute). Even a grueling, three hour 
exam will only let you ask about 150 questions, which will enable you to 
usually test only a very small proportion of the whole realm of subject 
matter being certified.

When the questions are known then you allow the candidate to _not_ study 
subjects that you want to certify but won't be asking about on one 
particular instance of the test. They only need to study snippets, not 
the whole realm you're certifying. And that _is_ bad.

>>If you have proper feedback, any candidate can comment on the questions
>>at the time they take the exam. This is an effective way to identify
>>problems with the help of those with a stake in it (the test takers),
>>and it doesn't matter if the whole pool is open or not.
>>    
>>
>
>That's nice in theory.  Try sitting through a multi-hour exam and then
>decide to critique the questions.
>
It's nice in practice. Thousands of LPI exams have had comments added at 
the end (both PBT and CBT) and many have emailed in comments soon after 
sitting the exams. I've proctored exams where people point out perceived 
problems as they're handing in the answer sheet. The volume of feedback 
has been such that we've had to automate it using the "request tracker" 
package.

IMO it's a matter of community. If you're just seen as a vendor peddling 
exams then candidates won't do much to help. If you're seen to be part 
of your community -- that your exam takers see themselves having a stake 
in the cert --  it's surprising how much unsolicited advice comes in. 
("Be careful what you wish for..." and all that :-) ).

Then again, maybe some of it's because we made a conscious decision to 
use multiple 90-minute exams rather than cram the whole cert into a 
single multi-hour marathon; I think we made the right choice. Consider 
factors such as "the ability of candidates to provide feedback" when 
choosing the number and duration of exams required to complete a cert.

- Evan