[Fwd: Re: [BSDCert] ARRL format]

[Evan Leibovitch]

Lucas wrote:

>>either you have the knowledge or not and if you have it you don't need to choose
>>    
>>
>the right answer because you know it.
>
>True, but multiple choice also has its advantages. It's a lot cheaper,
>
To be specific, the answers to multiple choice answers can be highly 
automated, to the point where an exam taken at a VUE or Prometric centre 
can offer a pass/fail score in real time upon completion. Diagrams, 
essay questions, etc. require a warm body to grade -- how much will that 
add to the cost of an exam? Can you really count on a volunteer 
infrastructure?

Also, it's important to consider how well a scoring process will scale 
as numbers increase. LPI's just purchased a scanner so that scoring 
speed (and capacity) can be increased from six per hour to six per minute.

There's a reason the "hands-on"Red Hat RHCE exam costs $700 -- and 
they're not making much money on that. RH's real money is in education, 
certification is just a tool (to them, like to many software vendors who 
do certs) that helps drive training sales.

A reasonable compromise is the "fill in the blank" item, which can still 
be automated while not spoonfeeding possible answers to candidates. Even 
if a FITB question has multiple possible answers (and most do), that can 
be accommodated in automated scoring. (Aren't regular expressions 
wonderful?)

>Besides, you can make multiple choice questions that
>need thorough understanding of the subject matter.
>
Very true. It's a big challenge, but a well-crafted exam containing 
multiple-choice, multiple-choice-multi-answer and FITB items can perform 
very well.  "Deceptors" (the psychometric term for incorrect options in 
a MC item) should be plausible yet not ambiguously wrong.

There is *no* science that proves hands-on exams actually do a better 
job of separating skilled from unskilled compared to hands-on exams. 
OTOH, hands-on exams have problems with reliability, sufficient that 
they can't be psychometrically validated.

- Evan