[Fwd: Re: [BSDCert] ARRL format]
evan at telly.org
Sun Jul 31 10:44:38 EDT 2005
>>either you have the knowledge or not and if you have it you don't need to choose
>the right answer because you know it.
>True, but multiple choice also has its advantages. It's a lot cheaper,
To be specific, the answers to multiple choice answers can be highly
automated, to the point where an exam taken at a VUE or Prometric centre
can offer a pass/fail score in real time upon completion. Diagrams,
essay questions, etc. require a warm body to grade -- how much will that
add to the cost of an exam? Can you really count on a volunteer
Also, it's important to consider how well a scoring process will scale
as numbers increase. LPI's just purchased a scanner so that scoring
speed (and capacity) can be increased from six per hour to six per minute.
There's a reason the "hands-on"Red Hat RHCE exam costs $700 -- and
they're not making much money on that. RH's real money is in education,
certification is just a tool (to them, like to many software vendors who
do certs) that helps drive training sales.
A reasonable compromise is the "fill in the blank" item, which can still
be automated while not spoonfeeding possible answers to candidates. Even
if a FITB question has multiple possible answers (and most do), that can
be accommodated in automated scoring. (Aren't regular expressions
>Besides, you can make multiple choice questions that
>need thorough understanding of the subject matter.
Very true. It's a big challenge, but a well-crafted exam containing
multiple-choice, multiple-choice-multi-answer and FITB items can perform
very well. "Deceptors" (the psychometric term for incorrect options in
a MC item) should be plausible yet not ambiguously wrong.
There is *no* science that proves hands-on exams actually do a better
job of separating skilled from unskilled compared to hands-on exams.
OTOH, hands-on exams have problems with reliability, sufficient that
they can't be psychometrically validated.
More information about the BSDCert