September 19, 2013 | Written by: Paul DiMarzio
Share this post:
A certification is the objective, external verification of the ability to perform a collection of tasks/activities critical to a job role. (IBM)
Many professions require practitioners to be certified—often by passing a written exam. In information technology there are many, many types of professional certifications offered by industry groups as well as product vendors (IBM alone manages just under 500 certifications; click here for a look). Certification is very serious business, as many jobs hinge on holding particular credentials, and I think it’s safe to say that certification exams generate a certain amount of anxiety for the examinee.
But what about the examiner?
That’s a question I never considered until I was asked to join the team preparing a new certification for System z Business Analytics Solution Selling. I have to say, it was quite an intense, yet very rewarding, experience. And certainly worth a blog post!
There is nothing simple or easy about developing a certification exam. Roughly speaking, the process covers the following phases.
Know your victim (er, I mean candidate)
There are no “grades” in certification; it is a pass or fail exercise. The target is what is called the minimally qualified candidate (MQC)—a practitioner that has just enough skill to fulfill job responsibilities. The test is built under the assumption that the guru will sail through and the clueless will fail; the goal is to set the test somewhere in between these two extremes and reveal the MQC.
Set the expectation
The first phase of our process, then, was setting out exactly what that level of knowledge is that enables someone to qualify as a System z business analytics solution seller. We developed an outline of the skills we felt were essential to the job, devised knowledge categories around these skills and established a baseline number of questions for each. This became the blueprint for the test question writing.
Understand A, B, C and D
I did not previously know, but in hindsight am not surprised, that there is an entire field of study, psychometrics, dedicated to the theory and technique of human measurement (and I have no doubt you must be certified to be a psychometrician!). Before we wrote our first question we had to receive a crash course in the art and the science—psychometrics—of question writing.
It has been determined that the ideal certification question contains four multiple-choice responses (less makes guessing too easy; more turns it into an exercise in logic).
Having a single, absolutely correct answer (as opposed to “all of the above” types) has proved to be the best measure of understanding level. The other three answers must be plausible, attractive and absolutely wrong.
We were reminded—many times!—that our goal was to discriminate between those who really know and those who think they know. After reading the perfect question, the MQC should already have the answer in mind and simply need to find it in the list of responses.
Write, rewrite and write some more
Writing the first third of the questions is relatively easy. The second third requires more effort. The final third is excruciatingly painful! At the end of the process we were pouring over technical manuals and IBM Redpapers to find those last few questions. Many times the hardest part of the process is coming up with the wrong answers (known as distractors). Thinking up alternatives that are plausible, attractive and defensibly wrong is a lot harder than you would think.
Of course our dedicated psychometrician was always on hand to gently guide us whenever we broke one of the cardinal rules of question writing (no negatives, no trivia or memorization, no deliberate tricks and so on—there are a lot of guidelines!).
Set a passing grade
We had to write enough questions to support multiple versions of the exam (one of the many safeguards against cheating). Unfortunately for us, the process does not end when all the questions have been written. The final phase is determining the passing grade or cutoff score.
According to psychometrics, the line between competence and incompetence cannot be arbitrarily determined; it must be empirically justified and defensible. IBM likes to use an industry-standard process called the Modified Angoff Method for this purpose. Our team was required to individually answer each question and assess, out of an imagined field of 100 MQCs, how many we thought would be able to answer the question correctly. The results were collated, outliers and anomalies were discussed and resolved, and the passing grade was set.
If you need to be certified in System z Business Analytics Solution Selling, you may be tempted to buy me a drink (or two or three) should we meet, in the hope of getting an edge in taking the exam. While I will appreciate the drinks, be aware that I signed a non-disclosure agreement not to reveal any of the details of the exam, so your efforts will fail!
But I do wish you the best of luck—and whether the exam leads to your certification, or becoming a certifiable wreck, please don’t blame the writers!