Mainframes

Are you certified or certifiable? I may be (partially) to blame.

Share this post:

A certification is the objective, external verification of the ability to perform a collection of tasks/activities critical to a job role. (IBM)

24301-06, System zMany professions require practitioners to be certified—often by passing a written exam. In information technology there are many, many types of professional certifications offered by industry groups as well as product vendors (IBM alone manages just under 500 certifications; click here for a look). Certification is very serious business, as many jobs hinge on holding particular credentials, and I think it’s safe to say that certification exams generate a certain amount of anxiety for the examinee.

But what about the examiner?

That’s a question I never considered until I was asked to join the team preparing a new certification for System z Business Analytics Solution Selling. I have to say, it was quite an intense, yet very rewarding, experience. And certainly worth a blog post!

There is nothing simple or easy about developing a certification exam. Roughly speaking, the process covers the following phases.

Know your victim (er, I mean candidate)

There are no “grades” in certification; it is a pass or fail exercise. The target is what is called the minimally qualified candidate (MQC)—a practitioner that has just enough skill to fulfill job responsibilities. The test is built under the assumption that the guru will sail through and the clueless will fail; the goal is to set the test somewhere in between these two extremes and reveal the MQC.

Set the expectation

The first phase of our process, then, was setting out exactly what that level of knowledge is that enables someone to qualify as a System z business analytics solution seller. We developed an outline of the skills we felt were essential to the job, devised knowledge categories around these skills and established a baseline number of questions for each. This became the blueprint for the test question writing.

Understand A, B, C and D

I did not previously know, but in hindsight am not surprised, that there is an entire field of study, psychometrics, dedicated to the theory and technique of human measurement (and I have no doubt you must be certified to be a psychometrician!). Before we wrote our first question we had to receive a crash course in the art and the science—psychometrics—of question writing.

14231-06-300x200, System zIt has been determined that the ideal certification question contains four multiple-choice responses (less makes guessing too easy; more turns it into an exercise in logic).

Having a single, absolutely correct answer (as opposed to “all of the above” types) has proved to be the best measure of understanding level. The other three answers must be plausible, attractive and absolutely wrong.

We were reminded—many times!—that our goal was to discriminate between those who really know and those who think they know. After reading the perfect question, the MQC should already have the answer in mind and simply need to find it in the list of responses.

Write, rewrite and write some more

Writing the first third of the questions is relatively easy. The second third requires more effort. The final third is excruciatingly painful! At the end of the process we were pouring over technical manuals and IBM Redpapers to find those last few questions. Many times the hardest part of the process is coming up with the wrong answers (known as distractors). Thinking up alternatives that are plausible, attractive and defensibly wrong is a lot harder than you would think.

Of course our dedicated psychometrician was always on hand to gently guide us whenever we broke one of the cardinal rules of question writing (no negatives, no trivia or memorization, no deliberate tricks and so on—there are a lot of guidelines!).

Set a passing grade

We had to write enough questions to support multiple versions of the exam (one of the many safeguards against cheating). Unfortunately for us, the process does not end when all the questions have been written. The final phase is determining the passing grade or cutoff score.

According to psychometrics, the line between competence and incompetence cannot be arbitrarily determined; it must be empirically justified and defensible. IBM likes to use an industry-standard process called the Modified Angoff Method for this purpose. Our team was required to individually answer each question and assess, out of an imagined field of 100 MQCs, how many we thought would be able to answer the question correctly. The results were collated, outliers and anomalies were discussed and resolved, and the passing grade was set.

Mission accomplished!

24313-06-300x200, Systems zIf you need to be certified in System z Business Analytics Solution Selling, you may be tempted to buy me a drink (or two or three) should we meet, in the hope of getting an edge in taking the exam. While I will appreciate the drinks, be aware that I signed a non-disclosure agreement not to reveal any of the details of the exam, so your efforts will fail!

But I do wish you the best of luck—and whether the exam leads to your certification, or becoming a certifiable wreck, please don’t blame the writers!

More stories

Met Office gives businesses the upper hand with fast, accurate weather data

Hybrid cloud, Mainframes, Storage

From time to time, we invite industry thought leaders to share their opinions and insights on current technology trends to the IBM Systems IT Infrastructure blog. The opinions in these posts are their own, and do not necessarily reflect the views of IBM. Weather is one of the few things in life none of us ...read more


IBM refreshes enterprise storage, bringing strong new storage technologies to IBM Z and IBM LinuxONE customers

Digital transformation, Mainframes, Storage

Written by Eric Burgener, Research Vice President, Infrastructure Systems, Platforms and Technologies Group – IDC, Sponsored by IBM With digital transformation (DX) underway in most enterprises, there is an ever-increasing need for higher performance, greater than “five nines” availability, and flexible data protection options that give customers the capability to implement a cost-effective approach in ...read more


Better together: The IBM Z and LinuxONE ecosystem reacts to IBM’s latest announcement

Encryption, LinuxONE solutions, Mainframes

Today’s IBM Z is built to enable the cloud you want with the privacy and security you need. In this hybrid multicloud world, we collaborated with an ecosystem of over 700 Independent Software Vendors (ISVs) to deliver more than 4,000 applications bringing additional value to you. Several of these ISVs shared their perspective and offered their ...read more