Any subject matter expert (SME) who has written test questions will tell you it is a lot harder and much more time consuming than they initially expected. Many believe they can simply write down any topic-related question that pops in their mind. Easy, right? Not so fast. Questions should be relevant to a defined job role, correctly map to a specific test objective, be written at the appropriate level of difficulty, and ensure the right answer is always right and the wrong answers are plausible, but wrong. Why go through all this work? Because each question should do its job in helping to determine if a candidate is qualified to pass the test.

Overall passing scores do not provide insight into how each question is performing but individual question statistics can. Below are two sets of statistics I use to see how well a specific test question is performing.

Table 1 below illustrates a question that is performing well. How do I know? I look at three data points:

• All (PVal) - This tells me 57.1% of all candidates who take this test are getting this question right. If the cut score is set at 65% then this question falls on the slightly more difficult side but still within an acceptable range.
• Discrimination Index - A positive discrimination index of 0.286 indicates there is a positive correlation between candidates getting this question right and passing the exam, and candidates getting this question wrong and failing the exam. The higher the number, the better the discrimination which means the question is helping to distinguish those who should pass versus those who should not pass.
• Option Analysis - This shows a nice, even distribution of the percentage of candidates selecting ALL options (A, B, C, D) which means each option is appealing and doing work to make this a solidly performing question.

Table 2 below illustrates a question that has some performance issues based on these same three data points:

• The PVal is high indicating 83.3% of all candidates are getting this question right (it’s too easy).
• The Discrimination Index has a negative correlation of -0.143 indicating candidates who should be failing the exam are actually getting this question right which is helping them on the exam. Conversely this question can also be hurting qualified candidates as they might be getting this question wrong for some reason (despite the high pass rate).
• The Option Analysis indicates no one is choosing option “B” so that answer choice is not doing any work in this question. If it’s a throw-away option then that increases the odds of a candidate guessing correctly.

Discriminating questions are the backbone of a high quality exam and they help differentiate between candidates who should be certified versus those who should not. So just remember that a good exam question goes well beyond grammar and punctuation - it’s how hard each question works for the good of the whole test.

----------------------------

Beyond the Credential, a blog from The SAS Learning Post, goes beyond the impersonal and sometime sterile world of technical exams and shines the light on the more human side of testing.