Psychometry Explainer #3

How distracting is your distractor!

Most professionals who use psychometric assessment instruments for talent-related decision-making place a great deal of emphasis on the quality of items; and rightly so. After all, what can be more important than the items used to measure latent constructs?

Well, there is something that is just as important as the item itself, if not more- the response options, or distractors, as they are known in psychometric parlance.

Distractors are used in MCQs and rating-scale-based items and play a critical role in determining the quality and rigour of an assessment tool. If the distractors are poorly designed, they can significantly undermine the quality of the test, no matter how well-crafted the items are.

Some common mistakes to avoid include:

1. Having distractors that are obviously wrong and easy to eliminate
2. Making distractors too homogeneous, to the point where the correct answer stands out.
3. Using factually incorrect distractors, rather than plausible alternatives that reflect common misconceptions.
4. In rating scales, failing to ensure a logical progression across response options; a property known as monotonicity in psychometrics.
5. Using distractors that are ambiguous or potentially defensible as correct answers.
6. Designing distractors that fail to attract lower-ability candidates.

So, when purchasing or evaluating an assessment tool, make sure you pay attention to the quality of the distractors, not just the items. Even if distractor analysis indices are not available, a quick review of the response options for the issues listed above can provide valuable insights into the quality of the assessment.