Overexposure: A Problem for Certification Tests
By David Foster
Certification Magazine, 8/2003

The problem comes when the exposure of test questions goes beyond reasonable limits, leading to general familiarity. It also means that the content of the items, including answers, is shared freely from candidate to candidate. When that happens, the question becomes worthless. It is no longer able to do its job, which is to help distinguish the competent candidates from the incompetent.

How do we know if a question has been exposed too much in too many tests or exposed improperly? You can tell that a ball is slowly losing air because it doesn’t bounce as high. The “bounce” of a test question can be seen in its statistics.

One statistic in particular, the point-biserial correlation, measures the relationship across all test-takers between answering the question correctly and total test score. That is, it is expected that those who score higher on the test will answer a particular question correctly, and those who score lower on the test will answer the question incorrectly. It is a simple statistical matter to correlate total test score with whether or not the item was answered correctly. A high correlation, closer to 1.0, means the question is performing well; closer to zero means it is not doing well.

When a good question is freely given out, as is the case when it appears at brain-dump sites, it loses its ability to discriminate high performers from low performers, simply because everyone now knows the question and how to answer it, regardless of overall knowledge or ability.

So, if a question can be monitored often and its point-biserial correlation calculated, it is possible to detect when it has been exposed improperly. When that occurs, it is time to replace the question with a more effective one.

There are several important item exposure factors that determine how long it takes before a question needs to be replaced. Here are a few:

• The number of times it is presented in tests. Obviously, the more test-takers actually see and respond to a question, the more likely it is that it will be shared with others.
• Type of item. Simple multiple-choice items that measure a person’s knowledge of facts are easier to memorize and share with others. By contrast, those that require the use of a simulation or an actual software product are more immune to exposure effects.
• The ethics of the candidate. Even knowing it’s wrong, candidates sometimes feel obligated to share the content of a test they have just taken with friends or colleagues. And worse, there are individuals and corporations that profit from gathering and selling test questions.
• The quality of security measures in place to protect it. Having too few questions on a test, or the lack of strict monitoring during the test, will allow easier and more effective strategies for stealing test questions.
• Importance of the test. Certification tests, resulting in high-stakes decisions, provide the highest motivation to remember and disclose questions.

So why should you care about item exposure issues?

First of all, increased security means that test prices will likely rise. One only has to look at the airline industry to see the effects of increased security efforts. The same is expected in certification testing. Creating more test questions to replace existing ones can be expensive. Second, there will be increased security steps to take a test. Biometrics will be increasingly used to verify the identity of the test-taker. Third, there will be more prequalifications to take a test. For example, a person might have to have two years of experience before being allowed to take the test. Today’s relatively easy road to certification in IT will become bumpier. And, as with air travel, we accept the inconvenience and cost in order to enjoy the advantages.

Now, none of these things may happen, but I see a growing problem with the casual approach to the security of tests, both by candidates and certification programs. Something will need to be done soon to shore up the integrity of exams and questions, and the value of IT certifications.