Unintended Consequences of Computer-Based Testing

If you plan to implement computer-based testing (CBT), you might consider the experience of those who have already done so. There is a temptation with CBT to automate as much as possible. In general, this is a good idea. But, unintended consequences may affect or even disrupt internal processes.

One such process that will undoubtedly be affected is that of score reporting. You will need to decide whether scores should be reported as soon as the test has been completed. Several Information Technology (IT) certification program managers have learned that immediate score reporting can be both good and bad. It is good because the candidates are able to move forward, by retaking the exam, applying for a job because a certification was awarded, or considering the next certification to earn. It can be bad when the validity of the reported score is called into question. When told their scores will be invalidated, test takers are not happy. This causes additional work for certification program managers. Facing score invalidation, test takers may even threaten to file law suits and other legal actions. In fact, some IT certification program managers, who strongly support score invalidation, are taking steps to eliminate immediate score reporting. These managers have rejected provisional score reporting as an option because it creates more problems than it solves. Many of them now favor delayed score reporting or a variation known as “results hold,” which is implemented when statistical security filters detect a potential problem with a test result.

Another process likely to be affected is quality control and key validation. Key validation is critical because a small proportion of items get published with incorrect answer keys, frequently. Several years ago, this happened in the State of Virginia’s item bank for the driver’s license exam. Items were randomly drawn from the item bank. The applicant was required to correctly answer 19 of 20 questions. Unfairly, applicants who received two of the incorrectly keyed questions were guaranteed to fail. Traditionally, programs used the time between test administration and score reporting to analyze the response data and detect these problems. Adjustments could then be made. Time intervals between test administration and score reporting are often compressed when tests are given by computers. As a result, programs have less time for quality control activities. Indeed, some of these tasks have been eliminated upon implementation of CBT. Doing so, poses quality risks that should be considered.

When the time interval between test administration and release of scores is shortened, other activities may also be affected. One such activity is data forensics or statistical analysis of test results in order to detect potential test security violations. Many programs have inserted Caveon’s analyses into this window. Sometimes the amount of time we have been granted to conduct our work is very short because the schedule is very tight. When this happens, we are required to shorten or eliminate expert review of anomalies. Experts do not work in nanosecond time. They don’t run on electricity. But, they have one thing that computers lack: the ability to make well-reasoned judgments in the face of ever-changing security threats. Because this takes time, squeezing the schedule through automation may cause some anomalies to receive only a cursory review when a more in-depth review is warranted.

Other examples could be listed. It is important to realize that your processes will be impacted as you transition from traditional paper-and-pencil testing to computer-based testing. Before making this move, I would suggest that you carefully review and document all of your current processes and consider how essential processes must be retained and implemented within the CBT environment.

Test

Dennis Maynes

Chief Scientist, Caveon Test Security

Leave a Reply