Are you ready for the future or stuck in the past?
Written by Saundra Foderick, Test Development Manager
June 27, 2016
The most damaging phrase in the language is “We’ve always done it this way!”
— Rear Admiral Grace Murray Hopper
It seems as if there is a new security threat almost every day: time zone scams, test dump sites, hidden cameras, custom printed water bottle labels, and thousands of other cheating methods can render test scores—even for non-cheaters—suspect and useless. Worse, it is hard to hold the respect of test takers, certifying bodies, employers, and the general public if we cannot stand behind the integrity of our exams.
In response, we make our exams more secure, use clones, insert watermarks, add Trojan horses, and hire experts to scan the web and analyze our test results. Each of these adds to the security of an exam. Still, we can go a step farther into the future by using DOMC (Discrete Option Multiple Choice) test items. DOMC items are a practical way to increase security, fairness, and careful measurement, while shortening testing time. Is your program stuck in the past or is it looking toward the future?
DOMC items significantly reduce the exposure of item content when compared to traditional multiple choice—perhaps as much as 50% for some items—while maintaining the ability, at a high level, to measure a person’s skills. Pirates cannot easily steal, memorize, or harvest a test form, even with a digital camera, since each examinee sees only a small portion of the entire exam.
The results? Test buyers will not be able to cheat their way to success, brain dump sites will have to invest far more time and money for even a partial form, forms sold will prove to be useless, and the exam will have a longer shelf life. Better yet, a greater percentage of your candidates will be taking the tests honestly—their scores will be accurate representations of their experience and knowledge. Examinees, employers, and relevant constituencies can trust those scores.
If test takers will not be able to cheat effectively using such pre-knowledge, the dishonest ones may resort to other less-effective more risky cheating efforts. The good news is that DOMC items can provide the data to catch the cheater. Every test taker sees a different set of options—a test-fingerprint of sorts. If those items are shared on the Internet, that “fingerprint” can be traced back to the items and options each test taker saw and can make a definitive match. The thief is caught—almost in the act!
When this partial theft is caught quickly, and sanctions are publicized and swift, honest examinees are given confidence that the test is fair and dishonest examinees are deterred.
Fairness is a wonderful standard in the field of high-stakes testing. It means that everyone has an equal chance of doing well on the test, given equal relevant training and experience. Unfortunately, many test takers are testwise, knowing the tricks that can unfairly increase their scores while penalizing those without those skills. The problem is made worse when the language of the test is not the primary language of the candidate or the candidate is not a strong reader.
DOMC takes away the test taking skills surrounding the comparison of options and the ability to winnow down the answer choices to 1 or 2. This makes the item slightly more difficult for the good test taker, but levels the playing field for all candidates.
In addition, DOMC is a simpler item: It is more straightforward and clear. This can mitigate the misunderstandings that can come from differing cultures, languages, backgrounds, ages, and more. It makes knowledge of the subject matter being tested the most important factor in answering a question well. Instead of testing reading ability, you are testing the candidate’s knowledge and skill.
Simply put, DOMC exams provide more valid scores and will be more reliable than if traditional multiple-choice items were used. Validity and Reliability are the two main pillars of the testing standards. How do DOMC items raise validity and reliability? A DOMC item provides about three times as much information about a test taker than does the traditional MC item because the test takers simply make more responses. They answer YES or NO to each option they see, instead of the single choice in most traditional MC items. Each of those responses is used to evaluate the candidate, making sure they know (or do not know) the content.
The extra data generated by the DOMC item is also used to evaluate the items at a later time. An analysis of options or items is easier when more data are available. Bad items (or options) can be tossed and replaced quickly. While traditional test takers grumble about unfair or wrong test questions, DOMC examinees will find that their exams seem more relevant all the time. Thus, your program managers will have fewer complaints about items and your test will be considered more reliable.
DOMC is Practical
Counter-intuitively, research has established clearly that DOMC-based exams take less time to answer than the same exam with traditional MC items. A savings of at least 10% is expected, but some research has indicated as much as 40% savings. For test candidates this means less time testing and less stress. For your program manager, this means less time proctoring and lower fees to testing vendors. Where a traditional 60-item exam might need 90 minutes to 2 hours, with DOMC you may be able to cover the same number of items in an hour or less.
Finally, not only is DOMC practical in a traditional sense, but it is forward looking in ways that other test forms are not, including use on mobile devices. Since DOMC requires less reading, less screen real estate is needed, fitting easily on a tablet or smartphone. As communication is moving to mobile devices, programs will be expected to allow test administration on tablets or smartphones. With DOMC, you can meet these expectations.
As you might expect, there will be objections, especially from those individuals with good test taking skills since DOMC takes away their advantage. Too, some may object to learning a new way to take a test; for those, practice exams can answer their concerns.
Those who have designed tests for years may object due to tradition: our practices have been in place for years and best practices have been identified. To a certain degree they are right: we have years of research to show some of what works and what does not. Still, we should use that research and more recent scholarly works to move into the future. The saying is, if you always do things the same way, you will always get what you always got. And, what we got these days is continual security threats and continual breaches, costing time, effort, and reputation. Where possible, we need to blend the old with the new in order to strive for something more secure, fairer, and trustworthy. Is your program ready for the future? It could be with DOMC.