Written by Steve Addicott, Vice President
August 29, 2014
The current thinking in measurement circles connects test security with test score validity and the integrity of your testing program. Doing so, we immediately focus on whether test takers may have cheated, whether the test questions may have been compromised, and whether we have the correct policies in place. But, one thing is lacking. Have you considered the effect that test design might have on test security? If not, you have neglected an essential aspect of building a secure testing program. Caveon’s offerings now include Caveon Test Development Services (TDS), which include test design and development, so we can help our clients improve test security. Let me illustrate the important role test design plays in test security by presenting and discussing an analogy from contact sports (The analogy resonates with me because I have two active, sport-crazed teenagers at home).
For decades, athletes at all levels, from youth to professionals, have sustained concussions. Only recently has the long-term health impact of repeated concussions been well understood. Many doctors, coaches, athletes, and parents were unaware that cognitive functions could be severely impaired by even slight concussions. To me, this situation is similar to that faced by many testing program managers who have not been aware that the usefulness and integrity of their tests can be severely impaired by seemingly random cheating and piracy.
Over time, medical researchers have learned that the impairment caused by brain injuries in sports was more severe than previously thought. This awareness spurred some changes. Manufacturers produced safer helmets, and players adopted new equipment, training, and techniques to better protect themselves. Despite these changes, the game’s fundamental design didn’t change or evolve in order to increase the protection of players. Similarly, testing program managers have “bolted on” new procedures to better protect the tests. Despite implementing new tactics (i.e., a test security “helmet”) the same testing models and designs have remained in place. The tests have not been redesigned to improve protection of the testing program.
In contact sports, we now know better. Our naivete regarding concussions in sports was forever erased during the last decade by a myriad of incidents involving high-profile athletes with tragic brain injuries from concussions. While there are many examples to cite, the suicide of NFL Hall of Famer Junior Seau was a frightening wake-up call not only for the public, but for the NFL. Similarly, the testing industry has dealt with numerous security breaches reported by many media channels. But in the last few years, the Atlanta Cheating Scandal has demonstrated just how bad things can get when test security threats remain unchecked.
In response to the threats posed by concussions as crystalized by Seau and other incidents—the NFL has implemented new rules and redesigned the game to minimize injuries and risks to the short and long-term health of its players. Helmet-led tackling is now illegal, and stiff fines are dealt to infringing players when they aggressively hit a “defense-less” player. Now, because of these changes, new, safer tackling methods are coached at all levels of the sport, and head injuries appear to be on the decline. The effect of redesigning the sport has been profound. Protection of players has improved. I believe that we can learn from this lesson. By weaving protection into the fabric of the game, sport leaders provide an example of how we can evolve beyond “bolt on” test security.
For test programs, changing the fundamental design of a program, test, and items can have the same profound effect on test security. I believe that we can deeply improve security and reduce threats by instituting innovative designs that centrally change our game. For instance, several of the following design innovations did not exist ten years ago, but are now available to help us:
- Item Cloning and Computer Adaptive test designs minimize item exposure which reduces test piracy and item harvesting;
- Embedded Verification Tests™ and Rapid-Republication of Item Families™ minimize unfair advantages gained by inappropriate access to items before, and collusion during, test administration;
- Internet Based Testing and other secure test-administration technologies negate efforts to tamper with tests and scores, and finally,
- Regular incorporation of Data Forensic analyses provides an effective means of measuring threats and risks so program managers are better prepared to deal with them.
In a recent meeting, a leader of a major medical licensure test program explained how his definition of test result validity has changed. In the past, he said, he considered score validity to involve the administration of a quality exam that is psychometrically sound. That perspective, though, has now evolved to where he will only consider a score to be valid if it stems from the administration of a SECURE, quality exam that is psychometrically sound. In light of his new definition of validity, I find that test designs and innovative test development as exemplified by Caveon TDS services are essential to “building in” and not “bolting on” test security. At Caveon, TDS is the foundation by which we can ensure secure, quality tests and trustworthy test results for our clients.