Written by Dennis Maynes, Chief Scientist, Caveon Test Security
Oh! what a tangled web we weave
When first we practice to deceive!
Sir Walter Scott from Marmion
My friend once advised me, “Don’t lie, because you can’t remember what you said.” Cheating is a form of lying. When you cheat on exams, you misrepresent yourself and your ability. When you share your test result obtained through cheating, you have practiced deception. The tried and true methods for detecting cheaters and liars are to catch them contradicting themselves or to find them in improbable and inexplicable situations.
These same basic principles apply to the statistical analysis of test result data (aka data forensics). We compare the test taker’s pattern of responses with known patterns of responses from normal test taking. For example, it is inconsistent to miss the easy questions and answer the hard questions correctly. It is inexplicable to pass the test more quickly than the questions can be read. It is downright suspicious to answer revised questions with the old (and no longer correct) answers. It defies reason when test responses between two test takers agree more closely than any other pair of tests from among thousands or even millions of tests which have been taken independently.
Recently, I have been working on verification methods which are based on the same ideas. Administer the regular test to the test taker, and then give a few more questions (known as verification questions) that the test taker could not have seen previously as they have never been administered. Compare performance between the two sets of questions. If the performance is inconsistent, then the test taker has been detected in an inexplicable situation. At that point, an explanation is needed.
Test takers are not the only individuals who may perpetrate some sort of test fraud. Sometimes, test site administrators or trainers are involved. For example, in November 2007 a trainer at the Denver International Airport was caught on hidden camera disclosing answers to questions while the students were taking the test. The training course? De-ice aircraft. Again, the same general principles may be used for detecting group-based test security threats. In this situation, we need to look for patterns or anomalies at the group level that are inconsistent with normal test taking. That is the general approach that was used by the State of Georgia in 2010 to expose the Atlanta Public Schools test-tampering scandal. There were way too many wrong-to-right answer changes on the answer sheets to represent “normal” test taking. Students don’t erase that often and they don’t almost always choose the correct answer when they make a change. So, an explanation was needed.
The same general approach is used for determining when test questions may have been compromised. Does the statistical pattern of the test questions conform to known patterns of normal test taking? Has something changed with the way the test questions perform, such as becoming much easier? If so, the disparity between normal and observed (or statistically speaking “expected” and “observed”) provides evidence that something may be amiss.
I thought it would be interesting to share a few simple principles with you that are the foundation for statistically detecting potential test fraud. By the way, you are welcome to join the Caveon team at the second annual Conference of Statistical Detection of Potential Test Fraud to be held in Madison, Wisconsin, October 17-19, 2013: http://cete.ku.edu/2nd-annual-statistical-detection-potential-test-fraud-conference At the conference, we will be presenting specific techniques for detecting collusion and tampering on tests.