A senior from Manatee High School passed the FCAT (Florida Comprehensive Assessment Test) in ten minutes by using a “secret pattern” after flunking the test three times. His score was invalidated. Apparently the test score was not invalidated because he used a pattern. Carla Frazier told the news, “FCAT rules do not prohibit students completing the test using any patterns, nor does the test have a minimum time requirement.”
We don’t know why the principal invalidated the score. We don’t know what “secret pattern” was used by the student. But, I have an idea what it might have been: “a-n-s-w-e-r-k-e-y.” Ok, I admit to being a cynic and a skeptic at times. This is one of those times.
Consider the facts, and then decide for yourself if you believe the student’s story.
- Test publishers are very careful to make answer keys as unpredictable as possible. They are well aware of the guesser’s adage, “If you don’t know, choose ‘C’.”
- Item writers and item reviewers are careful in writing distractors and answer choices to prevent guessers from gaming the test and gaining an advantage. They know that guessers will attempt to deduce the correct answer by analyzing the answer choice lengths and details.
- Having analyzed a lot of high school exit exam data, I know that pass rates go down with every make up test. Students who fail three times are very lazy, easily confused or just not proficient. Passing the test in ten minutes is not consistent with any of these.
- Cheaters are often very creative liars and they prey on our gullibility. The news reporter was gullible in writing the story and, for some reason, expects us to be equally gullible.
There are a lot of ways to detect cheating. In this particular case we might have seen any of the following:
- An extremely high score after having flunked three times previously would be a clear warning sign to the principal.
- The FCAT, according to the district FCAT coordinator, often contains pilot questions. If the student did very well on all the questions, except the pilot questions, and the answers to those questions matched the answer key form a different form of the test, then the principal would definitely have a “smoking gun.”
- Sometimes the answer sheet can be modified after the fact. With the right inducement, an insider may be persuaded to change the answers. Erasure analysis would detect this kind of tampering. Perhaps the principal was suspicious and saw a lot of erasures on the answer sheet.
- It is often the case that the cheaters boast of their exploits and in this case the principal may have gotten wind of the boasting.
Being a student of statistics, I imagine that the student could have finally gotten lucky and passed the test. Distribution theory states that the maximum observed value in a distribution has a much higher mean than the distribution from which the value was drawn. In this case, we have repeated scores on the FCAT for the student. Just by chance alone, if the student’s expected score is reasonably close to passing, after repeatedly taking the test a passing score will be observed eventually.
But, suppose that in my skepticism I am correct. Suppose the student did have the answer key. How would the forensics analyst detect that an answer key had been stolen and used? I have seen three answer-key arbitrage techniques used for exam security purposes, and which could be used in similar situations.
- The FCAT coordinator disclosed that pilot questions are often used on the exam. Scoring the pilot questions with alternate keys could provide probability evidence that an answer key was in play.
- I know of a situation where items were intentionally miskeyed and left unscored with the goal of determining whether the answer key had been stolen and used.
- In another situation, the exam contained a few poorly written questions where the provided answer was ambiguous (This often happens on exams). These questions were exploited in a similar manner to compute probability evidence that an answer key was stolen and used.
The test publisher has many tools and techniques that can be used to trap the unsuspecting cheater. Answer-key arbitrage is one of those.