“It’s the Test Score, Stupid!”

The 1992 presidential campaign slogan of William Jefferson Clinton was “It’s the Economy Stupid!” This slogan captured the essence of Clinton’s message. We, test security professionals, can profit from this. There is one reason and one reason only why tests are administered. Tests are administered in order to measure what the test taker knows or is able to do. But, we seem to forget this when test security has been violated. Instead of worrying whether the test score was valid, we begin to worry about who was responsible for the test security violation. This should not be. Let me illustrate with a few examples.

On November 13, 2013, the headline from the Seattle Times proclaimed: “Police academy recruits broke rule, but didn’t cheat on exams, probe finds.” Investigators found that test security was violated when the instructors shared the actual exam questions with the recruits. The recruits did not know they had been given actual exam questions. They did not intentionally seek to gain an unfair advantage. However, no one knows whether the test scores measured anything beyond the recruits’ ability to memorize test questions. Because of this, the validity of the test scores was questionable. In my opinion, the appropriate action would have been to invalidate the test scores and NOT presume the recruits cheated. Instead, the test scores were upheld even though the validity of the scores was doubtful.

When test security violations are suspected, investigators appear to focus on determining who was responsible for the test security breach and NOT whether the test scores were affected. This strategy can backfire. For example, I was involved in a case where educators were accused of erasing and changing answers in the students’ test booklets. The data were very egregious. There was no doubt that test tampering had occurred. However, test scores were NOT invalidated. As a result, the attorney for the educators was able to argue that no harm had been done. And, without the school district being able to demonstrate which educator was responsible, the hearing examiner agreed.

In a similar situation, the Florida Department of Education’s inspector general’s office investigated four schools which had high numbers of wrong-to-right erasures on the 2011 FCAT administration. After one year, the investigation failed to turn up causes for extremely high numbers of erasures (Tampa Bay Times, August 17, 2012, http://www.tampabay.com/news/education/testing/causes-of-fcat-erasures-remain-undetermined/1246528). The report noted that multiple individuals had custody of a key to the secured room where the test materials were stored. Reading between the lines, we may infer that the Department of Education had cause to believe that tampering had occurred but could not assign culpability. As a result, the test scores were allowed to stand. The tragedy of the situation is that education officials could not ascertain whether the students’ performance was measured properly or not. As a result, students who may have needed assistance did not receive it.

Speaking at the annual conference of the Association of Test Publishers in 2012, I remarked that we should try the test score, not the test taker. In other words, we should NOT let concerns about test taker behavior prevent us from invalidating test scores which have been impugned by test security violations. If test security problems cause us to doubt the trustworthiness of test scores, those scores should NOT be upheld, even though culpability may never be determined.

 

Dennis Maynes

Chief Scientist, Caveon Test Security

Leave a Reply