Trouble in Section K
Elf mistress Heloise entered Elvin’s office (Head of Section K) quickly. “For the eighth week in a row, the reject rate from Section K is three times the rate from the previous twelve months,” she said, handing the weekly quality report to Elvin. She continued, “I was so impressed when your section scored higher on the elf proficiency exam than any other section in the Mechanical Doll Department nine weeks ago that I awarded your elves with assemblage of gears and levers, but this is unacceptable.” Heloise crossed her arms and waited for a reply.
Elvin wrinkled his brow and frowned ruefully. This was unwelcome, but not unexpected, news. He picked up a thick folder and opened it. He leafed through one report after another and muttered, “We have eliminated transportation, storage, tools, assembly, parts, fatigue, and sabotage as explanations. There’s only one conclusion. At least one, and maybe several, of the elves in Section K is incompetent. But how can that be? Is the proficiency exam flawed?”
“Let’s find out,” replied Heloise. And together, they visited the proficiency exam designer. After explaining the problem, the proficiency exam designer shook her head and said, “You need to see the data forensics analyst.” The data forensics analyst listened with deep concentration, scanned page after page of test results, whistled softly, and finally exclaimed, “It looks like elves in Section K have cheated on the elf proficiency exam. Now, how to prove it?” he said mysteriously, and then immersed himself in complex symbols and calculations. Heloise and Elvin excused themselves, but the data forensics analyst didn’t even turn his head as they left. Much later, the proficiency exam designer listened intently while the data forensics analyst described his plan for catching the cheaters in Section K.
Three weeks later, the schedule for the quarterly elf proficiency exam was posted throughout the Mechanical Doll Department. On the day of the test, elf examiners throughout Santa’s workshop reported to a different department than usual to conduct the examination. For example, elf examiners from Remote-Controlled Toys reported to the Games and Puzzles Department. It so happened that an elf examiner from each of the other departments reported to the Mechanical Doll Department. Some administered the elf proficiency exam, and others just watched and waited. All test responses were recorded meticulously. After a long and grueling day, all the elves had been tested.
The data forensics analyst worked all night, making calculations and graphs and charts. At the break of day, Heloise and Elvin knocked at his door. “Enter!” they heard. They stepped into a bizarre scene: scraps of paper were strewn about, charts with bars and circles were plastered on the walls, and a wizened elf was humming in the midst of chaos. “Done!” he shouted. “Oh, it’s you. Well, I have the answer,” he said with absent-minded aplomb.
Then noticing their impatient expressions, he said, “Oh, let me explain.”
“None of the examiners are involved. I know this because there are no patterns of inconsistent answering associated with the examiners. It was important that no examiner give the test to any elf with whom he or she normally associates.
“There were extremely similar test answers between four elves in Section K. It is almost certain that they did not take the tests independently,” The data forensics analyst concluded.
“But, how can that be?” queried Heloise. “They were all watched carefully. There was no way that they could have shared answers or communicated during the test!”
The data forensics analyst minutely explained, “I suspected this might be the case. So, I asked the proficiency exam designer to create two test forms. She very carefully changed a few of the questions between the first and second test forms, so that the correct answers would be close, but not the same. The master test booklet for the first form was locked away in test booklet storage. The proficiency exam designer kept the master test booklet for the second form with her at all times. Even though the elves in the Mechanical Doll Department were given the second form of the test, our four culprits answered all the changed questions with answers from the first form of the test. There is no doubt in my mind. They broke into test booklet storage and memorized the test answers!”
Elvin brought the four suspected cheaters into Heloise’s office. Each elf vigorously denied any wrongdoing. At that point, the data forensics analyst dimmed the lights. He splayed an infrared beam across the hands of each suspected cheater. All of their hands glowed eerily with a blotchy red hue. Then, using gloves to handle the master test booklet from storage he shined the beam on the pages. They glowed red. He touched the booklet pages against his bare arm. Shining the bean on his arm, it also glowed with a blotchy red hue. Heloise barked, “You are red-handed! Now stand still while I consider your punishment!”
“Tomorrow,” pronounced Heloise. “You will report to the master of the Quality Department for ‘R and R,’ where you will begin the repair and refurbishment of all toys in the Rejected Toy Warehouse. You will work there until all the broken toys are operating perfectly and to the satisfaction of the master of quality.”
“Elvin,” Heloise continued. “Section K can no longer be responsible for assemblage of gears and levers. Your section must repair its damaged reputation from producing so many rejected mechanical dolls. Even though you will not receive replacements for these culprits, your production quota will remain the same.”
Elvin wrinkled his brow and frowned ruefully. This was unwelcome, but not unexpected, news. He remembered another time, when he was an impetuous, lazy elf; and when he had cheated. The punishment seemed harsh, but he had learned his lesson and was glad that the cheaters had been apprehended.
Moral: Just as dishonesty betrays the cheater, it injures all who are around him.
Addendum: The cheating detection and prevention techniques described in this story are among best practices. I have described use of the data forensics methodologies in two actual cases we have analyzed at Caveon: The case of the waylaid answer key and The case of the befuddled answer copier.
The State of Mississippi has put together a very nice power-point presentation on test administration auditing and monitoring: www.mde.k12.ms.us/ACAD/osa/DTC_Test_Security_Fall_07.pps