Written by: Steve Addicott, Vice President of Client Services, Caveon Test Security
Are you considering instituting a cheating detection (aka “Data Forensics”) program to help protect your tests? Great idea! Before doing so, we recommend that you heed the advice of renowned Test Security expert Greg Cizek (he wrote the book, literally, on “Cheating on Tests”). He advised, “I would first strongly caution everyone NOT to do any analyses at all until first developing and adopting policies and procedures about what to DO with the results of any analyses.”
At Caveon, we wholeheartedly agree.
Over the years, we have worked with many testing programs. Their use of cheating detection analyses varies considerably. In each case, use is driven by the policies and procedures they’ve adopted and implemented.
Before describing these uses, it will be helpful for me to describe the Caveon Data Forensics approach.
In general, our analyses may be classified into two primary categories: 1) Broad Monitoring Scans; and, 2) Deep Investigative Analyses. These types of analyses can be compared to the medical uses of Magnetic Resonance Imaging (MRI) and/or CT Scan machines.
A Broad Monitoring Scan is like a general body scan or a health checkup that I might choose to undergo. I’m not aware of any particular malady, but in order to detect health risks early I undergo the procedure so that an expert may gauge the level of my health and whether previously undetected health threats exist. That’s the attitude of most our clients, who monitor for test security threats so they can detect them early. They desire to measure risks to trustworthy test results so they can better manage against them.
A Deep Investigative Analysis is more like the MRI or X-Ray machine that is used when a medical professional suspects a particular illness or ailment. The machine is focused in a specific area and manner to help identify whether a condition exists. The goal is to determine the scope of a problem’s impact upon the individual’s health so a trained professional may prescribe an appropriate remedy. It’s the same thing with our Data Forensics–Our team sifts through the data focusing on specific, worrisome situations, locations, and/or test takers.
As a program considers its policies and procedures, it should carefully consider the intended focus of the data forensics analyses. Caveon identifies statistical irregularities involving individual test takers, and groups of test takers. Just as our clients vary, the groups that may be cause of a concern also vary. K12 education clients are concerned with what is happening in classrooms and schools; many international test programs want to flag test centers with concentrations of unusual results; and others like to spot trends in groups with similar “demographics" —that is, test takers that studied in the same school, attended the training program, work for the same company, etc.
Having identified the appropriate targets for analyses, a program needs to consider the sanctions it may want to levy if test improprieties are identified. We’ve seen a multitude of sanctions successfully utilized, including:
- Formal Warnings—In many cases, these also require some sort of explanation from flagged test takers.
- Score Invalidations—With score invalidations, we recommend our clients focus on the test results, not the behavior of the test taker. We don’t want to call anyone a “cheater”, we are simply indicating the test result is “indeterminate” and require a retest. We recommend that clients have an appeal process in place.
- Revocation and/or Suspensions—If a certification is received by the test taker before an analysis is conducted, some clients will revoke or suspend it. Again, we advocate that an appeal process be implemented in conjunction with a revocation policy.
- Formal Follow Up “Interviews” —Usually involving a group (i.e. a school or test center), a program may conduct formal interviews of test takers or staff to learn more about unusual test instances.
- Security Investigations —In more extreme cases, a program may conduct or have conducted a thorough investigation. We recommend that investigation protocols be implemented with and supported by a Data Forensics programs.
- Recommendations to Board or other oversight committee —Some situations require the involvement of a program Board in order to levy other types of sanctions.
At Caveon, we’re committed to helping our clients administer tests securely, fairly and with integrity so that the test results can be trusted and decisions regarding test takers can be made with confidence. From experience, we know that a sound, well-planned Data Forensics program is a powerful tool in achieving these goals.