January 20, 2015
Written by John Fremer, President, Caveon Consulting Services
Test program managers want their examinations and related test taker and program information to be secure. The fairness and validity of the results for employment, admissions, certification and other high stakes testing purposes make the need even more compelling. That said, security is not the only issue that managers face. Test items must be written or acquired, candidates must be informed about how to take tests, test administrations provided for, etc. At every step, particularly during test administrations, there are security risks, not just from individual test takers, but often from illicit businesses profiting by undermining the integrity of your program. This is particularly true if you test internationally and even more so if your program extends into high test security risk countries such as China, India, Nigeria, South Korea, and/or The Philippines. Please be aware, moreover, that this is not a comprehensive list. These are countries, though, that present very substantial challenges to test security.
So the question that Managers face is what level of security can they afford? Saying that you want your program to be “totally secure” while noble and understandable is simply not an attainable situation within the restraints of virtually all programs, whether they be public or private, large or small, paper and pencil or using some form of electronic delivery. In this blog, I focus on what I view as the most troublesome security vulnerabilities, particularly those related to test administration which is the area where the greatest number of problems occur. You can come close to zero cheating and item theft if your program has these characteristics:
- Brand new items for each candidate
- A very disciplined approach to deciding whether, where, and how to test
- Carefully trained and monitored proctors
- Absence of any incentive for proctors for test takers to do well
- Using proctors who can unerringly determine that the test takers are who they represent themselves as being
- Maintenance of high test administration standards across every testing site
In deciding how secure a program the manager can afford, the level of stakes associated with the program should be a factor and the manager needs to evaluate the risks associated with testing in places where there is a very high probability of test compromise. In some countries, theft of your items and pools is virtually guaranteed, without heroic and costly test administration arrangements which carry substantial costs. (In one instance, an entire item pool was stolen before a program administered a single exam!)
So other than wringing your hands, joining a test program manager support group, or finding some other line of work, what is a manager to do? First of all, acknowledge and make sure that your Board and/or other managers know that cheating and item piracy is an ongoing “fact of life” not some rare event like being struck by lightning. In planning and budgeting, this reality must be considered. Next do your best to have test security be an issue when major program decisions are being made. Yes, there may have been a credible study showing how there is a large potential candidate pool in “Country X,” but it may well be that any pool used in Country X will have to be considered “publicly disseminated” very quickly. It is not “good business” to bring in more candidate testing dollars if the loss of test items and pools resulting from this change has financial consequences that more than cancel out any apparent gains. Another consideration is the type of testing administration you provide. For some countries using the most secure level of service offered by vendors has to be your choice and you may well need to adopt additional security practices not employed elsewhere in the world. Don’t get stuck in what is sometimes called a “foolish consistency.” You must do more in some countries to achieve the basic level of security that is needed to have fair and valid test results. Unless you are willing to spend more than has been your pattern in other countries, you will not be able to maintain industry and professional test security standards and your test scores will not have the same quality.
In this blog, I have sounded an alarm that my colleagues in the field and industry who have been testing internationally for many years did not need. They have learned, often bitter lessons about what some of my Caveon colleagues call “Testing in Hell.” If this is new information to any test program manager new to international testing or considering expansion, I urge you to pay close attention, the future of your program may depend on it.