The Federation of State Boards of Physical Therapy takes test security seriously. The reputation of the FSBPT and the U.S. states that license physical therapy professionals depends on the ongoing excellence, reliability, defensibility, and validity of the licensure examination. Consequently, when suspicion concerning the unlawful trafficking of test content at a few overseas test prep centers arose, the FSBPT took swift and thorough action. Through its own private investigation efforts, as well as Philippine government surveillance and raids of two Manila test centers in January 2007, FSBPT confirmed that the centers distributed to customers compilations of actual NPTE test questions memorized and shared by prior test takers.

The FSBPT was forced to take legal action against the test prep centers but even while doing so was faced with a perplexing dilemma. The damage had already been done. While the FSBPT had to assume that a number of the examinees who had participated in the Philippine test centers had gained an unfair advantage through pre-knowledge of the test content, there was no way of determining which examinees exactly had gained an advantage. Forcing everyone to retake the exam would unfairly place the burden of proof on legitimate test takers and be very costly in terms of lost income due to delayed licensing. Doing nothing to validate the test scores, on the other hand, would undermine the integrity of the entire testing program. How could the FSBPT protect its own reputation as well as that of prospective physical therapists?

Enter Caveon Test Security. The FSBPT decided to pursue a new approach to validate the test scores by contracting Caveon to perform a data forensic analysis of the test result data. Simply put, test takers who have gained pre-knowledge of test content do not behave like normal test takers. They leave a trail in the form of anomalous answering patterns which Caveon is able to uncover through its sophisticated statistical analyses.

To remove any bias in the investigation Caveon asked the FSBPT for the test results from all candidates and not just candidates that potentially participated in the test preparation course. Caveon used three separate statistical indices to identify aberrant tests. The likelihood that a test would be aberrant on any one of these indices is 1 in ten thousand—the likelihood that any one test would be aberrant on all three is more than 1 in a million. After review of the entire data set Caveon determined that 20 of the tests were statistically inconsistent. All 20 were anomalous on all 3 of the indices Caveon used to identify irregularities. And all 20 of the examinees had access to the implicated test preparation centers.

With the conclusive evidence from Caveon the FSBPT was able to invalidate the compromised test scores. “As with every decision we’ve made in addressing the troubling use of recalled items, the FSBPT Board did not take this action lightly,” stated E. Dargan Ervin, Jr., FSBPT President. “We made the decision only after careful consideration of the issues and in light of the overwhelming statistical data that calls into question the legitimacy of these scores.”

Caveon’s work with the FSBPT did not end with the invalidation of these test scores. Understanding the real risk from new and innovative threats to test security the FSBPT felt it needed to continue working with Caveon, a company they trusted which could provide new and innovative solutions to their test security issues.

“Caveon’s work was invaluable in helping us respond to this very serious and damaging security breach. They were both responsive and helpful in developing and implementing the forensic analysis plan,” said Mark Lane, Vice President at FSBPT.