Caveon Case Study - FSBPT
(View PDF)

Caveon Data Forensics™ helps the Federation of State Boards of Physical Therapy invalidate ill-gotten test scores.

Overview

In its effort to protect the public by providing service and leadership that promotes safe and competent physical therapy practice the Federation of State Boards of Physical Therapy (FSBPT) has developed the National Physical Therapy Examinations (NPTE) for Physical Therapists and Physical Therapist Assistants.

Increased public confidence in the FSBPT has resulted in a growing number of employers who have come to rely on the NPTE exam to determine the competence of physical therapists and hire licensed professionals. An unfortunate but unavoidable consequence of this growth is an increased incentive to cheat on the exam.

The Challenge

The Federation of State Boards of Physical Therapy takes test security seriously. The reputation of the FSBPT and of those licensed under its authority depends on the ongoing excellence, reliability, defensibility, and validity of its examinations. Consequently, when suspicion concerning the unlawful trafficking of test content at a few overseas test prep centers arose, the FSBPT took swift and thorough action. Through its own private investigation efforts, as well as Philippine government surveillance and raids of two Manila test centers in January 2007, FSBPT confirmed that the centers distributed to customers compilations of actual NPTE test questions memorized and shared by prior test takers.

The FSBPT was forced to take legal action against the test prep centers but even while doing so was faced with a perplexing dilemma. The damage had already been done. While the FSBPT had to assume that a number of the examinees at the Philippine test centers had gained an unfair advantage through pre-knowledge of the test content there was no way of determining which examinees exactly had gained an advantage. Forcing everyone to retake the exam would unfairly place the burden of proof on legitimate test takers and be very costly in terms of lost income due to delayed licensures. Doing nothing to validate the test scores, on the other hand, would undermine the integrity of the entire testing program. How could the FSBPT protect its own reputation as well as that of paying customers without costly retakes?

The Solution

Enter Caveon Test Security. The FSBPT decided to pursue a new approach to validate the test scores by contracting Caveon to perform a data forensic analysis of the test result data. Simply put, test takers who have gained pre-knowledge of test content do not behave like normal test takers. They leave a trail in the form of anomalous answering patterns which Caveon is able to uncover through its sophisticated statistical analyses.

To remove any bias in the investigation Caveon asked the FSBPT for the test results from all candidates and not just candidates that potentially participated in the test preparation course. Caveon used three separate statistical indices to identify aberrant tests. The likelihood that a test would be aberrant on any one of these indices is 1 in ten thousand—the likelihood that any one test would be aberrant on all three is more than 1 in a million. After review of the entire data set Caveon determined that 20 of the test were statistically inconsistent. All 20 were anomalous on all 3 of the indices Caveon used to identify irregularities. And coincidentally enough, all 20 of the examinees were “Philippine educated”.

The Results

With the conclusive evidence from Caveon the FSBPT was able to confidently invalidate the compromised test scores. “As with every decision we’ve made in addressing the troubling use of recalled items, the FSBPT Board did not take this action lightly,” stated E. Dargan Ervin, Jr., FSBPT President. “We made the decision only after careful consideration of the issues and in light of the overwhelming statistical data that calls into question the legitimacy of these scores.”

Caveon’s work with the FSBPT did not end with the invalidation of these test scores. Understanding the real risk from new and innovative threats to test security the FSBPT felt it needed to continue working with Caveon, a company they trusted which could provide new and innovative solutions to their test security issues.

“Caveon’s work was invaluable in helping us respond to this very serious and damaging security breach. They were both responsive and helpful in developing and implementing the forensic analysis plan,” said Mark Lane, Vice President of Professional Standards and Assessment at FSBPT.

Send this page to a friend.