Item Matching Analysis

Recently Caveon compared an actual client test with a "practice test" that was purchased from the Internet. There are 121 items in the client test and 60 items in the practice test.

Using our proprietary Item Matching Analysis tools, the client test was deemed to have been 49% compromised. In fact, every question in the 60 item practice test was found to be a copy of one of the items in the client test!

The most disconcerting finding of this analysis was illuminated when the order of the matching items was compared using a scatter plot. This plot is presented below:

The item pairs that were highly similar or exact matches are shown in the table below.

While the tabled pairs are not especially revealing, the scatter plot shows there is a strong relationship between the two documents. The strength of the relationship indicates that the practice test is most likely a copy of an earlier version of the live exam.

This reasoning is as follows:

1) The question ordering relationship is not due to chance alone,
2) The live exam contains many more questions than the practice test, and
3) The relationship between the orders of the items is consistent with the types of changes that one expects to occur during a versioning change.

In the above case, the client suspects a previous version of the exam file may have been stolen and sold on the Internet.