What is your top security concern?

The number one security concern of testing professionals is exam theft and piracy, according to a survey that Caveon conducted at NOCA in 2005. We asked the question: “Which of the following are security concerns for you? (Please check as many as apply).” One hundred participants responded in the following manner:

Concern

Number Responding

Proxy test taking

37

Lax proctoring at test sites

44

Stealing items, pools, or tests

63

Posting of secure items on the Internet

48

Attempts to hack into your item banks

15

Use of your secure items by training programs or coaching schools

41

Leakage of items by item writers, reviewers, or other contributors

45

Given the news article from the Boston Globe, “Job exam piracy rising,” published December 26, 2007, it would be interesting to repeat the above survey.

http://www.boston.com/news/nation/washington/articles/2007/12/26/job_exam_piracy_rising/?page=full

This is a very important article because while data are not provided to support the headline that exam piracy is really on the rise, it strongly illustrates the impact of exam piracy on the testing industry and the fact that current remedies cannot effectively counter many instances of test theft. This is particularly true for information technology certifications.

I have been studying the problem of exam piracy for a long time, and can offer a few insights. First, the asset that must be protected by exam security is the integrity of the examination process and the credibility of the test result, not the item bank or the test form. Second, the correct perspective of the relationship between certifying authority and test thief is a host-parasite relationship. The exam pirates live and draw from the vitality of the certification, devaluing it with their success. Lastly, a year ago we analyzed the data forensics analyses that we had performed for more than 20 certification programs. We determined that three main factors were directly related to exam piracy: (1) the mission and role of the certification, (2) the test administration model, and (3) the security of the test administration channel.

Protecting the integrity of the examination process – Current legal protections against exam piracy involve copyright and trade-secrecy statutes. Unfortunately, these can only be invoked after the integrity of the test is breached. They usually involve protracted investigations followed by even lengthier legal proceedings. In the meantime, the test is compromised and keeping it in service further erodes credibility in the examination process. The DMCA (Digital Millennium Copyright Act) provides some assistance when the stolen content is accessed through on a US-based ISP. But, legal remedies are few. In fact, legal jurisdiction of crimes committed over the Internet is at times very unclear, compounding the problem.

Host-parasite relationship – A certifying authority such as the FSBPT (Federation of State Boards of Physical Therapy) derives its existence from maintaining and administering the exam. An attack on the integrity exam is an attack against its very existence and must be countered. On the other hand, a company such as Microsoft provides certifications in support of its business. The vitality of such a company is derived from product sales and service, not from the certifications. Thus, as long as attacks on the exam do not adversely affect the core business of the company, it may be able to withstand parasitical infestations. In either case, the parasitical exam pirate bears no goodwill toward the certifying authority and has no compunction in destroying it.

Mission and role of certification – Resources within any organization are deployed according to its core mission or function. In the context of exam security this means that operational budgets and legal expenditures are prioritized accordingly. For example, the lawyers for an organization such as FSBPT will be more willing to tackle exam security issues than will lawyers for the typical IT company. This is because the lawyers for IT companies are involved in patent protection, maintaining business contracts, and other core business functions.

Test administration model – Most high-stakes testing programs administer tests according to pre-determined testing events. A new test (which may use previously administered items) is constructed for each event, thus decreasing the chance that stolen test items will be present on the new test. This practice means that it is more difficult for the exam pirate to profit from the testing program. On the other hand, when the same test forms are kept in service for a protracted length of time, the exam pirate has a distinct advantage in stealing and selling the test content.

Security of the test administration channel – The article from the Boston Globe states, “Technology companies in particular have accepted lower levels of security in order to have testing centers in distant corners of the globe.” The lower levels of security involve contracting the test administrations with third-parties who may have never had a background check, who may be operating cheat sites, or who don’t care exactly how they make money. A rogue test site administrator can very easily steal a test by merely recording every testing session (i.e., with a video camera) and then transcribing it. I believe that some these individuals have discovered how to actually pilfer the test content electronically, avoiding the need for transcription.

Hopefully, thinking about the above observations will help you understand why exam piracy is not going to be solved easily. Some testing organizations are being seriously affected by exam piracy. Only time will tell whether they will be able to successfully ward off the pirates, or not.

Dennis Maynes

Chief Scientist, Caveon Test Security

Leave a Reply