Test Piracy: The Darker Side of Certification
By David Foster
Certification Magazine, 1/2003

It’s no surprise that one definition of piracy in the dictionary is “an act of robbery on the high seas.” But a second definition states, “the unauthorized use of another’s production, invention or conception, esp. in infringement of a copyright.” Except for the “high seas” part, both definitions of piracy describe exactly what has been going on in the darker places of IT certification. Stealing questions, changing test results, taking tests for someone else and unauthorized use of materials during testing are some of the daring, illegal and unethical actions becoming more commonplace as individuals attempt to achieve unearned certifications. The problem has become epidemic, and serious efforts are underway throughout the industry to combat it.

But here’s the bright side: Today’s piracy efforts for the most part are juvenile, occur infrequently, remain unorganized and have occurred in an environment of trust and developing technology. With industry-wide support and organization, along with new tools and industry commitment to protecting the value of certification, these piracy efforts can be countered.

Scope of the Problem: Larger than IT Certification

In August 2002, the Educational Testing Service (ETS) temporarily suspended testing of the Graduate Record Exam (GRE) in China, Korea and Taiwan because of suspicious testing patterns in these regions, as well as Web sites with questions and answers from live GRE exams. Paper-and-pencil testing was re-instituted in these countries. More recently (October 2002), the National Board of Podiatric Medical Examiners canceled the scores of 272 students from four schools because there was evidence that the students had the test content before the scheduled exam date.

Recently in Massachusetts, 19 teachers, administrators and students broke the rules for administration of a statewide assessment known as the Massachusetts Comprehensive Assessment System (MCAS). Teachers sent test questions to colleagues, returned test booklets and asked students to revise them and gave some pupils two days instead of one to complete the test essays.

In IT, results of a recent survey of 629 respondents indicated that 86 percent of certification candidates felt that it was acceptable to use “brain dumps,” Web sites that contain illegally obtained questions from live exams, to prepare for the exams. Only 9 percent felt that such an activity was cheating.

A few years ago, Nancy Cole, president of ETS, described the United States as a “nation of cheaters” and cited evidence to support her claim that cheating has become commonplace, with up to 98 percent of college students admitting to cheating in high school (Miami Herald, Dec. 15, 1998). Other researchers, Linda Trevino, a professor at Penn State, and Donald McCabe from Rutgers reported that cheating on tests for college students increased from 25 percent to 50 percent between 1963 and 1993 (reported in New York Times, May 10, 2002).

It is clear that when the stakes are high, such as those for certification/licensure or college admission exams, the motivation to cheat or commit other types of testing fraud exists. Individuals will try to get a higher score or a passing decision without having the sufficient skill or knowledge. While there are many honest individuals, it must be recognized that others will try to gain from the lack of security measures taken to protect exams and the programs that rely on them.

Computerized Testing: Good News and Bad

There is no doubt that the switch from paper-and-pencil tests to computerized tests has improved the security of the exams. In fact, the major security problem associated with paper-and-pencil testing, that of students copying the answers from students sitting near them, has been completely wiped out by the computer’s ability to present questions in a random order. Here are some other security advantages:

Complete and equivalent test forms are randomly selected. Test-takers can’t anticipate which set of questions they will see.

Computerized adaptive testing creates individually tailored exams, reducing the exposure level of questions and making it difficult for less competent individuals to succeed by cheating.

Encryption and password protection provide secure transmission of tests and test results.

Test development that is entirely computerized prevents the copying and distribution of question pools and tests.

Now the bad news. One of the major advantages of computerized testing to certification candidates is the ability to give the test any time at convenient locations. Some IT certification tests have been available for more than two years. This convenience brings with it a huge security problem: It lets earlier test-takers share questions with later test-takers. Paper-and-pencil tests do not have this problem, because they are usually given only once a year at set locations. While the best solution to this problem is to increase the size of the item pool and create more test forms, this takes time, uses valuable resources and is expensive.

Types of Security Problems

What exactly are the types of security problems occurring in IT certification? I have categorized them by those occurring before, during and after testing. Here are two or three examples of each. The examples aren’t all from IT certification tests, but illustrate what has occurred or what is possible:

Before Testing

Arranging to have someone take the test for you. Last year in the United Kingdom, an employee of a testing center offered a passing score to anyone who would pay him his asking fee, about $500. He would have the candidate meet him at the testing center where he would perform the required security measures. Then once the test began, he would answer the questions for the candidate.

Obtaining the questions prior to taking the test. One of the more popular forms of testing fraud occurs when a certification candidate obtains alleged actual test questions prior to taking the test. This is done in several ways, but the most popular seems to be visiting brain-dump sites. Brain dumps are Web sites where others who have already taken a particular test have supposedly memorized the questions and placed them on the site. Sometimes the access to the site is free, but often the candidate pays a fee to receive access to the questions.

During Testing

Using unauthorized materials. IT certification candidates have been caught bringing a wide variety of “aids” with them into the actual room where tests are delivered. These include copies of training and technical manuals, PDAs, digital cameras, tape recorders, cheat sheets (papers with information on them that would help to answer questions) and blank paper for copying questions and options. These and other materials and techniques are, of course, forbidden and should have been left outside the testing room.

Receiving help during the exam. Because many testing centers exist within the walls of training centers, there exists strong motivation to help the center’s students pass the test. On occasion, instructors or other employees of the training centers have “helped” the examinee answer questions. While this wasn’t necessarily done for a fee and may even have been spontaneous, it is still an example of fraud.

Leaving the testing room for the purpose of getting help. During lengthy tests, it is possible to obtain permission to leave the testing room to get a drink of water or to go to the restroom. Some test-takers have taken advantage of this privilege and have used it as an opportunity to consult unauthorized materials or ask someone for help.

After Testing

Memorizing questions with the intent of sharing them later with others. A few years ago, a test-preparation company hired individuals to take the GRE in its computerized form, memorize the questions and return and report. The questions were then used to create more accurate and realistic practice materials. In IT certification, the same practice occurs, particularly for more popular exams.

Changing the test results after the test has been taken and failed. Attempts have been made to hack into the testing center systems to access the test results and make changes to the data, raising the score and changing the fail decision to a pass decision.

I could provide examples of other types of fraud and more examples of the types I listed; however, it is clear that testing fraud is not limited to a single type of behavior. The creativity of individuals desperate to pass a test (or make money off the desperation of others) is alive and well.

Costs to You and Others

A new IT certification test: $35,000. A new test question: $350. Confidence in a test result: priceless.

A secure test that accurately measures the competence of each test-taker from the time it is published to the time it is retired is a very important tool to you, to the certifying body and to the industry. If it serves its purpose, it provides unshakeable confidence in the holder of the certification. The certifying body can confidently stand behind the individual, and those hiring the certificant can expect a level of competence equal to the job needed.

Once the test is compromised, the confidence in the test is shaken. This triggers the decision to create more items, more forms of the test and perhaps an entirely new test, bringing to bear at least the costs given previously. And let’s be clear on this point: Those costs are eventually going to be taken out of the test-taker’s pocket in the form of higher fees. In fact, the existing price of a test today is higher because of the costs of managing security. With the increase in security issues, that cost will rise even more before all is said and done.

Of course, the rising costs to a program are small compared to the decreasing value of the certification. Each instance of violation reduces that value in a very real sense for the particular certification, as well as the industry as a whole.

So, What’s Going to Happen Now?

There are specific actions being taken by organizations as well as industry-wide efforts to combat the fraud that is occurring. There is no doubt that efforts will be doubled, money will be spent, and legal action will be taken. Because the issues are so damaging to the industry, organizations will join to share expense and information, and to support one another’s efforts.

An industry security group has already been formed, the Information Technology Certification Security Council (ITCSC). The ITCSC, supported by the Association of Test Publishers and CompTIA and comprised of active and influential IT certification programs, has as its mission, “To promote and protect the integrity and value of IT certifications for test-takers, employers and the industry through enhanced security standards and public awareness.” The ITCSC will be influential in producing guidelines and standards of test security, developing industry policies, promoting collaboration within the IT industry and with national and international organizations. More information about the ITCSC can be obtained from its Web site at www.certsecurity.org.

Security efforts in IT certification will focus on detection and prevention. Obviously, detection refers to detecting or discovering an instance of testing fraud, either after it has already occurred or while it is in progress. This can occur at any number of levels: making sure a test-taker is authentic, validating test results and data transmissions, etc. Prevention efforts will make sure that attempts at fraud will be unsuccessful, eliminating opportunities and reducing further motivation to cheat. I believe that both detection (followed by action) and prevention activities will be ultimately very successful in reducing instances of fraud to relatively harmless levels. Let’s take a closer look at what these efforts might look like.

Appeal to Candidates

It’s a fact that many candidates cheat. It is to these people that anti-piracy messages are targeted. Efforts have been made to describe the reasons why cheating on certification exams is a bad idea, that it raises costs and devalues the certification. The premise is that certification candidates, being adults, will be persuaded by the reasoning and will cease doing things to harm the program and themselves. Hopefully they will also help to monitor and report the activities of those who prey upon the industry for personal gain. Visible PR campaigns will help spread the anti-piracy message, and incentive programs will encourage involvement and improvement.

Reduced Exposure to Questions

Of course, with a single set of items, a certification test is at great risk. But when the pool of items is large enough to support many equivalent forms, it is more difficult to memorize all of the questions and pass on any practically useful information to others. Most IT certification exams use two to three forms. This can be increased if necessary.

Computerized adaptive testing is a measurement technology currently in use by several IT certification programs. The test presents questions based on how the person has answered previous questions, essentially tailoring a unique test for each candidate. This technology can provide a valid and reliable certification decision using 40 percent to 70 percent fewer items. Obviously, tests built in this way are very difficult to exploit. Items memorized are quite useless, since it is unlikely they will be seen on subsequent exams by other test-takers.

Also, it is possible to stop tests early for good reasons. For example, if a person has answered enough questions incorrectly that he has logically failed the test, there is no reason to show any more questions. The test should stop immediately, and a fail decision should be given. Similarly, if a person has answered enough questions correctly to pass the test, the test should be stopped.

A test can be stopped if it is suspected that the test is being taken for reasons other than obtaining certification. Item response theory, a measurement theory providing interesting applications to high-stakes testing, provides the statistical foundation for discovering inappropriate patterns of answers. For example, a test is obviously appropriate if a person is able to answer more easy questions correctly than moderate questions, and more moderate questions correctly than difficult questions. An abnormal pattern can be easily detected if the test-taker answers the same percentage of questions correctly whether they are easy or difficult or, even more strange, a person does better on the difficult questions than the easy ones.

Patterns of responding based on correct/incorrect answers and the time it takes to answer can be analyzed as well. Keep reading for more detail on this approach in the Data Analysis section.

Tighter Security During Development

While security problems during development have been rare, tools and procedures used during the development of a test must ensure that test questions are kept secure. Access to item banks must be tightly controlled. Transmission of item banks should be encrypted and password-protected. There should be no paper versions of items available for any purpose.

Tighter Center Security

Many security problems occur because of sloppiness or intentional laxity at testing centers. Consequently, this has been a strong target for improvement. Better monitoring of center performance, better training and selection of personnel and harsh de-authorization and criminal penalties have all been used to strengthen the security at centers over the past few years.

The use of biometrics, techniques to verify the identification of candidates, promises to reduce the number of problems associated with individuals taking tests for others. Imagine a center where a person has to provide a thumbprint prior to taking a test. Immediately the center accesses a database and matches the thumbprint with the name of an individual in the database. The name is compared to the ID provided by the candidate. If it all checks out, the candidate is allowed to take the exam. The exam results are then forever connected to the name and thumbprint of the candidate. That person would not then be able to take the test for another person. If a thumbprint were shown to match more than one name, the test would not be allowed to continue, and all records would need further examination.

Improved Data Analysis Tools

Probably the area that holds the greatest potential for fraud detection is the use of sophisticated data analysis tools. Taking a test results in the production of data. Two important pieces of data collected are right and wrong answers and the time to answer a question. It is possible, for example, to use the time it takes to answer each question to determine that a person was taking a test appropriately. Inappropriate tests can be easily detected. For example, if a person is taking the test for the purpose of memorizing questions, she will have an unusual response-time pattern compared to those who are trying to answer the questions. See the chart for a list of three test-takers and their response-time patterns for a prominent certification program. The average response time is also given, with the questions ordered from briefest to longest. Notice the correlation of response times for the appropriate test and the lack of correlation for the inappropriate ones.

As more is known about how tests are taken appropriately, it is possible to monitor tests in real timeÐwhile the test is in progressÐand determine before it is finished if the test-taker is behaving as a motivated candidate or is taking the test for some other reason, perhaps only to memorize questions.

Data analysis tools are currently being used to study pass rates at testing centers across different countries or regions of the world. Even testing patterns for individuals can be analyzed. Can you imagine a reason why a person might retake a test after he has passed it? Such a pattern should be available as the person attempts to retake the test and could be used to prohibit it.

Legal Solutions

Those involved in testing fraud can expect aggressive legal action in the future. In August, Microsoft announced the successful prosecution of a man who pled guilty to the theft of trade secrets. He had sold stolen versions of several Microsoft exams on his Web site. The FBI investigated the case, and it was prosecuted by the U.S. Attorney's office.

In another recent and ongoing case dealing with testing fraud, defendants were indicted for arranging to have people take a test for them. The test involved was the Test of English as a Foreign Language (TOEFL), owned by the Educational Testing Service of Princeton, N.J. The U.S. Attorney's office used the federal mail fraud statute to prosecute the individuals.

As certification candidates, each of you signed non-disclosure agreements stating that you would not disclose any material that you saw on the exam. That agreement is a legal binding agreement that could serve as a strong basis for legal action.

Summary

Quality certification programs are difficult and expensive to build. They provide high value to the organization sponsoring the certification, to those holding the certification and to those using the credential for decisions on partnering, hiring, promoting, etc. Certification in IT is a healthy, growing enterprise, serving a specific purpose.

The certification industry will continue to support certification efforts and will respond to direct threats of piracy and other types of testing fraud with effective efforts at detection and prevention. Fraud that is discovered will be dealt with quickly using appropriate, including legal, means. Those of you who value your own certification should applaud these efforts.