Exam Security Audit Highlights Challenges and Strengths

NCEES Licensure Exchange
October 2004
Volume 8, Issue 4

The stakes are high in protecting NCEES examinations. The value of licensure relies on their validity. Volunteers devote countless hours to their development. Careers begin based on their credibility. The Council’s reputation rests on their integrity. Above all, the public health, safety, and welfare depend on their soundness.

If an exam had to be re-created immediately because of a complete breach, the cost would be enormous. The intangibles—such as the number of volunteer hours—are not quantifiable, but the tangible costs are estimated to be as much as $400,000.

These are reasons security has always been and continues to be a high priority for the Council. But how much is enough? What is NCEES doing right? What can it do better?

NCEES recently commissioned an independent security audit to help answer these questions. This summer, recognized industry experts from test security firm Caveon LLC spent several days at NCEES headquarters in Clemson conducting the audit. They looked for operational risks associated with test development, publication, shipping, and administration. They searched for physical and procedural security weaknesses. They recommended ways NCEES can improve.

“One of the advantages of the audit is finding out where vulnerabilities are before problems occur,” says Betsy Browne, NCEES executive director. “Each exam administration, two to five ‘incidents’ occur that have to be investigated to determine if an exam should be declared as breached. Over the last three years, one exam has been declared breached. Ultimately, the goal is to have no exam breaches.”

Challenges

Cheating is not new. Many of the means to carry it out are. Calculators can communicate with each other. Internet access is wireless. Cell phones are cameras.

“The biggest challenge that almost all testing organizations face is keeping up with technology,” says Jim Impara, senior director for Caveon Test Security Services. “There are many ways to cheat, and it is very difficult to stop everyone who wants to try. Though expensive, new technology makes cheating much easier.”

It’s no surprise that many of the security audit report recommendations focus on areas where cheating can occur and how to combat the rapid advances in technology. The Council has been addressing these issues for several years, most recently with the more stringent enforcement of policies about what’s allowed inside the exam room. Many of the high-priority recommendations focus on measures already in place or under way. In fact, at this year’s Annual Meeting, the Council approved measures to provide a list of approved calculators and to establish an examination retake policy to control question exposure.

Strengths

The security audit report focuses on correcting security problems rather than highlighting security strengths. However, it does commend NCEES for its corporate culture, which emphasizes that security is an important part of everyone’s job.

“NCEES has a number of security strengths,” says Impara. “First and foremost is that everyone we spoke to is very aware of security. Just having people think and talk about it on a regular basis is the first step in having a secure operation.”

That the security audit even took place is a reflection of that culture.

“This was the first security audit by an outside company,” says Jerry Carter, NCEES associate executive director. “It was held now because of our increasing awareness of security issues and the liability associated with the safety of the exams. We wanted to know how we compare against the industry standard. The audit is a way of getting someone to objectively review measures we have in place and offer suggestions.”

Chuck Wallace, NCEES director of exam development and the staff liaison for the audit, says the next step will be for staff to prioritize the recommendations and come up with an implementation plan to present at the Board of Directors’ November meeting.

NCEES Staff

Exam security audit recommendations

Caveon Test Security Services prepared a report with many recommendations for improvement.

These are a few of the high-priority ones.

  • Prepare a comprehensive security plan.
  • Complete the search under way for the NCEES compliance and security manager position.
  • Create a comprehensive security manual that can serve as a reference for staff and others who contribute to test and information security.
  • Continue the initiative that is under way to institute retake policies that control item exposure by limiting the number and frequency of test retakes.
  • Use multiple forms or versions of a test within a test administration site.
  • Undertake item pretesting to provide a better evaluation of item quality and to minimize the need for item review following test administration.
  • Revise proctor manuals to better reflect a variety of situations that might threaten test security (either cheating or stealing items, test booklet pages, or entire booklets), and explicitly specify what actions the test administrator and proctors are to take.
  • Restrict the reference materials permitted in the exam room.
  • Provide examinees with a short list of approved calculators that may be used during the test.
  • Continue to enforce reasonable retake policies, and complete the process that has already been initiated of adopting registration and tracking procedures for candidates that will permit enforcement of these policies.
  • Facilitate retake identification by having all registration done in one central location.
  • Conduct cheat analyses, especially analyses that will detect answer copying.
  • Conduct routine data forensics to check for cheating and other test fraud that may be occurring. This includes continuing to run the random guessing analysis.
  • Eliminate the distribution of items and forms to and from subject-matter experts (volunteers who write the exams) via e-mail attachments.
  • Update policies, procedures, and training materials to provide more comprehensive direction to management, staff, and test administrators regarding the handling of security breaches.
  • Consider using a Web-based meeting under the control of NCEES staff when items require review after being flagged during preliminary items analysis.

Posted with permission: NCEES Licensure Exchange

Caveon

Leave a Reply