The Economics of Risk Management (view PDF)
By Rob Pedigo

Running a testing program these days is challenging. The current business environment presents some formidable obstacles in the form of tight budgets, headcount restrictions, increasing pressure to perform, and more sophisticated security challenges to your tests.

It is no surprise that the knowledge necessary to create, promote, and protect a testing program is both strategic and detailed. Of those three essential tasks, one is increasingly problematic: protecting your program from the theft of your intellectual property (IP) and the effects of cheating. To successfully combat IP theft and misuse, you need a clear understanding of the value of your testing program and the business risks it faces. If you don’t know what you have, how can you make intelligent decisions to safeguard your asset?

This article describes a method to identify the financial costs of security business risks facing your testing program. Based upon discussions with fellow education and testing managers over the last 15 years, it is clear to me that the costs of test security problems are often not fully understood and appreciated, and as a result, are underestimated even by highly-experienced test program managers.

What’s at stake?
It is a poignant fact that test theft and misuse occur all the time. These attacks reduce the measurement accuracy of your tests, injure the reputation and utility of your program, and necessitate a variety of ongoing costs to fix the damage. Taken together, these costs represent your business exposure caused by IP theft, random or systematic acts of cheating, and test question overexposure.

These costs broadly fall into several categories:
unplanned test redevelopment;

• loss of program credibility with the
  associated costs of damage control;
• loss of testing revenue and other related
  revenue streams;
• loss of test measurement reliability with
  associated legal liabilities;
• and, perhaps worst of all, reacting to constant
  emergencies instead of pursuing new
  profitable business (known as opportunity
  cost).

Because Caveon specializes in security for a wide variety of testing industry segments, not all of the costs listed here may apply to your program. Similarly, some costs that would accrue to your program may not be listed here.

Let’s look at the broad categories of cost that are incurred by security problems in greater detail. Premature item exposure from cheating, or outright publication and sale of your test on the internet can require you to either rewrite your test well ahead of schedule or even pull the test from the market pending redevelopment.

The creation of a validated test is expensive. Organizing subject matter experts’ time and travel, project management expense, psychometric consulting, test publication and delivery, and beta testing expenses can very easily exceed $150,000 for a basic validated test. At best, an unplanned redevelopment effort is disruptive, stressful, inconvenient and embarrassing.

When test security is compromised, it creates a cascade of unwelcomed consequences. Individuals with unfair knowledge of tests typically do not fail; skewing the pass rate and reducing revenue from repeat testing. In time, word of mouth tends to make many security problems public knowledge, resulting in a damaged reputation for your program. This leads quickly to a loss of revenue from sitting fees. For testing programs with ancillary revenue streams such as training and book sales, the revenue downturn is magnified.

Beyond the reduction in revenue, many costs are increased. Unplanned test development has already been discussed and, unfortunately, the costs don’t stop there. Marketing and public relations costs must normally be increased to rebuild an injured program. Similarly, program operating costs tend to rise from an increase in test challenges and complaints.

A great deal of time, money and care go into the creation of a psychometrically validated test so that credentials are granted to qualified individuals. For some testing programs, granting a credential to an unqualified individual could create serious legal liabilities. Just as well-documented development procedures have become a legal standard, failure to maintain a reasonable standard of security throughout the test development, delivery, and scoring process could lead to heightened legal exposure.

There is one last element to calculate—opportunity cost. It’s a straightforward concept. If you were able to redeploy the budget and headcount used to react to security problems toward building your program, what would be your return? Once you total the financial costs, opportunity cost, and revenue reductions, you have arrived at your security business exposure. Knowing clearly what is at risk allows you to proactively manage security rather than face the expense, frustration, and distraction of putting out fires.

Someone recently said to me that having effective security policies, procedures and detection tools was good insurance. At the time, I agreed. However, as I signed a check for an auto policy last weekend, I realized that I would be overjoyed if my insurance policy were able to reduce my car’s operating costs, give it an extra 200,000 miles of engine life, and keep it washed and waxed. rob.pedigo@caveon.com