What to do About Test Prep
by David Foster
Certification Magazine, 8/2000

In my many years in this business, I've heard numerous complaints by certification program managers about the increasing number of so-called test prep companies and their products. These companies provide practice and learning aids of varying quality that are meant to help certification candidates pass the exams. The offerings are pretty typical, providing screen after screen of multiple-choice questions that are similar to the multiple-choice questions in the real exam and that cover a similar range of content. Some provide scores or other feedback.

The most common complaint I hear is that these products are not providing substantive training in the skills the exam is measuring, only simple practice questions. I've even heard accusations that, for some products, the practice questions may be identical to the actual questions on the certification exam. Additionally, these products are accused of undermining the purpose of the certification and providing candidates with a false view that they can be certified and can work effectively on the job.

Setting the complaints aside, are these programs effective at helping someone pass the tests? It bothers me to say that they probably are. To the degree that the materials match the actual exam format and cover the content well, they should work fine. So, why does that bother me?

Do these test prep programs help a person practice the skills that are used on the job? Probably not. It's not often that a job requires the employee to answer memory-based multiple-choice questions about the job he or she has. A job has important skills that require problem-solving and analysis, manipulating hardware, using software and/or demonstrating customer-service skills. These skills cannot be easily measured by simplistic multiple-choice questions.

Ideally, test prep should be the same as "job prep," helping a person practice the skills needed on the job. But it isn't. And you can't blame the test prep companies. Nor does the fault lie with the candidate for using such programs. Both are reacting to the rule: Answer enough multiple-choice test questions correctly on the certification exam and we'll certify you. Test prep will remain simplistic and unrelated to the real world until the certifying organizations decide to create tests that measure actual job skills more directly.

So, the solution to better test prep is entirely within the control of the certifying organization. Why not create exams that require the candidate to actually complete the job skills? Using a very effective method, some certification programs have simulations of their software in the exams, requiring the test taker to actually complete the job tasks in order to pass the test. Others have required candidates to attend a lengthy lab-based exam where they are observed and monitored in their use of the actual equipment. Some exams include more interactive item types, such as drag-and-drop, essay and point-and-click, to measure the job skills more directly. All of these efforts should help to identify the competent candidates and to motivate them to prepare for the exams by gaining experience.

Additional benefits to the certifying organization will be: more effective support of their products; partners and customers will benefit from the more capable group of certificants;

on-the-job experience will be valued; and experience with products will become a critical component of training programs.

A final benefit will be seen in the offerings of the test prep companies. To stay in business, test prep companies will respond in kind to decisions by certifying organizations to use more performance-based measures in their tests. They will reduce the number of simple multiple-choice questions and add more performance-based items to match the certification exams. When that happens test prep will truly mean job prep and will simultaneously prepare people to pass the exam and to be successful on the job.