On Peanut Butter and Exam Writing
Written by Patrick Martin, Assessment Project Manager
Recently, my work with item writers reminded me of my seventh grade science class. In order to teach us how to write procedures for our upcoming science fair projects, my teacher asked the class to write down the instructions for making a peanut butter sandwich. After he had collected our work, he brought out the required materials and tools for making the classic sandwich. He then selected one of our instruction lists at random, and read the first step out loud.
“Put the peanut butter on the bread.” He asked if that sounded right to us, and we all nodded yes. Obviously, that was the first step. We had no idea how it would not be the first step until my teacher picked up the unopened peanut butter jar, and set it on the bread bag. We began to protest, “No, you’re supposed to take the bread out of the bag, and use the knife to put the peanut butter on the bread.” My teacher just smiled and said, “That’s not what the instructions say.”
The goal of the lesson was to show us that we had to be clear and specific in our instructions, because we cannot assume a person who later attempts to follow the procedure will think the same way that we do.
It was a fun activity that I had mostly forgotten about until I began training item writers, and I discovered that most writers, regardless of the objective, will write an identification item or write an item about some arcane piece of trivia they found in the learning resource. The initial items are sometimes so far off, that I wonder if the writer read the objectives.
However, there are times when the alignment problems are not entirely the writer’s fault. Very often we encounter objectives that are unclear, or are so broad that although the items support the objective as written, they miss course designers’ intentions. In the worst instances, the objectives are impossible to support within the context of a multiple choice exam.
These problems are the most frustrating for someone in my position. I have to translate the client’s request for the writers. Sometimes the client provides clarification, but sometimes the best we can do is make an educated guess. Inevitably, at some point, we put the jar on the bread bag and wind up having to replace items late in the process. To make it worse, just a little bit of extra information would have prevented the problem in the first place.
This does not mean I want especially elaborate or complicated objectives, but I would like course designers to remember objectives are instructions. If you are writing objectives, take the time to ensure that you have selected the right action verb for the objective. That will solve most alignment problems. I have also found statements clarifying the scope of the objective to be helpful. Scope statements are also handy because they can reveal overlapping objectives early in the process.
Finally, it helps to recognize that some problems with objectives will not be revealed until item writing begins, so I recommend having a process in place for addressing problematic objectives, at the very least, be prepared to answer the question “what do you want the items for this objective to look like?” If you put in the extra effort in these early stages, you’ll be far more likely to be happy with the exam in the end.