Written by: Patrick Martin, Assessment Project Manager, Caveon Test Security
Because I spent a few years working as a carpenter in my dad’s construction company, I get a lot of questions from friends about home repair. Once, I received an email with the subject “I think I messed up.” Attached was a picture of a dryer duct sitting inside the wall with four fresh nails holes punched through. My friend had been searching for wall studs using a trick I taught him and had mistaken the duct for a two by four. It was a classic rookie mistake. I wrote back to tell him how to fix the holes and congratulate him on achieving a home repair rite of passage.
Now that I work in test development, I’ve found that working with new items writers can be a lot like advising my friends in their construction efforts. New writers often make the same mistakes as they learn to write items. In most cases, they make just one mistake per item, but sometimes they’ll combine the errors like in the item below.Objective: Explain how to securely mount a shelf or towel bar to an interior wall Item DYI.121.2.1 Answer: C Drywall, sometimes called sheetrock, is a panel consisting of gypsum pressed between two sheets of heavy paper. Which statement about drywall is true? A: It is rarely used in bathrooms.
B: It is similar to plywood in durability.
C: The paper provides much of the material’s strength, but is not strong enough to hold a screw in place unaided.
D: It is used as a base for ceramic tile
This item demonstrates three of the most common rookie mistakes:
1. The item fails to support the objective. Knowledge of drywall’s limitations is related to the objective, in that it can help candidates understand why one method is more effective than another. However, testing this knowledge will not show that the candidate understands how to keep the towel bar from falling off the wall. Questions for this objective should be more procedural or at the very least ask about the tools and hardware one would need to finish the job.
2. This mistake is also in the stem. In an attempt to make the item appear more robust, the writer included unnecessary information about drywall. This type of error has a few variations. New writers will also try to make questions more difficult by burying the essential information in a mound of details or telling a story that has nothing to do with question being asked. At best, these tricks just add a few extra words for the candidate to read; at worst, they make the item needlessly confusing.
3. The correct answer stands out. Many new test writers fail to understand that the wrong answers are just as important to a successful item as the right one. As a result, they will make the correct answer longer than the distracters, or structure it differently. Most of the time a candidate who does not know the correct answer will simply pick the option that looks different. If this were the only problem with the item, I would have the writer shorten and restructure answer C, so it matches the remaining options.
If the writer corrected all of the errors, the resulting item might look something like this.Item DYI.121.2.1 Answer: D A contractor is mounting a 14-inch shelf on an interior wall. The shelf weighs five pounds, and will support a small statue weighing 6 pounds. The customer has directed the shelf be placed in a spot that prevents the contractor from attaching either mounting bracket to a wall stud. What is the most efficient method of securely hanging the shelf? A: Open the wall and install additional blocking to support the shelf. B: Install plastic expansion anchors for the mounting bracket screws. C: Attach the brackets to a 1×4 long enough to reach the nearest studs. D: Mount the brackets directly to the drywall using 3/16” toggle bolts.
This item does a better job measuring the candidate’s ability to complete the task. The stem is longer than the original question, but all of the information is necessary because placement and total weight affects the method the contractor would use. Finally, all of the distracters are roughly the same length and use roughly the same construction. Like a properly installed shelf, the item just feels solid.
These three errors are, unfortunately, not the only mistakes rookie item writers make. If you want to learn about other newbie gaffs and pick up a few tips for writing stronger test items, then take a look at Caveon’s new online item writer training course. Just follow this link and enter the code newbiewriter.