Written by: Patrick Martin, Assessment Project Manager, Caveon Test Development Services
It does not seem like this sort of thing should exist, but I am an item writing nerd.
Seriously, I have found myself on more than one occasion bordering on giddy when I come up with a novel way of approaching an objective and genuinely frustrated when I have to begin a question with “Which statement is true.” In short, I spend a lot of time thinking about items and how to improve them. I assume that all item writers share my obsession until I put on my project manager hat and find myself reviewing questions like the one below.
Objective: Explain how specific features of the WonderWidget can improve end user efficiency.
Which WonderWidget Industries’ solution is best for increasing end user efficiency by providing for the creation of user customized keyboard short cuts and live data displays within the desktop environment?
B: UltraFaxPro Enterprise
C: WonderWidget 9
D: WonderWidget Lite
This item is technically accurate and just barely supports the objective, but I would most likely bounce it back to the writer. The stem is awkward, and the distracters are useless. Moreover, there are better and more interesting approaches to this topic.
For example, scenario-based questions are made for this objective. When I ask the writer to fix the question, I would suggest that she or he present a problem to the test taker and ask how the WonderWidget could solve that problem. Ideally, the stem would look something like this:
A technical support representative communicates with customers via an online messaging service. The representative has identified the ten most frequent problems customers encounter and has pre-written responses in separate documents, which can be cut and pasted into the chat window. The system is moderately faster than retyping the answers, but finding the correct response still takes more time than the representative would like.
How should the representative use the WonderWidget 9 to best improve efficiency?
In addition to being better suited to the objective, this stem demands more of the examinees. They first have to analyze the scenario to identify the user’s needs and then come up with a recommended action. A knowledgeable candidate will know the answer to this question before looking at the answer options.
However, since many candidates will ultimately try to reason through the answer options, I’d push my writer to spend extra time on the distracters and would recommend specifically describing the course of action the representative should follow. Instead of something like, “Use the text expander,” I’d want to see:
A: create a text expansion macro for each response
B: copy each response into the desktop quick display
C: assign each response document to a keyboard hotkey
D: place shortcuts to each response in the tool bar
None of the options stand out as obviously correct. They are all of similar lengths and identical construction. Now instead of a crummy give-away item, we have the kind of item that separates the technical sales masters from the technical sales poseurs.
I’d love to hear from you – what do you do to help your item writers create items that elicit candidates to think and reason?
If you need some help with your item writers, Caveon can help. With our new Rent-a-Writer program, we can send one of our experienced item writers to your next test development meeting. The quality of your items will increase dramatically.