Written by: Jamie Mulkey, Vice President, Caveon Test Development
Breaking Bad, the critically acclaimed TV series about a high school chemistry teacher gone methamphetamine drug dealer returned last week drawing 5.9 million viewers.
For those of you unfamiliar with the series, the premise of the show begins when the protagonist, Walter White, has learned he is dying from inoperable lung cancer. All he wants to do is to make sure his family is provided for. As the show progresses, Walter becomes a hardened, unsympathetic drug kingpin. It is apparent that as he causes more mayhem and damage, he will have to atone for his sins in some form or fashion. After all, nobody gets away with murder on TV or in the movies… or do they???
The premise for this TV show got me thinking about the consequences of writing bad test items. How, many times, misguided item writing habits lead to a set of unintended consequences that are less than favorable. It becomes a spiral effect. Sort of like the Direct TV commercial where the guy wakes up in a ditch because he has cable TV installed instead of Direct TV.
A commercial about bad item writing habits might go something like this:
When you have untrained SMEs, you get poorly constructed items.
When you get poorly constructed items, your test takers complain.
When your test takers complain, they resort to getting the answer key from braindump sites.
When they resort to using braindump sites, the wrong people pass the test.
When the wrong people pass the test, you have unqualified people.
When you have unqualified people, someone just does it wrong (like the time my dentist said “Oops!”)
What can we do about breaking bad item writing habits?
First, we can make sure we have provided good solid training for our item writers. This should include:
- providing a solid foundation on psychometric item writing principles
- teaching writers to create items that test a single concept, and
- verifying that the items they create are congruent with the objectives
Next, we can educate the item writer as to their role in the item development and test construction process. This should include:
- understanding the test’s purpose and its intended audience
- highlighting the expertise they lend to the process
- appreciating the importance of maintaining security during test development
- following the item writing guidelines and learning how to work within the item development tools
Lastly, we as test developers can facilitate good item writing habits. This should include:
- assisting item writers with item construction and wording ideas
- verifying the importance of the concepts being tested, and
- challenging item writers to come up with new ways to assess knowledge, skills, and abilities
I’m sure those of you who are Breaking Bad fans will be sitting on the edge of your seats waiting to see what consequences await Walter White. I’ll be sitting on the edge of my seat hoping we can get rid of bad item writing habits.