The Secret Psychometrician’s Society
Written by: Jamie Mulkey, Vice President, Caveon Test Development
I recently spoke to a young woman who was lamenting about her previous job as Test Development Project Manager. Apparently, her job was to focus on true project management skills; managing the project schedule, interacting with clients, and making sure milestones were being met. She was judged to be lacking the ability to lead test Technical Reviews or Standard Setting sessions. That was left up to the psychometrician.
Hmmm, I thought. Was there anything so special about a psychometrician’s capabilities that would preclude others from performing the leadership role in these two functions? I thought about this more. What were the critical skills necessary to run tech reviews and standard setting sessions for developing exams:
- Facilitator – managing a large group and make sure everyone’s voice is heard
- Content Smoother – knowing enough about the content being reviewed to ask questions of the content experts; enabling changes to items that will make the test a better measurement instrument.
- Time manager – making sure all items are reviewed given the time allotted.
- Item construction Evaluator – identifying flags such as ‘none of the above’, the ‘correct answer is the longest’, and ‘negatively-worded stems’ to create more measurable items
- Mediator – knowing when to cut off the discussion because it has become a ‘battle of expertise’.
- Consensus builder – making sure everyone is happy with the end result of these review meetings.
- Statistician – knowing enough about classic test theory to apply it to standard setting practices.
In reviewing these skills, I didn’t see anything that could not be taught to the most enthusiastic project manager. Maybe there’s a secret handshake? A phrase or code word only uttered by those who carry the title of psychometrician? Maybe as part of a psychometrician’s dissertation oral defense they have to construct a cut score by hand?
My point here is that by adding these skill sets to the Exam Project Manager’s tools, you not only make a more well-rounded project manager, but you extend the capabilities of your work force. Let’s face it, there are more project managers than there are psychometricians. It would be a shame to create a bottleneck in the test development process waiting for a psychometrician to conduct a technical review or a standard setting session.