It Takes Teamwork to Ensure the Validity of High Stakes Assessments
March 24, 2015
Wrtten by: Dennis Maynes, Chief Scientist
Recently, we finished preparing a 6 1/2 hour workshop on test security. The workshop surveys the entire test security landscape and process. Being a “big picture” person, I find such a presentation to be incredibly valuable. For example, as a result of preparing for the workshop, we can list the essential security controls that need to be implemented by all programs, the threats they are designed to mitigate, and where they fit within the testing process and the test security process. After finishing this work, I came away more deeply impressed, than ever, that “it takes teamwork” to ensure the validity of high stakes assessments.
In many ways, this work is like rocket science. NASA requires many disciplinarians to accomplish its space missions. Likewise, those of us in testing utilize many experts and disciplinarians to accomplish the important work of ensuring our assessments are valid. While this work does not have the complexity of physics associated with rocket science, it shares the complexity of coordinating the strengths, knowledge, expertise, and input from many generalists and specialists.
Psychometricians lead the way for ensuring the validity of exams. They have spent decades producing measurement models that are relevant, defensible, and reliable. These models undertake to perform one of society’s most important tasks, that of measuring human capacity. However, in the same way that rocket scientists cannot complete a moon shot by themselves without the help of engineers, managers, and many other highly trained personnel, psychometricians cannot administer valid tests by themselves. They need the help of content specialists, project managers, item writers, editors, lawyers, statisticians, mathematicians, information and physical security professionals, investigators, public relations professionals, proctors, and software engineers.
The testing process requires a team of highly trained professionals. In a like manner, the test security process requires teams of professionals. Some skill sets are shared between the two processes. There also are unique skills that are required for each.
Because valid test administration requires interdisciplinary cooperation, it is not the domain of any single discipline. This means that process definitions are essential for training each of these experts in their roles and responsibilities. For example, investigators might be highly skilled in conducting interviews, but they will need to learn “the ropes” or “the testing industry” and its unique nuances in order to operate effectively. In the same manner, intellectual property attorneys need to understand the unique requirements associated with secure test content, such as being well-versed in the U.S. Copyright Office’s special procedure for securely registering copyrights.
The testing program manager has the responsibility to ensure that all of these individuals can work well together. Because “it takes teamwork” that is interdisciplinary in nature, most team members have restricted views of the entire process. Some of the team members need to be able to see and manage the big picture. These people are the executive management of the testing program. This means that other individuals, such as psychologists and organizational behavior professionals, may be helpful in coordinating the team’s work and performance.
In my opinion, high stakes testing doesn’t get as much respect as it deserves by the public. We should emphasize that measuring human capacity is one of society’s most important tasks, and it deserves high quality efforts. Despite focus on technical areas, such as item measurement models, the testing industry is really all about people. People are valued and society’s most important aspect.