2018 marks the 15th anniversary since Caveon Test Security first opened its (virtual) doors. Caveon has come a long way since David Foster and the small band of colleagues first recognized and decided to address the industry-wide need for test security. What started as a small team of enterprising pioneers has grown into a one-stop shop for test security solutions and thought leadership. In this article, Caveon founding member Dennis Maynes speaks to his experience at the forefront of this burgeoning field, reminisces on the past decade and a half of progress, and ruminates on what the future might hold.
When we started Caveon 15 years ago, we were enthusiastic—we knew we had important services—but we were a bit naïve. We didn’t realize just how hard it is for organizations to switch from passively endorsing test security to actively managing and promoting test security. I knew we could make an important contribution with statistical analyses—I just didn’t realize how much we would accomplish.
We have brought to market world-class services, which I had hoped to produce, but never considered the total effort that would be required. On the other hand, solutions for some test security challenges and problems are still elusive. Much work remains ahead of us. These past 15 years have been professionally rewarding and challenging. It is a joy to puzzle through the data to bring solid answers to our clients.
The Caveon Family in 2003.
In my opinion, our biggest obstacle was doubting ourselves—not believing that we could really do this. Oh yes, we had challenges that most start-ups face with staffing, organizing, managing, and marketing. After we caught the fire and paid the price for delivering these services, we have continued to progress and make significant contributions to test security and to our clients.
I have the same job title—Chief Scientist—that I had 15 years ago, but my job has changed. Instead of developing software and algorithms, I am now focused on cohesive client support. Instead of wading through data, I am now focused on saying the right things properly. Instead of debugging code and proving difficult computational complexity theorems, I am now focused on communicating and coordinating the solutions to test security problems. Formerly, my primary language was “C” and “Python,” now, my primary language is expressing myself properly in English.
I have several favorite memories. Most of them were due to knowing that we made a difference. I remember reading a tweet from a high school student in Florida which made my day. He wrote, “I used to be a big-time cheater until my FCAT got invalidated!” Those statements are magical. It means that people are taking test security seriously.
Another favorite memory was when the judge commented upon my expert witness testimony in 2017:
“Maynes was an exceptionally competent and credible expert statistician… His demeanor maintained a fine balance between demonstrating mastery of the subject matter, while remaining patient and mild with both representatives. He was, in short, a highly credible and persuasive expert witness.”
I never expected such a comment to be made about my work or testimony. I was overwhelmed and grateful to be able to do that work.
The Caveon Family in 2005.
Caveon is unique in at least three ways. First, as an independent third party, our work is fair, balanced, and unbiased. Second, Caveon’s commitment to clients is exceedingly strong. Every single employee at Caveon knows the work being done is valued by our clients. Third, Caveon remains intensely motivated to strengthen exam security. This motivation permeates all our services and drives our desire to innovate and improve.
After we started Caveon and I was asked what I did for work, I would reply, “I use data to chase cheaters.” I finally understood that the primary goal of test security was NOT to change the behavior of a few. Indeed, the goal of test security is to ensure that the test scores reflect what they were designed to measure.
The number one rule is that the test scores should be trusted and they must be valid. If the test scores don’t mean anything, why do we give tests? Just, to mark an item as “complete”? No, tests and test scores are important. They are very important to judging competence and capability. Besides this importance, the tests need to be fair. Test security violations undermine validity, trustworthiness, and fairness. Our world relies upon tests to accomplish their purpose. If these things are important to you, then you have a reason to care.
I initially thought that security was a behavioral problem. I now know it is a validity issue. I initially thought that security could be improved by simple measures. I now know that test security is best served when “a village” comes together and works cooperatively towards the common goal of strengthening test security. Everyone who touches tests has a critical role in test security. I initially thought that test security was a process that could be summarized as “prevention-detection-remediation.” I now know that test security requires a tight laser focus on protecting against threats and risks. The correct test security solutions vary between clients. There is no “one size fits all” approach to test security.
The internet has matured, evolved, and enlarged in 15 years. This means there are more nooks and crannies, and more deep, dark, hidden places where pirates, item harvesters, and cheaters lurk online. It’s also become simple and easy to steal exam content with hidden cameras and upload them to the internet. So, the theft and disclosure of secure exam content is easier by orders of magnitude than it was 15 years ago.
At the same time, many of our test security processes have remained unchanged from those that were used in the last century. The number one security concern of every testing program manager is the theft and disclosure of content that took months and years to produce at the cost of tens or hundreds of thousands of dollars. These concerns are even greater today than they were 15 years ago.
Caveon has been the catalyst to foster myriads of test security conversations around the world. I, personally, have discussed test security with people from the USA, Canada, Mexico, all of South America, Africa, Europe, Asia, Australia, and elsewhere. Fifteen years ago, The Standards on Educational and Psychological Testing only briefly touched on test security, and now they are highly regarded by the United States and many other countries and represent the gold standard in guidance on testing. Today, the National Conference on Measurement in Education is producing position statements on test security. Fifteen years ago, test security standards in state assessments did not exist. Today, joint publications on test security have been produced by the Association of Test Publishers and the Chief Council of State School Officers. Not coincidentally, with Caveon’s embarkation on this journey, awareness and discussion of test security has come to the forefront in many organizations.
Innovation is hard. It takes time. You need to prepare to run the marathon. This is not a sprint. As you pursue improved test security solutions, there will be many setbacks and challenges. Do not give up. Persist. Learn from others.
The future will bring even more technological change. Caveon Test Security will either adapt and embrace the future, or it will become obsolete. (We prefer the former.) Because we know we must adapt, I am counting on Caveon continuing to change, to improve, and to support the mission of testing and test sponsors.
The Caveon Family in 2018.
Topics from this blog: Test Security Stories
