When was the last time you had a really fabulous dining experience? I mean, a really memorable meal? Was there great food? Wonderful conversation? At a price that wasn’t equivalent to a car payment?

When I think of a great dining experience, I have several criteria I consider:

Not unlike a great dining experience, a good testing organization needs to bring a lot to the table when it is designing good tests. The right processes have to be in place, team members have to be knowledgeable, and most importantly, credible items need to draw in test takers so that they see value in the testing experience. Here are a few criteria you may consider for you own test development program:

I’ve spoken with many testing organizations recently. There is pervasive need for test development in today’s market. With this demand, it sometimes becomes necessary to include a partner with a solid approach to test development. An organization that complements what you do; much in the same way the right beverage complements a good meal.

We believe our new Caveon Test Development™ services provide a five-star test development experience at a price that won’t leave you seeing stars. If you’re considering a test development project and need help from Caveon, please let us know info@caveon.com. No reservations required.

Oh, and leave us a comment about your favorite restaurant!

My long-term friend and assessment colleague, Doug Rindone, who I have known from his days with the Connecticut Department of Education, was talking about how he evaluated whether a security incident was serious enough to warrant a thorough investigation.  “To me it came down to Motive, Means, and Opportunity,” he said. “Just like you see in movies and TV shows about police work.” I have been thinking about the fine logic of Doug’s approach and decided to make that perspective my contribution to this week’s Caveon Security Insights blog.

Motive

If there is a report or some evidence of failure to follow a testing rule, how can you evaluate the issue of motive of the individual or individuals involved?  A key issue is what is riding on the test results.  If it is a state assessment program, the status of a school or class may be at risk of receiving a negative classification. A teacher or principal’s job may be at stake.  If an admissions or certification test is at issue, access to a highly sought credential or program may hang in the balance.  It may be the case that a teacher has concluded that raising scores is essential regardless of how that outcome is attained even if that was not the intent of school officials.

It is easy to fall into the trap of talking always about our goal of increasing test scores instead of about improving students’ college and career readiness or other skills-based statements.  It is important to always emphasize real educational goals, not “proxies” such as test scores.

Means

Suppose we have established a motive; what about the issue of means?  How might someone have gone about improperly influencing test scores?  The distinction of actions that might be taken before, during, or after testing can be very useful here.  Looking first at the time before testing, did the person being evaluated have a way of looking at tests before hand?  As to the time of testing, was the person the administrator for the test? And did he or she have a more or less free hand to conduct themselves as they judged appropriate or was there close monitoring of the testing process? Or was there possibly even a second educational professional observing all testing activities down to the classroom level?  Turning to the period after testing, did the individual have access to answer sheets and test keys in a manner not likely to be observed?  If test books are closely monitored, opportunities for testing misbehaviors by anyone are much reduced.

Opportunity

Suppose that we have determined both a conceivable motive and a possible means by which misbehavior may have occurred, was the specific opportunity present, as best we can determine, for the violation of testing rules to have occurred? It is virtually standard practice in state assessments, for example, to set quite explicit rules regarding the “chain of custody” for test materials.  If the rules are followed, the classroom teachers who are the usual administrators for state assessments receive the test books just before testing and return them very promptly at the close of testing. 

What is the evidence for a particular district or school regarding adherence to these rules?  If all procedures were the result of explicit training and if the process was carefully monitored, it is unlikely that classroom teachers acting on their own would have markedly departed from established testing rules.  If we were to determine, though, that the “locked cabinet” within which tests were to be stored, turned out to have many keys and was in a school area where it would be fairly easy to gain access to tests and answer sheets, the problem of opportunity to misbehave becomes much more worrisome.

Closing Thought

The Means, Motive, Opportunity perspective seems a possibly valuable way to help determine how far to pursue a possible security breach.  It is not a “magic bullet” and solid professional judgment will continue to be essential when monitoring and following up on reported security interests.

Please Follow @Caveon on twitter

What might happen if just anyone could walk into your home and take it over as if their own, because it wasn’t secure? What if they moved in and filled your home with all their “stuff”? Just imagine every drawer, cupboard, and even the fridge filled with unusable “junk”. Now, go one step further, what if they polluted your air with toxins and antigens? Such a security breach would devastate the quality of life that you and your family enjoy. Your safety, security, and hygiene would be compromised on every level and the feelings of security you have when coming home after a long work-day would be gone.

The above situation is how many people experience the Internet. The web is very useful, but it’s also filled with junk and even toxins. For example, there are squatters (i.e., individuals and companies) engaged in illicit activities; and there are scammers who will exchange your money for nothing of value in return. Some of these miscreants will ruin your credit by compromising your financial information or by stealing your identity. Others spread misinformation that could jeopardize the health of those who rely upon their “snake oil.” Using bad information obtained online can have serious repercussions. A student might get a lower grade because information obtained from the Internet was wrong. Likewise, bogus and bad test preparation materials obtained from the Internet can hinder candidates from earning desired certifications. These scenarios and countless others happen when we utilize the Internet naively, without verifying and double-checking sources.

Illicit websites can and will compromise an individual’s security without a second thought. Ongoing investigations have shown that misinformation provided by unreliable sources can and will harm those who use it. Malicious Internet activities are pervasive. Even with the best safety precautions, individuals and corporations can’t entirely prevent valuable information from being stolen and compromised. When such a breach occurs, thousands of dollars may be required to repair the breach. It is of the highest importance that we, as users, identify and avoid online scammers and squatters who are doing all they can to take advantage of us, steal our information, and compromise our security.

Great news!

Caveon just expanded its services by buying Catalyst Test Improvement Services, a company that has been quietly authoring, cloning and editing items for Fortune 100 companies for more than ten years. The acquisition gives Caveon a running start in its plan to offer a unique set of test development and psychometric services. And, even more importantly, the acquisition enhances Caveon’s value proposition of proven practical protection.

As there are many organizations providing test development services, how is Caveon going to be different? Good question. Let me list just a few ways in which we hope to enlarge our vision of administering test fairly and securely.

First, we will take particular care of our client’s items and tests at every step of the process, guaranteeing that those who “touch” those items are trusted individuals, undergoing background checks and signing non-disclosure contracts.

Second, we will focus on helping our clients have more secure exams. One way we will do this is by cloning or refurbishing items when additional forms, clean forms, or a larger item pool is needed.

Third, since we are experts in secure item and test design, we will specialize in the production of security-friendly item and test designs. On the item side, such designs use automated item generation, item families, and specific formats like the Discrete Option Multiple Choice to minimize item content exposure, and bring with them a number of other advantages to a testing program. Test designs that reduce item exposure rates, such as adaptive tests, tests with early stopping rules, tests that detect cheating or theft, and many others. As we are doing with our data forensics service, we will maintain a research program regarding optimal secure design.

Finally, we will continue to protect test results and other data for our clients as if it were our own.

At Caveon, we are excited that secure test development services enhance our core mission of assisting our clients to administer tests fairly, securely, and with integrity. I invite you to find out more about these services. Contact us. We will be glad to explain how we create tests for you more quickly, with better quality, less expensively, and, of course, more securely.

Next month, on the 23rd and 24th of May at the University of Kansas there is going to be a very important conference. For the first time ever people are getting together to present and discuss principles of data forensics. Data forensics, the statistical detection of exam fraud, has proven its worth to testing programs in many ways over the past few years including directly detecting “hidden” fraud, determining the extent of damage of a breach, and validating the effects of a security vulnerability. It can even reliably tell if a single examinee has produced a valid score or one tainted by cheating.

In the news, data forensics has been the foundation of investigations in highly visible breaches. The Atlanta Public Schools scandal and many other similar occurrences of fraud come to mind. Recognizing its value, the US Department of Education highlighted data forensics methods in a recent symposium on test integrity in Washington DC. And specific innovative methods were presented at the February ATP conference. Obviously, data forensics will play a larger role going forward in every high-stakes test analysis.

Given the cheating that is rampant worldwide, using forensics analyses is a wonderfully effective arrow in our security quivers. Click here for the link to a description of the conference at the University of Kansas and to the program.  While the content is somewhat technical, each presenter is showing and discussing how these methods apply in real-life testing programs in certification, licensure, education, and in the workplace. While the statistical theory and methods may be difficult to follow–and I certainly will have that problem–the application of those methods will be clear. Besides learning how these new methods can help you, you will gain an appreciation of the fact that they are solid and legally defensible if challenged. If you are not able to attend yourself, then recommend a colleague. Make sure somebody from your organization is learning about this emerging science. I hope to see you there. We’ll get together between sessions and discuss what it all means and how to use it.

Last Sunday, a team of journalists at the Atlanta Journal Constitution (AJC) published a series of stories describing a gain-score analysis that they had conducted with the intent of uncovering cheating in schools. There were 69,000 schools from over 3,000 districts represented in the analysis. The overall finding reported by the journalists was “high probability of cheating” in 196 school districts. This inference was made by extrapolating patterns observed in Atlanta to the entire nation.

Since the story was first published, I have read the nationwide reaction with great interest. Readers have reacted in a variety of ways, some of them are:

1. Yes, standardized testing is subject to manipulation and it should be abolished,
2. The newspaper analysis was flawed, so we shouldn’t take it seriously,
3. There’s no surprise that teachers are cheating, and
4. Our district has experienced mobility, boundary changes, or other events which easily explain gains and losses.

Being a professional who performs data forensics analyses on a daily basis, I have been intrigued by these and other responses. I am quite aware of the strengths and weaknesses of statistical cheating investigations conducted by the AJC and others. The overriding thought that I continue to have is: “We must improve.”

Simply creating lists and disrupting organizations from pursuing their core mission should not be done haphazardly. A charge as serious as “high probability of cheating” should be backed up by numerical estimates. Unless cheating and other alternative hypotheses have been explicitly modeled, there is NO way to compute probabilities of cheating. We can compute probabilities of observed data when probabilities of flagging classes are assumed to be the same for the entire state. And, those are the numbers that were reported by the AJC (the analysis assumed flag rates were independent of educational factors such as grade level, subject area, class sizes, teacher effects, and school demographics). But, those numbers should NOT be interpreted as probabilities of cheating. Doing so is erroneous and commits a logical fallacy.

The article by the AJC contained this interesting statement: “Federal education officials and testing experts have begun working on new recommendations for detecting and investigating test-score anomalies.” I support this initiative. The new methods that must be developed need to have broad-based support. They need to be credible. They need to be supported with realistic guidelines for conducting investigations. Many cheating investigations were conducted during the past twelve months. Most of them found no wrongdoing. Our statistical methods must be stronger than this. The new methods must have defensible estimates of false positive and false negative rates.

Arthur Conan Doyle put these words into his famous detective’s mouth: “Circumstantial evidence is a very tricky thing. It may seem to point very straight to one thing, but if you shift your own point of view a little, you may find it pointing in an equally uncompromising manner to something entirely different.” In the future, we will require data forensics methods that interpret circumstantial evidence appropriately and are not subject to equivocation. In spite of my call for improvement, let me state as a final thought that the current methods are better than no methods. And, the AJC analysis of data in Georgia DID indicate serious testing irregularities, which were confirmed by investigation.

According to some sources, the phrase above originated in ancient China.  And, depending upon who you ask, it reflects either a curse, or a blessing.  The phrase rings loudly these days for all kinds of reasons–political, economic, environmental, etc.  This is particularly true in the testing industry and profession, where I’m witnessing a convergence of events, movements, and initiatives involving test security which is, at the very least, interesting–A better description may be “ground-breaking.” There is so much happening on various test security fronts right now, it may require more than one blog entry to explore them all.  Interesting times, indeed!

Next week, my colleague John Fremer and I will present a webinar entitled “Lessons Learned at ATP and NCES.”  It’s interesting to me that two very significant players in the wonderful world of testing, the Association of Test Publishers and the US Department of Education, held concurrent events in late February where test security was a primary theme.  Just a few years ago, an attendee might have seen only one or two workshops at the ATP Annual Conference dealing with security issues.  By my count, there were at least ten sessions focusing on the topic this year, many (sadly) at the same time!  Then, at the early morning ATP Security Committee breakfast meeting, there was “standing room only” as various volunteers and contributors shared the work and progress on various security fronts accomplished in the last year.

During the same week as ATP, the US DOE’s National Center for Education Statistics held a “Testing Integrity Symposium” to generate ideas and discussion to help ensure the fairness and validity of state assessment test results.  This watershed gathering brought state assessment leaders, academics, vendors, and government officials together to collaborate and share important trends and technologies that can improve the integrity of K12 testing.

This session followed up an important summit last fall where the US DOE asked a select group of test security thought-leaders to convene and begin to build a plan for confronting the challenges of test fraud in education.  The NCES symposium was one step in that plan, and based upon a notice sent to Chief State School Officers by Secretary of Education Arne Duncan, we can expect more from the Feds.

These events are not the only things afoot.  Later this year, the revised “Operational Best Practices for Statewide Large Scale Assessment Systems” will be published.  The document is the culmination of a fruitful collaboration between ATP and the Council of Chief State School Officers (CCSSO), and an entire chapter is dedicated to security issues.

Another part of CCSSO, the Technical Issues in Large-Scale Assessment (TILSA) State Collaborative on Assessment and Student Standards (SCASS) just contracted with a vendor to conduct extensive research, surveys, and to prepare a Guidebook on analyzing test results for state assessment leaders

With so much focus from so many directions on test security, it’s not surprising that Data Forensics, the field involving statistical analysis of test response data, is garnering ever-greater interest and attention. Inevitably, top psychometricians and statisticians would decide to gather and explore the most cutting-edge innovations in this new, exciting field, and share research regarding its scientific defensibility.   That gathering is now scheduled for late May, as the University of Kansas’ Center for Educational Testing and Evaluation is hosting the first-ever, “Conference on Statistical Detection of Potential Test Fraud.”

Where will the industry go from here?  Well, the stakes in so many test programs continue to rise, so the subset of test takers seeking a short cut will undoubtedly grow, too.  For this reason, it’s likely we’ll see more and more research, events, innovations, and initiatives focused on test security.  Indeed, here at Caveon, we consider ourselves both cursed and blessed to live in these interesting times.

“I never hear from them anymore.” When was the last time you thought that about someone you know? When was the last time you were the person a friend said that about? In our busy lives we get pulled in so many directions in today’s world that some of our most important relationships, personally and professionally, can unintentionally fall by the wayside. That’s a shame because these contacts, acquaintances, friends and family can sometimes be just what we need to lift our spirits, be the catalyst for a new idea, or help us with a difficult problem.

This same relationship management is vital in your online efforts to protect your valuable intellectual property. You have many points of online contact, yet as the weeks, months and years roll by you tend to focus on those contacts that seem immediately critical to the security of your exams and other valuable content. However, it is important to continue to grow the more peripheral online relationships that may on the surface seem somewhat trivial yet if properly and diligently nurtured can become a powerful and free source of intellectual property protection for years to come.

We’re talking about the relationships you have with the administrators and moderators of the discussion forums, blogs and chat rooms where your content or industry is discussed.

You do have those, right?

Site administrators and moderators are the first line of defense when it comes to what is posted on their sites and by nurturing friendships with these people they will be willing to help you in many of the following ways:

• Be willing to post a non-disclosure statement you’ve written in a prominent and permanent location on their site

• Keep an extra watchful eye for signs of cheating during the administration periods of your exams

• Notifying you of possibly infringing posts in real time

• Immediately remove infringing information if you deem it necessary

• Permanently banning guilty parties from ever posting on that site

However, if you haven’t been in contact with these individuals and fostered good relationships with them it can be difficult and expensive to have this information removed.

All it takes is a little message now and then to the various site administrators and moderators from time to time asking how their site is doing and maybe offering the latest company updates or something else that may be helpful to their readers, anything to let them know you’re still there and that you care they exist, so they never have to say about you…”I never hear from them anymore.”

These are times of great stress for measurement staff in many kinds of testing agencies, particularly at the state assessment level.  The temptation to “flee the field” must sometimes be strong, when yet another development complicates your work, perhaps a new focus on using student test scores to evaluate teachers or a series of articles criticizing your testing program or you personally.  One possible thing to try to boost your spirits is to turn to Lord of the Rings for comfort.  Scroll down to watch the clip “Aragon – Rally the Troops Speech” and listen to the wonderful words that ring out “The day may come when the courage of men might fail.  But it is not this day.”

If that does not work, take a look at another site, that for the National Center for Educational Statistics (NCES) Testing Integrity Symposium of February 28, 2012.  It does not have the fabulous production values of the Lord of the Rings movies. It does, however, have a great deal of solid good sense.  NCES assembled a group of experts for a day-long meeting on preventing, detecting, and investigating possible testing misbehaviors and did it extremely well.  I participated in this fine event and I was so impressed to see speaker after speaker follow instructions about the kinds of standards we have or still need, the methods that we can follow, and the resources that we can draw on. 

We had as presenters “the guy who wrote the book,” indeed two authors who have contributed very helpful and influential books – Greg Cizek “Cheating on Tests: How to Do It, Detect It and Prevent It” and the co-author of “Freakonomics” – Brian Jacobs.  Other speakers were State Assessment Directors, Researchers, Legal Scholars and Practitioners, and testing industry professional staff.

For a complete set of presenter slides, see the NCES website and look for a promised white paper based on the Symposium.  To whet your appetite, here are some of my personal takeaways:

• Take advantage of existing projects to obtain information and ideas, e.g., the Chief State School Officers and Association of Test Publishers “Operational Best Practices” documents
• Develop and strongly support standards of test fairness and adherence to the rules of testing in your setting
• Find ways to train the many people who play key roles in testing, but who have very little education about measurement
• Conscientiously follow up on any indication of testing misbehavior and follow through as warranted when you confirm misbehavior

This is only a “starter set” of ideas that were presented at the NCES Seminar.  Please feel free to contact me directly with specific questions or requests for information. The Symposium has made me even more determined than usual to be helpful to those working so hard to deliver fair and valid tests.

The Reader’s Digest had a humorous anecdote in the January 2012 issue. A marine recruiter helped a color-blind recruit memorize the order of the colors on the test to help him pass the physical. When shown the test page during the physical exam, the young man repeated the color order perfectly, at which point the doctor replied, “But, I showed you page 2 of the test.” BUSTED!

My favorite story of catching cheaters was related by a college professor. Four students did not attend class on the day of the final exam. And, they asked to take the final late. Their excuse? They went out of town together and had a flat tire. The spare was also flat. Having no proof that the story was false, the professor relented and allowed them to take the final. The professor placed an additional question at the end of the final given to these students: “Which tire?” BUSTED!

That professor understood the power of statistics. Each of the four students had four choices for the last question. The probability they would all choose the same choice was 4x(1/4)x(1/4)x(1/4)x(1/4) which is one chance in sixty-four, or about 1.6%. The professor played a very sure bet. You might feel that it would be unfair to ask for all four to remember. But, they knew the exam schedule and they took their chances. This was particularly dicey for the students because if the story was true all four would need to remember which tire was flat. The probability that exactly three of the students would agree was twelve in sixty-four, or nearly 20%. It’s likely the professor would only believe the story, if all agreed.

Both of the above stories illustrate that tests can be manipulated in order to catch cheaters. The second story illustrates the use of an embedded verification test. An embedded verification test is a test contained within a test that is designed to establish the veracity of a test result when the test questions may have been compromised. Next week, at the annual ATP conference, I will be presenting research on the use of embedded verification tests. I am intrigued and excited about this technology because it provides testing programs a principled way to verify that test takers are not accessing brain-dump content. The impetus for using the approach is not to detect cheaters. It is to ensure that tests were administered fairly and with integrity.

Just as the professor did with the four students and the flat tire, the power of statistics is unleashed to accomplish this purpose. By adding ten multiple choice questions to a fifty question exam at the same difficulty level as the other questions, a test taker with a score of 90% on the main questions (45 out of 50) would have a probability of less than one in one hundred of answering fewer than six of the ten EVT questions correctly, which is very likely if the test taker had prior access to the main exam but not to the EVT questions. These are even lower odds than those accepted by the professor. If the professor wanted better odds, she could have asked the time of day or the location where the flat tire occurred (if those details had not been provided by the students). In the same way, the score threshold of the EVT questions can be changed, or more EVT questions can be used in order to obtain stronger results.

Next week, I will present answers to the following research questions:

1. How many EVT items should be included in the test?
2. Which EVT item types provide the greatest discrimination power between ethical test takers and those who use brain-dump content?
3. Can simple statistical tables and rules be created for using EVT items?
4. How can we be assured the EVT items are truly catching cheaters?

At the end of the day, some cheaters will learn of these new technologies when they are BUSTED! Honesty is truly the best policy.

Thanks for reading! We’ll see you next week at ATP in Palm Springs!

Cheaters never win and winners never cheat.  This is an old saying that may sound confusing because cheaters may win in the moment.  However, in the long run, they may feel embarrassed, guilty, feel they have let themselves or others down; and, worse, they have not learned what is necessary by doing the work it takes to learn a process, a historical fact or a method of progress they can use the rest of their lives.

In our world of increasing technology and mobility, we are able to access information wherever we may be or whenever we need it.  Because we have this ability, we can, if we want, take it a step further by cheating to access the information or stealing the information and then profiting from it by selling this ill-gotten information to others so they benefit as well.  But only in the moment!

One of the definitions in Webster’s on-line dictionary indicates that piracy is “the unauthorized use of another’s production, invention, or conception, especially in infringement of a copyright.”  The pirate’s sole purpose is to illegally copy these products for financial gain.  This piracy happens in the movie/music industry, book publishing, manufacturing, schools licensing exams, to name just a few.  Intellectual property, such as test questions are stolen and sold on the black market.  Students buy the questions in order to pass an exam and/or complete a class without doing the necessary work.  In addition, they may sell it for personal gain.

In one instance, a student had a pen with a camera.  As he was completing an exam, he took pictures of the test questions with his camera pen.  He was then able to take the questions home, typed them into his computer and sold them to other potential candidates who were scheduled for that particular exam.  By doing this, a specific license or exam is compromised.  Companies who have worked hard to establish a reputation of integrity and honesty are then questioned.  When a person or company becomes compromised, it impacts their ability to sell their products or services to the public.

As you can see, cheaters may win in the moment but what have they lost, or negatively impacted in the long run?  Their personal integrity, their reputation, they may lose their jobs, be expelled from school or cause serious monetary or personal harm in their chosen profession.  Cheaters ultimately lose.

Don’t miss our Webinar, “Catch Them If You Can” this wednesday, Feb 15th at 12 pm EST, featuring Faisel Alam of Law School Admissions Council.

Please leave your comments about this article in the space below!

Join the conversation on twitter @caveon or on linkedin on our Caveon Test Security group!

Sign up for this Webinar here!

Dr Fremer has been asked to speak about “detection” and the many types of activities that fall under the heading of “data forensics.”

For more information about this meeting and other events related to the CCSSO, click the following link.

“Insert Quarter, Avoid Klingons” is cited in Walter Issacson’s biography of Steve Jobs as the complete instructions for an electronic game that caught the attention of Jobs.  According to Issacson, Jobs was very taken by the notion that having a product that you could use right out of the box without complex directions. Consequently, he worked with extreme diligence to produce products with that characteristic.

None of us ordinary human beings are going to come even close to imitating very much of what Steve Jobs accomplished.  We can learn from Jobs, though, which is why I am reading the Issacson book.  The idea that it is important to make things simple for the clients of Caveon Test Security really strikes a powerful chord for me.  Personally, I almost never read instructions.  If your product can be explained in very few words or even better, be self-explanatory, you might sell me something even though I am a reluctant buyer.  I mistrust people who tell me something is simple.  If that person is a real expert, I hardly ever find that it is simple for me.

So how am I putting the “Insert Quarter, Avoid Klingons” approach into practice in my Caveon Test Security work?  I have begun asking what our work can do for clients or prospective clients.  How might it save them time?  How might it save them money?  How could it help them better serve their clients?  This client focus on simple, but powerful results, is very different from explaining the features of a service such as Caveon Data Forensics or Caveon Security Audit.  I have almost an automatic tendency to talk about what we do instead of what a client actually receives and how they might use it. I need to restrain that tendency and focus on what our services can do for clients.

In addition to client focus, I am spending more time listening to people talk about their job and organization, the challenges they face and the context within which they work.  When I do that, it leads me towards sentences like “maybe our Data Forensics service could help you focus on the cases that are most problematic and spend less time with ones that you will end up clearing anyway.”

I know that success in what I and  my colleagues are trying to do is occurring in only one part of our society.  It is an important area, though, and one that makes me want to draw on as many sources of assistance as I can.  I am delighted that reading about Steve Jobs is proving to be very fertile and productive.

Footnote:  if you don’t actually know about Klingons, I can put you in touch with my son who speaks halting Klingon.  My son was a passionate Star Trek fan, where the evil Klingons caused much mischief.

Today reports came out regarding the most recent senate hearing about the SAT proxy testing scandal in New York state. Several individuals from College Board, ACT and ETS, along with others from organizations with authentication methodologies and technologies, provided testimony at the hearing. Senator LaValle from New York expressed dismay at the apparent lax security surrounding these very important admissions exams and admonished the organizations to better protect the examination process so that the results could be trusted. The hearing focused on the taking of tests by others, and rightfully exposed the problems with using student ID’s, or even government-issued ID’s, as a method to authenticate a student prior to taking a college admissions exam. Several organizations described their products and services as “improved” methods of authentication.

One of the problems I see with the handling of this security breach, the extensive media attention regarding this breach, and even these hearings, is that they are myopic, ignoring other important security threats. (To be fair to the media, for the past couple of years there have been almost daily reports of other types of incidents of cheating in the US and across the world.) The implication is that solving this authentication problem will do away with the problem of cheating. While proxy test-taking is one way to cheat, there are dozens if not hundreds of others. Improving the authentication process will reduce the rate of proxy test-taking, but will do nothing for the other threats.

Before following this track too much farther, Senator LaValle and the organizations involved in this scandal should step back and help capture the big picture of cheating. What are the other threats? What risks do they pose and what damage have they caused and continue to cause to the value of testing, to society and to our educational system? How do we get a better measure of what is really happening? How do we deal with each threat individually? What can we do to impact the general cheating disease currently afflicting our country and the world? While I appreciate what the Senator is doing to try to correct this one problem, we need a more comprehensive approach. Getting a better view of the big cheating picture—as a starting point—should be our top priority.

My boss may be upset with this essay, because if I persuade you to quit cheating on tests Caveon could go out of business in the same way that we would have less need for police if crooks would quit law-breaking. But, I suspect many of you that cheat will keep at it, despite anything I say. For those of you who have cheated, but are willing to consider changing, here are ten reasons why you should stop cheating on tests and in other areas in your life.

Number 1. (Unpopularity) People might not like you anymore. Cheating is very anti-social and people, especially successful ones, don’t care to associate with cheating or cheaters.

Number 2. (Disrespect) You might get a bad reputation. When others hear about your cheating, their opinion of you will go down. Not only that, when you apply for a job and the hiring manager asks about your work ethic, you probably will not like the information that is passed along by those who know you.

Number 3. (Addiction) You might not be able to quit cheating easily. The thought of doing it “just this once because I have to” can become an ingrained habit. Just like gambling, infidelity, stealing, and lying, cheating can be compulsive and even addictive. When this happens, research suggests that you will start cheating in other areas of your life.

Number 4. (Failure) Your laziness might contribute to your failure. Success takes hard work, persistence, dedication, diligence, and willingness to sacrifice. Cheating and trying to find the easy way out just don’t help you learn and develop these traits. If you persist in laziness (and cheating) you will continue to find it hard to attain your goals.

Number 5. (Unemployment) You might not be able to keep a job. Your boss won’t appreciate someone with a poor work ethic who takes credit for the work of others. Your coworkers won’t appreciate the lack of respect that cheating reflects. And, you might have cheated yourself out of gaining the knowledge that is required to perform well on your job.

Number 6. (Unpleasant Consequences) You might face some tough discipline. Current trends suggest that members of “the establishment” (i.e., those who are in charge of making things happen) are getting fed up with cheating. Lawmakers are passing tougher anti-cheating laws. College admissions officers are being informed whether you cheated. And, you might be blacklisted from the profession of your choice, because you were caught cheating.

Number 7. (Loss of Perspective) You might lose perspective as to what is acceptable behavior. When you cheat, you show a blatant disregard for others. It’s not acceptable to cut in line. It’s not acceptable to lie. And, it’s not acceptable to accept a grade or promotion that was gained through cheating and not an honest effort.

Number 8. (Becoming a Major Liar) Your cheating may require you to lie and steal. In other words, to cover up your cheating you may find it necessary to come up with a story, which isn’t true. The problem with lying is that you can never keep your story straight, because it never happened. This can lead to being trapped by a web of lies.

Number 9.  (Loss of Self Respect) You might lose self-respect. If you have enough bad experiences as a result of your cheating, you might realize that you have brought these things upon yourself by willful and wanton behavior. In this situation, when you awake to your awful situation you will see that you are a chump and not a champ.

Number 10. (Embarrassment) Cheating is a reflection of who you are and who you want to be. Cheating is a tacit admission of incompetence, laziness, selfishness, arrogance, and disrespect. If you want to see yourself as others see you and as you really are, you should admit that your cheating has hurt others, including yourself and those who care about you.  And, then you should change your cheating ways.

The internet is the greatest knowledge sharing and communication innovation in our history.  It is pervasive and changing constantly, at electron speed. As the tools and technologies that transform our work and communication evolve, so must the manner in which we use them.  The internet of 2003 when we founded Caveon is VERY different than the internet of 2012.  This evolution is manifested in how we use the web, how it empowers us, and unfortunately, increased risks to our testing programs.

At Caveon, part of our business is built upon consistently and continually trolling the web for risks to our client’s test programs.  Our Web Patrollers spend time combing through the internet’s darker shadows, searching for places where miscreants sell and distribute copyrighted test material.    Other parts of the day are spent on the cheaters’ shopping list, seeking great deals on stolen test content, a rogue review prep course, or the services of a willing, capable proxy to take a test.  More and more, though, they spend time lurking in various online conversations, carried on by candidates and students seeking help passing exams and sharing test questions.

Early on, the threatening websites we encountered were similar to other websites one might visit.  “Braindump” websites are just like any other for-profit, ecommerce website.  They typically offer many  “test preparation” products, appealing to as broad a consumer base as possible.  Many of these products are copies of pilfered exams. Regardless of what test you may need to pass, these one-stop shopping sites likely have what you need.  Often, their sites contain hundreds of important tests, and they boast that passing is “guaranteed.”  No hollow promise considering that most braindumps are selling live test items.

Over time, powerful marketing techniques focused on a “Customer of One” have emerged.  These new technologies capture web surfing and purchasing tendencies, enabling online purveyors of stolen exam content to present unique offerings that might better catch the attention and whimsy of those who are seeking an “edge” in passing their exams. Transactions in pirated exam content happen with satellite speed, connecting buyers and sellers globally.

Today, I can effortlessly find a proxy to help me complete any academic assignment imaginable, or find a hired gun to sit for any high stakes exam in any corner of the world.  If one has money to spend, the internet empowers us to find just the right vendor to satisfy any requirement.

The internet is continually evolving. The web’s pervasiveness coupled with amazing new applications have allowed communities of like minded individuals to grow and connect.  Social Media allows online communities to gather and exchange content in creative ways.  The incredible phenomenon of Social Media is, literally, changing how we work, play, and of course, gain unfair advantage on important exams.

It used to be that our web patrollers primarily found and analyzed websites that were selling test items or soliciting proxy services.  Today, they search out the online “watering holes” where students and candidates gather to exchange ideas. For many of the members of these online communities–young people who have grown up as “digital natives” (as opposed to we Baby Boomers and GenXers, considered “digital immigrants”)—sharing is a core tenet.  Indeed, discussing the test items I just encountered in my exam isn’t cheating, it’s sharing and contributing to the greater good of the community.

Innovation is the internet’s ecology.  Technology advancements will continue to alter and shape our ability to create content and communicate it broadly.  As testing industry professionals, it is our duty to remain vigilant and monitor how these innovations may impact our ability to ensure fair and valid test results.

Look for a dedicated Web Patrol Blog to be released soon! Please join our Caveon Test Security Group on LinkedIn or follow us on twitter!

Lose some weight? Stop smoking? Be more patient?  These appear on many lists of New Year’s resolutions, but they were not proposed when I asked my Caveon colleagues what we should commit to doing as a company to increase the value of our services to clients.  The five ideas that seemed most promising to me – actionable, practical, and likely to matter to our clients, as best I could judge, are presented below.

1. Become Better Listeners – “Shut up and listen” – something my wife has said to me more than once is the proposal that rang truest to me.  Stop focusing on what we feel we “must” say and instead allow our clients to describe their situations and needs.  Show that we are really paying attention by asking for clarification of the issues they raise.  What is the challenge facing them?  What are they are doing to cope?  Have they heard of any new approaches that companies in their network have tried? We will gain a better appreciation of their world and, if we are asked for our perspective, will be more likely to address our client’s interests rather than our own.

2. Develop Clearer Explanations – If people do not understand fully something we explain, the fault is with us not them.  We live Data Forensics and Security Audits on an ongoing basis and are in our tenth year of doing this.  To many of our clients and prospective clients, they are very new to the idea of a rigorous external review against 250 security standards (Caveon Test Security Audit) or carrying out seven different types of analyses of test data to help spot and quantify testing misbehaviors (Caveon Data Forensics).  Which of our efforts to describe what we do have been best received?  What questions did we receive? Have we asked our clients what would help them better understand what we provide?  It makes sense for us to keep improving our methods, but we also need to keep getting better at describing what we do and its value to clients in terms that they find useful.

3. Help Make the Financial Case – Does it make sense financially to use a Caveon service?  How can we help clients make this determination?  They can determine the costs of developing items, pools, and tests.  What are the costs of security breaches of different types?  How do the costs of preventive actions compare with the benefits that can be counted on?  In an environment where all expenditures need to be justified and where there may well be significant competing demands, how can we help clients figure their return on possible security investments?  During the many years that I worked as an administrator of test developers, I had a sign in my office “There is never enough time to do it right, but there is always time to do it over!” There is a real parallel in test security, where the same agency that felt that it could not justify increased preventative or early detection security methods later spends huge amounts on an emergency basis when the “unthinkable” has clearly happened and is now public knowledge.

4. Hone Our Investigative Skills – Clients are asking for our help in planning and conducting security investigations and we need to build our skills to deliver what is being sought.  In some instances, it is building easy to implement models that our clients can employ with their own staff.  To get better at this work, we need to follow up with the agencies that we have already helped.  How have our recommended approaches worked?  What was good, what not so good?  How can we do a better job going forward?  One of the things I have liked most about working in testing is that we truly believe in studying our work and in continuous improvement.  We want to learn from each test administration lessons we can apply to our future work.  We need to apply this model to our investigations whether we serve as the investigators or behind the scenes helpers.  Once again listening to our clients will be one of the best ways to build our ability to help. (I also find it helpful to observe how interrogations are handled on NCIS, Law and Order, etc., but I will hold my discussion of this to a future posting.)

5. Learning What We Did Not Learn in Kindergarten  (About Social Media) – We need to get very good at using social media to assist fellow testing professionals who are fighting fiercely to maintain the fairness and validity of their tests.  I pay a great deal of attention to the advice of Jeffrey Gitomer who writes frequently about how to increase the value of what you provide as a company.  Gitomer is not one to mince words and he says “Use Social Media to Stay Close to Your Clients or Get Out of the Way as Your Day is Ending.”  This is what we are saying to each other at Caveon and we would value your feedback on our activities.  For example, do you find these resolutions promising?  Are there other changes you would like to see us work on?  I am so open to hearing your ideas.   I promise to really listen.

Thanks for reading. We look forward to a great 2012!