Cheaters never win and winners never cheat.  This is an old saying that may sound confusing because cheaters may win in the moment.  However, in the long run, they may feel embarrassed, guilty, feel they have let themselves or others down; and, worse, they have not learned what is necessary by doing the work it takes to learn a process, a historical fact or a method of progress they can use the rest of their lives.

In our world of increasing technology and mobility, we are able to access information wherever we may be or whenever we need it.  Because we have this ability, we can, if we want, take it a step further by cheating to access the information or stealing the information and then profiting from it by selling this ill-gotten information to others so they benefit as well.  But only in the moment!

One of the definitions in Webster’s on-line dictionary indicates that piracy is “the unauthorized use of another’s production, invention, or conception, especially in infringement of a copyright.”  The pirate’s sole purpose is to illegally copy these products for financial gain.  This piracy happens in the movie/music industry, book publishing, manufacturing, schools licensing exams, to name just a few.  Intellectual property, such as test questions are stolen and sold on the black market.  Students buy the questions in order to pass an exam and/or complete a class without doing the necessary work.  In addition, they may sell it for personal gain.

In one instance, a student had a pen with a camera.  As he was completing an exam, he took pictures of the test questions with his camera pen.  He was then able to take the questions home, typed them into his computer and sold them to other potential candidates who were scheduled for that particular exam.  By doing this, a specific license or exam is compromised.  Companies who have worked hard to establish a reputation of integrity and honesty are then questioned.  When a person or company becomes compromised, it impacts their ability to sell their products or services to the public.

As you can see, cheaters may win in the moment but what have they lost, or negatively impacted in the long run?  Their personal integrity, their reputation, they may lose their jobs, be expelled from school or cause serious monetary or personal harm in their chosen profession.  Cheaters ultimately lose.

Don’t miss our Webinar, “Catch Them If You Can” this wednesday, Feb 15th at 12 pm EST, featuring Faisel Alam of Law School Admissions Council.

Please leave your comments about this article in the space below!

Join the conversation on twitter @caveon or on linkedin on our Caveon Test Security group!

Sign up for this Webinar here!

Dr Fremer has been asked to speak about “detection” and the many types of activities that fall under the heading of “data forensics.”

For more information about this meeting and other events related to the CCSSO, click the following link.

“Insert Quarter, Avoid Klingons” is cited in Walter Issacson’s biography of Steve Jobs as the complete instructions for an electronic game that caught the attention of Jobs.  According to Issacson, Jobs was very taken by the notion that having a product that you could use right out of the box without complex directions. Consequently, he worked with extreme diligence to produce products with that characteristic.

None of us ordinary human beings are going to come even close to imitating very much of what Steve Jobs accomplished.  We can learn from Jobs, though, which is why I am reading the Issacson book.  The idea that it is important to make things simple for the clients of Caveon Test Security really strikes a powerful chord for me.  Personally, I almost never read instructions.  If your product can be explained in very few words or even better, be self-explanatory, you might sell me something even though I am a reluctant buyer.  I mistrust people who tell me something is simple.  If that person is a real expert, I hardly ever find that it is simple for me.

So how am I putting the “Insert Quarter, Avoid Klingons” approach into practice in my Caveon Test Security work?  I have begun asking what our work can do for clients or prospective clients.  How might it save them time?  How might it save them money?  How could it help them better serve their clients?  This client focus on simple, but powerful results, is very different from explaining the features of a service such as Caveon Data Forensics or Caveon Security Audit.  I have almost an automatic tendency to talk about what we do instead of what a client actually receives and how they might use it. I need to restrain that tendency and focus on what our services can do for clients.

In addition to client focus, I am spending more time listening to people talk about their job and organization, the challenges they face and the context within which they work.  When I do that, it leads me towards sentences like “maybe our Data Forensics service could help you focus on the cases that are most problematic and spend less time with ones that you will end up clearing anyway.”

I know that success in what I and  my colleagues are trying to do is occurring in only one part of our society.  It is an important area, though, and one that makes me want to draw on as many sources of assistance as I can.  I am delighted that reading about Steve Jobs is proving to be very fertile and productive.

Footnote:  if you don’t actually know about Klingons, I can put you in touch with my son who speaks halting Klingon.  My son was a passionate Star Trek fan, where the evil Klingons caused much mischief.

Today reports came out regarding the most recent senate hearing about the SAT proxy testing scandal in New York state. Several individuals from College Board, ACT and ETS, along with others from organizations with authentication methodologies and technologies, provided testimony at the hearing. Senator LaValle from New York expressed dismay at the apparent lax security surrounding these very important admissions exams and admonished the organizations to better protect the examination process so that the results could be trusted. The hearing focused on the taking of tests by others, and rightfully exposed the problems with using student ID’s, or even government-issued ID’s, as a method to authenticate a student prior to taking a college admissions exam. Several organizations described their products and services as “improved” methods of authentication.

One of the problems I see with the handling of this security breach, the extensive media attention regarding this breach, and even these hearings, is that they are myopic, ignoring other important security threats. (To be fair to the media, for the past couple of years there have been almost daily reports of other types of incidents of cheating in the US and across the world.) The implication is that solving this authentication problem will do away with the problem of cheating. While proxy test-taking is one way to cheat, there are dozens if not hundreds of others. Improving the authentication process will reduce the rate of proxy test-taking, but will do nothing for the other threats.

Before following this track too much farther, Senator LaValle and the organizations involved in this scandal should step back and help capture the big picture of cheating. What are the other threats? What risks do they pose and what damage have they caused and continue to cause to the value of testing, to society and to our educational system? How do we get a better measure of what is really happening? How do we deal with each threat individually? What can we do to impact the general cheating disease currently afflicting our country and the world? While I appreciate what the Senator is doing to try to correct this one problem, we need a more comprehensive approach. Getting a better view of the big cheating picture—as a starting point—should be our top priority.

My boss may be upset with this essay, because if I persuade you to quit cheating on tests Caveon could go out of business in the same way that we would have less need for police if crooks would quit law-breaking. But, I suspect many of you that cheat will keep at it, despite anything I say. For those of you who have cheated, but are willing to consider changing, here are ten reasons why you should stop cheating on tests and in other areas in your life.

Number 1. (Unpopularity) People might not like you anymore. Cheating is very anti-social and people, especially successful ones, don’t care to associate with cheating or cheaters.

Number 2. (Disrespect) You might get a bad reputation. When others hear about your cheating, their opinion of you will go down. Not only that, when you apply for a job and the hiring manager asks about your work ethic, you probably will not like the information that is passed along by those who know you.

Number 3. (Addiction) You might not be able to quit cheating easily. The thought of doing it “just this once because I have to” can become an ingrained habit. Just like gambling, infidelity, stealing, and lying, cheating can be compulsive and even addictive. When this happens, research suggests that you will start cheating in other areas of your life.

Number 4. (Failure) Your laziness might contribute to your failure. Success takes hard work, persistence, dedication, diligence, and willingness to sacrifice. Cheating and trying to find the easy way out just don’t help you learn and develop these traits. If you persist in laziness (and cheating) you will continue to find it hard to attain your goals.

Number 5. (Unemployment) You might not be able to keep a job. Your boss won’t appreciate someone with a poor work ethic who takes credit for the work of others. Your coworkers won’t appreciate the lack of respect that cheating reflects. And, you might have cheated yourself out of gaining the knowledge that is required to perform well on your job.

Number 6. (Unpleasant Consequences) You might face some tough discipline. Current trends suggest that members of “the establishment” (i.e., those who are in charge of making things happen) are getting fed up with cheating. Lawmakers are passing tougher anti-cheating laws. College admissions officers are being informed whether you cheated. And, you might be blacklisted from the profession of your choice, because you were caught cheating.

Number 7. (Loss of Perspective) You might lose perspective as to what is acceptable behavior. When you cheat, you show a blatant disregard for others. It’s not acceptable to cut in line. It’s not acceptable to lie. And, it’s not acceptable to accept a grade or promotion that was gained through cheating and not an honest effort.

Number 8. (Becoming a Major Liar) Your cheating may require you to lie and steal. In other words, to cover up your cheating you may find it necessary to come up with a story, which isn’t true. The problem with lying is that you can never keep your story straight, because it never happened. This can lead to being trapped by a web of lies.

Number 9.  (Loss of Self Respect) You might lose self-respect. If you have enough bad experiences as a result of your cheating, you might realize that you have brought these things upon yourself by willful and wanton behavior. In this situation, when you awake to your awful situation you will see that you are a chump and not a champ.

Number 10. (Embarrassment) Cheating is a reflection of who you are and who you want to be. Cheating is a tacit admission of incompetence, laziness, selfishness, arrogance, and disrespect. If you want to see yourself as others see you and as you really are, you should admit that your cheating has hurt others, including yourself and those who care about you.  And, then you should change your cheating ways.

The internet is the greatest knowledge sharing and communication innovation in our history.  It is pervasive and changing constantly, at electron speed. As the tools and technologies that transform our work and communication evolve, so must the manner in which we use them.  The internet of 2003 when we founded Caveon is VERY different than the internet of 2012.  This evolution is manifested in how we use the web, how it empowers us, and unfortunately, increased risks to our testing programs.

At Caveon, part of our business is built upon consistently and continually trolling the web for risks to our client’s test programs.  Our Web Patrollers spend time combing through the internet’s darker shadows, searching for places where miscreants sell and distribute copyrighted test material.    Other parts of the day are spent on the cheaters’ shopping list, seeking great deals on stolen test content, a rogue review prep course, or the services of a willing, capable proxy to take a test.  More and more, though, they spend time lurking in various online conversations, carried on by candidates and students seeking help passing exams and sharing test questions.

Early on, the threatening websites we encountered were similar to other websites one might visit.  “Braindump” websites are just like any other for-profit, ecommerce website.  They typically offer many  “test preparation” products, appealing to as broad a consumer base as possible.  Many of these products are copies of pilfered exams. Regardless of what test you may need to pass, these one-stop shopping sites likely have what you need.  Often, their sites contain hundreds of important tests, and they boast that passing is “guaranteed.”  No hollow promise considering that most braindumps are selling live test items.

Over time, powerful marketing techniques focused on a “Customer of One” have emerged.  These new technologies capture web surfing and purchasing tendencies, enabling online purveyors of stolen exam content to present unique offerings that might better catch the attention and whimsy of those who are seeking an “edge” in passing their exams. Transactions in pirated exam content happen with satellite speed, connecting buyers and sellers globally.

Today, I can effortlessly find a proxy to help me complete any academic assignment imaginable, or find a hired gun to sit for any high stakes exam in any corner of the world.  If one has money to spend, the internet empowers us to find just the right vendor to satisfy any requirement.

The internet is continually evolving. The web’s pervasiveness coupled with amazing new applications have allowed communities of like minded individuals to grow and connect.  Social Media allows online communities to gather and exchange content in creative ways.  The incredible phenomenon of Social Media is, literally, changing how we work, play, and of course, gain unfair advantage on important exams.

It used to be that our web patrollers primarily found and analyzed websites that were selling test items or soliciting proxy services.  Today, they search out the online “watering holes” where students and candidates gather to exchange ideas. For many of the members of these online communities–young people who have grown up as “digital natives” (as opposed to we Baby Boomers and GenXers, considered “digital immigrants”)—sharing is a core tenet.  Indeed, discussing the test items I just encountered in my exam isn’t cheating, it’s sharing and contributing to the greater good of the community.

Innovation is the internet’s ecology.  Technology advancements will continue to alter and shape our ability to create content and communicate it broadly.  As testing industry professionals, it is our duty to remain vigilant and monitor how these innovations may impact our ability to ensure fair and valid test results.

Look for a dedicated Web Patrol Blog to be released soon! Please join our Caveon Test Security Group on LinkedIn or follow us on twitter!

Lose some weight? Stop smoking? Be more patient?  These appear on many lists of New Year’s resolutions, but they were not proposed when I asked my Caveon colleagues what we should commit to doing as a company to increase the value of our services to clients.  The five ideas that seemed most promising to me – actionable, practical, and likely to matter to our clients, as best I could judge, are presented below.

1. Become Better Listeners – “Shut up and listen” – something my wife has said to me more than once is the proposal that rang truest to me.  Stop focusing on what we feel we “must” say and instead allow our clients to describe their situations and needs.  Show that we are really paying attention by asking for clarification of the issues they raise.  What is the challenge facing them?  What are they are doing to cope?  Have they heard of any new approaches that companies in their network have tried? We will gain a better appreciation of their world and, if we are asked for our perspective, will be more likely to address our client’s interests rather than our own.

2. Develop Clearer Explanations – If people do not understand fully something we explain, the fault is with us not them.  We live Data Forensics and Security Audits on an ongoing basis and are in our tenth year of doing this.  To many of our clients and prospective clients, they are very new to the idea of a rigorous external review against 250 security standards (Caveon Test Security Audit) or carrying out seven different types of analyses of test data to help spot and quantify testing misbehaviors (Caveon Data Forensics).  Which of our efforts to describe what we do have been best received?  What questions did we receive? Have we asked our clients what would help them better understand what we provide?  It makes sense for us to keep improving our methods, but we also need to keep getting better at describing what we do and its value to clients in terms that they find useful.

3. Help Make the Financial Case – Does it make sense financially to use a Caveon service?  How can we help clients make this determination?  They can determine the costs of developing items, pools, and tests.  What are the costs of security breaches of different types?  How do the costs of preventive actions compare with the benefits that can be counted on?  In an environment where all expenditures need to be justified and where there may well be significant competing demands, how can we help clients figure their return on possible security investments?  During the many years that I worked as an administrator of test developers, I had a sign in my office “There is never enough time to do it right, but there is always time to do it over!” There is a real parallel in test security, where the same agency that felt that it could not justify increased preventative or early detection security methods later spends huge amounts on an emergency basis when the “unthinkable” has clearly happened and is now public knowledge.

4. Hone Our Investigative Skills – Clients are asking for our help in planning and conducting security investigations and we need to build our skills to deliver what is being sought.  In some instances, it is building easy to implement models that our clients can employ with their own staff.  To get better at this work, we need to follow up with the agencies that we have already helped.  How have our recommended approaches worked?  What was good, what not so good?  How can we do a better job going forward?  One of the things I have liked most about working in testing is that we truly believe in studying our work and in continuous improvement.  We want to learn from each test administration lessons we can apply to our future work.  We need to apply this model to our investigations whether we serve as the investigators or behind the scenes helpers.  Once again listening to our clients will be one of the best ways to build our ability to help. (I also find it helpful to observe how interrogations are handled on NCIS, Law and Order, etc., but I will hold my discussion of this to a future posting.)

5. Learning What We Did Not Learn in Kindergarten  (About Social Media) – We need to get very good at using social media to assist fellow testing professionals who are fighting fiercely to maintain the fairness and validity of their tests.  I pay a great deal of attention to the advice of Jeffrey Gitomer who writes frequently about how to increase the value of what you provide as a company.  Gitomer is not one to mince words and he says “Use Social Media to Stay Close to Your Clients or Get Out of the Way as Your Day is Ending.”  This is what we are saying to each other at Caveon and we would value your feedback on our activities.  For example, do you find these resolutions promising?  Are there other changes you would like to see us work on?  I am so open to hearing your ideas.   I promise to really listen.

Thanks for reading. We look forward to a great 2012!

I recently attended the SLOAN-C international conference in Orlando, Florida. The SLOAN-C conference serves distance education organizations, covering the many issues and new technologies for this fast growing higher education alternative. There were a few sessions dealing with assessment and testing, and I attended those that I could. Authentication of students was a main assessment topic, an issue at the top of the list for these organizations. The recent federal government Higher Education Opportunity Act (HEOA) now requires that distance education institutions assure that the student taking a test is actually the student enrolled in the program. In a couple of the sessions at the conference, different authentication models were compared ranging from the simple use of usernames and passwords to using biometric methods such as keystroke analytics and devices such as fingerprint readers. The better programs will choose authentication methods that have a high degree of accuracy, doing their best to make sure that tests are taken by the right student.

But the HEOA doesn’t go far enough. For example, it doesn’t require that a distance education program monitor a student who is taking an exam to make sure that he or she isn’t cheating. The HEOA does not require that programs protect their test content from theft and sharing. Requiring authentication is a good first step, but it’s like requiring a bank to lock the front door with a simple padlock, but being okay if the vault is left open. Some distance education programs are not at all interested in proper authentication or in the security of their exams, but are simply interested in how many students they can get enrolled and paying tuition. That’s probably a small percentage of such universities; most programs I’ve come to know care about the education of their students and in providing fair and competent assessment.

Distance education programs are not comfortable with the difficulties and costs associated with students having to travel to testing centers or campus-based locations to take tests. Their instructional models are all universally based on online learning—allowing students to learn at their convenience and in the comfort of their own homes or workplaces. Tests for distance education courses should be securely administered in these places as well. New test administration models were presented at the SLOAN-C conference as well, including models that allow remote proctors to use webcams and other technologies to monitor students taking exams. Some of these models actually improve on the ability to proctor an exam by making sure that proctors are trained and properly motivated and have tools to help them. It’s an exciting time for distance education universities as they are able to provide online testing that accurately and confidently measures what their students are learning.

How do individuals react when a question is raised about the possibility that testing rules were violated?  Most responses can be divided, like “all Gaul” if you have ever had high school Latin, into one of three categories:

• I am innocent! (or my son, daughter, teacher, or principal is innocent)

• How do you know there was misbehavior?

• We should “kill the messenger”

Your Comments Below Are Greatly Appreciated!

After so many years working for a company that has built our business by helping protect important test programs from all kinds of security threats, nothing should surprise me anymore.  However, the last 24 hours have presented a series of interactions and discoveries that are, at the very least, troubling, leaving me shaking my head.   They raise serious questions in my mind regarding our ability to realize fair and valid test results.

Deep down, I’m an optimist, and I generally believe in the goodness and fairness of people. And, I know from the work we conduct that the percentage of cheaters on tests truly is small (though growing).  Despite that, as recent events illuminate, the impact of the malfeasance is still all too large. 

Yesterday’s concerns took seed as I gathered details to scope a special investigative data forensic analysis for a state department of education.  Turns out the state seeks to gather evidence regarding allegations against the principal of an elementary school.  A “whistle-blower” teacher in the school provided details that the principal forced teachers to systematically cheat by walking through test booklets with kids before filling in answer  sheets.  If the allegations are corroborated, it’s staggering to consider the impact one bad apple, the principal, has had on so many students, parents, and teachers. 

A bit later, during a web-based presentation to over 100 school district leaders in a different state, one participant innocently asked, “How could two students in opposite ends of the test center collude together and cheat?”  During the ensuing discussion, a colleague passed along a URL for a web site that overtly markets a technology product to help students cheat.  The system, an “invisible spy earpiece” incorporates Bluetooth technology, hidden earpieces, signal buttons in shoes, and a cell phone to allow the very-covert use of a cell phone during a test to communicate with others.  The purveyors make it absolutely clear that students can use it to  thwart the security of any important test administration.  Take a look here, and if you have a few minutes, please watch the audacious video.  I’ll be surprised if you come away any less disturbed than I did:  www.spycheatstuff.com

This morning, I participated in a weekly conference call regarding the ATP Security Council’s soon-to-be-published survey report.  A fellow volunteer had just compiled survey results provided by twenty two test publishers concerning the financial impact of test fraud upon their programs.  While we all know that cheats cost our programs money, it is not always clear on how expensive this impact truly is.  Based upon the information provided by respondents, these programs are contending with losses that exceed a whopping $60,000,000.  That’s the low end.  The high end calculation is, obviously, even greater, and even more alarming.  For all the details, look for the ATP Security Survey Report when it is published next month, January 2012.

Just when I think all the news is bad, a glimmer of hope.  Later this morning, I spoke to a test program manager whose organization recently won a major court battle with a “Rogue Review Course.”  The operators of the course had been systematically pirating live test items and selling them as legitimate test prep materials.  According to my contact, the case took over a year to prosecute, but in the end the good guys were on top, recouping their test development costs and legal fees.  The operators of the course have even taken the steps necessary to become a legitimate test preparation partner.  As my contact stated, it makes sense to “keep their friends close, and their enemies closer.”

Clearly, this is an important win for our industry.  Consider, though, the time, conviction, and resources (both human and financial) required to wage such a effort.   With the stakes so high for so many programs, it’s evident that extraordinary efforts like this one will be required to protect our programs. 

Thanks for Reading! Feel free to leave Comments below and/or find us on LinkedIn to discuss more topics like this one

Recently, the Huffington Post reported that taxpayers in Georgia would probably pay more than $9 million to deal with the cheating scandal in Atlanta Public Schools (APS) (November 22, 2011: Atlanta Public Schools Cheating Scandal Could Cost Taxpayers $9 Million,    http://www.huffingtonpost.com/2011/11/21/atlanta-public-schools-ch_n_1105228.html). This is a tangible amount that can be tracked and measured. According to the special investigators’ report, 182 educators were involved in the scandal from 44 schools. That places the cost at $50,000 per educator who was alleged to have acted inappropriately.

The actual costs that cannot be tracked using conventional accounting methods seem to be much higher. We were not told how many students were cheated out of their education, but it is on the order of 13,000 to 14,000 students. Because many of these students were not prepared for the next grade, it is highly likely that their graduation rates from high school will be reduced. Let’s assume that 2,000 students will not graduate, as a result of the cheating.  It is also highly likely that entrance rates into colleges and vocational training will be reduced. Let’s assume that 2,000 more students will graduate but will not receive post-high school training or education, as a result of the cheating. From the National Center of Statistics, we have the following information: “In 2009, the median of the earnings for young adults with a bachelor's degree was $45,000, while the median was $21,000 for those without a high school diploma or its equivalent, $30,000 for those with a high school diploma or its equivalent, and $36,000 for those with an associate's degree.” (http://nces.ed.gov/fastfacts/display.asp?id=77). Thus, in the first year these 4,000 students will have an approximate income shortage of $9,000 for those not graduating and $6,000 for those not receiving additional training. This represents $30 million per year in lost wages.

In addition to lost future wages, there are costs associated with being undereducated. The most notable cost is a correlation between lack of education and incarceration rates. A study in this area is “The Effect of Education on Crime: Evidence from Prison Inmates, Arrests, and Self-Reports” by Lochner and Moretti (2003, http://elsa.berkeley.edu/~moretti/lm46.pdf). The authors suggest that the social savings from crime reduction associated with high school graduation among men is approximately a 14-26% return. The lost return represents a significant cost to society and especially in Georgia where the education loss due to cheating by educators will be felt most heavily. The estimated impact of increased crime associated with non-completion of the high school diploma is $2,000 per year. Or, as much as $4 million per year associated with these students.

Finally, the costs of misdirected efforts were tremendous. The APS school board degenerated for a time into a dysfunctional governing unit. The Georgia Bureau of Investigation allocated significant resources which were needed elsewhere. The APS has significant costs associated with hiring and training new teachers. Other losses were lost credibility to the school system, significant lost reputation of the falsely accused schools (at least 14 of the 58 schools originally accused were cleared), and the threatened withdrawal of APS’ accreditation. To be sure, the dollar figure reported by the Huffington Post is large, but it appears to be the proverbial “tip of the iceberg” that citizens of Georgia will pay.

On January 7, 2011, the Dr. Phil Show aired an interesting interview with Jolie Fitch. Ms. Fitch was an organizer of the cheating effort that came to be known as the “Steinmetz Academic Decathlon Scandal.” You can see excerpts from the interview at the Whatever It Takes web page. I was as amazed Dr. Phil was at Ms. Fitch’s answers. Some of the questions and answers were:

- Is there any doubt that you cheated? "No"

- You knew you were cheating when you cheated? "Yes"

- Are you sorry about this? "No, not really. Other than getting caught"

- What do you tell your kids about cheating? "I tell them that life is a gray area"

The attitude that cheating is ok, as long as you don’t get caught seems to be accepted by many. Cheaters seem to justify their actions with statements like: “Anything goes,” “Others are doing it,” “If I don’t cheat, I can’t pass,” etc. When confronted with cheating, those involved deny or require “proof.” It is very difficult to prove cheating. There is no physical evidence of loss or harm. And, even if proved, there are very few laws against cheating. If the cheating is caught on camera, the evidence may be inconclusive. Even actual observation of the misbehavior by a proctor may be contested. Because of this, attempting to gather proof may play into the cheater’s hands.

Testing program managers need to realize that chasing and catching cheaters is not optimal for ensuring test security and exam integrity. Instead, a different approach is required. That approach is based on the propositions that:

- The testing examiner has the responsibility to certify test results, and

- A test is only a tool used by the testing examiner and is not the final authority.

By doing this, we are in a position to take the high ground against cheaters. We do not require their moral acceptance of our system of ethics. We do not need to “prove” cheating. We do not need to obtain confessions. In fact, the testing examiner may invalidate test results for many reasons other than suspected cheating.

Exam integrity is strengthened when each and every test result is scrutinized to ensure that it properly measured the test taker’s competence or knowledge. If the test score cannot be trusted, for whatever reason, it should be set aside. Cheaters have no defense against this, because the testing program manager is faithfully acting according to his or her responsibility.

Be Sure to Register for Caveon's Webinar on Nov 15th "International Testing: Crossing Global Borders While Protecting Your Own"

Following my presentation, we had a very productive and lively discussion of test security issues and challenges with the state representatives in TILSA.  I would be pleased to provide additional information about any or all of these recommendations.  Just send me an Email or give me a call.  If you have substantial interest, we can perhaps arrange a face-to-face meeting.  I am enough of an “old fashioned guy” to place a great deal of importance on being able to see the person that I am listening to, especially on matters of high importance to both of us.

*************

Thanks for reading. Register for Caveon's Webinar on November 15: "International Testing"

*************

Assange claims the blockade is illegal and has filed anti-trust lawsuits against Visa and Master Card. On the day before the blockade, WikiLeaks received $135,000. Currently, WikiLeaks receives less than $10,000 per month. The net effect of the blockade to WikiLeaks has been the loss of 95% of its operating cash.

Whether you agree with WikiLeaks’ goals or not, it is clear that WikiLeaks has routinely infringed upon the rights of copyright holders by distributing information and documents without authorization. If it is not obvious why this story has important test security ramifications, let me make it clear: (1) many websites, operated by pirates and thieves, infringe upon the copyrights of secured exam content, (2) it has been very difficult to effectively shutdown this activity, which is costing testing organizations millions of dollars per year in lost test development expenditures, and (3) if payment processors would agree to cease providing services to these thieves and pirates, many of them would fold. The WikiLeaks story demonstrates that copyright infringers will have a difficult time remaining in business without the support of payment processors.

At Caveon, we have been very successful in removing copyrighted exam materials from the Internet. Often our success is based upon respectful and courteous requests to unintentional copyright infringers. However, respect and courtesy do not work against pirates and thieves. At that point, potentially expensive legal action must be commenced.

An alternative to expensive legal proceedings is to work with payment processors to protect their brands. For example, Visa does not want any transaction to bring disrepute upon its brand (source: http://corporate.visa.com/_media/visa-international-operating-regulations.pdf). If we, as an industry, can convince the payment processors that the sale and distribution of pilfered exam content is disreputable, we may be able to slow the flow of money to the test thieves and protect valuable exam content.

What do you think? How can we help payment processors understand that their services facilitate the distribution of stolen exam content? Should ATP (Association of Test Publishers) contact the payment processors, on behalf of its members?

Several months ago, Ben Mannes, Test Security Director at ABIM, expressed this thought: “ATP should be trying to get a meeting with Victoria Espinel [White House intellectual property czar], bring 1-2 industry security experts, and state the case as to why exam content is a vital component to our nation’s infrastructure requiring heightened public sector IP enforcement.”

***************

Please Comment Below, Thank you for Reading

Fast forward to 2011, and it’s gratifying to consider what this entrepreneurial group of test security zealots has accomplished. Since that fateful October day, we have conducted over 50 Security Audits of leading test organizations and vendors, flagged and removed tens of thousands of internet-based risks, and conducted statistical analyses of over 30,000,000 test instances for many of the largest, most important test programs in the world.

As I consider the number and breadth of these engagements, perhaps it is worth sharing a few of the core values under which we always operate:

Confidentiality

Throughout our years of operation, one fundamental operating principle has always applied: client confidentiality. We never reveal the details of our client engagements without the express approval of our clients. Our clients require and appreciate this sensitivity as we investigate security incidents and provide reports on our forensic analyses. This is not secrecy– this privacy stems from respect for our clients and for the right to privacy of individuals and organizations.

Innovation

We constantly strive to improve means and methods for strengthening exam security. We are always interested in sharing the nature of our work. Not only do we share our methods and science with clients, client stakeholders, TAC members, educational measurement researchers, and other appropriately interested parties; we are committed to furthering the science around test security. We regularly present at conferences and webinars where we openly share our Caveon approach, theories and methodologies. In fact this last year, we have presented at conferences in Phoenix, Orlando, Chicago, Seattle, Washington DC, Hong Kong, and Prague.

Conservative Recommendations

When we conduct an engagement, our approach is to focus on the situations and incidents that are most egregious, as evidenced in the data and the results that we analyze. We highlight those problems that are most readily identified, documented, and ideally, resolved. Dealing with these problems effectively will have the greatest positive impact to the overall validity and security of test results. This reasonable approach helps our clients, most of which suffer from ever-constrained budgets and resources, effectively concentrate their time, resources, and dollars where the likelihood of inappropriate test taking is highest.

Lastly, our growth and success is directly attributable to a few overarching principles—We always strive to exceed our clients’ expectations, comport ourselves honorably, provide valuable services, and share, as openly and honestly as we can, recommendations for improving the fairness and validity of our clients’ test programs. These principles result in proven, practical protection for our clients, and we intend to follow them for another eight years!

Please Submit Your Comments Below. Thank you!

It does not follow that item exposure is the same as item compromise. In fact, I’ve seen items compromised with an extremely small number of presentations. Some items have even been compromised prior to the first test being administered!

In my opinion, the notion that item compromise results from item exposure—as defined above—leads to improper conclusions, decisions, and ineffective procedures. I have a few reasons for this opinion, a couple of which I’ll give here. First, item exposure is absolutely necessary. It is obvious that no test can be effective unless its items are exposed during the exam. Test designers even let examinees view an item multiple times encouraging them to return to and review previous items again and again. Second, item compromise has very little to do with the definitions of item exposure given above. Consider this simple example: Suppose that an item was shown to one million test takers and was presented on every exam administered. This would be considered a very high number of exposures along with a 100% exposure rate. But, suppose that none of those examinees were able to share the item with others. In this simple example, the item remains uncompromised and perfectly secure, and can be continued to be used on the exam.

If we wish to reduce item compromise, the example illustrates that limiting the number of presentations or rate of presentations of an item is not as important as the methods used to secure the items, to protect them from theft, and to keep them from being used for cheating. For this reason we need improved item security, which means better ways to keep items from being stolen and used for cheating on subsequent exams. We need methods to detect when an item is truly compromised and then immediately to take it out of service. Instead, we often see stubborn adherence to a century-old model of relatively unsecure test administration, and believing that keeping an item from being presented on a test is a sensible way to secure it.

It is certainly possible to improve the way we secure items. As examples, there are protective item and test designs available, and certainly better test monitoring procedures, that we can use. And perhaps we can learn a little from other industries as well. Consider the problem with the theft of music over the Internet. No one would suggest that music is stolen because it was listened to by too many people. Instead, we see serious efforts to protect the music, to keep it from being stolen, to detect when it is stolen, and to punish those that are responsible. We should be doing the same.

We welcome comments below!