Keeping your Rock Garden Tidy; Measure and Manage your Test Security Risks

When I was a child in South Carolina, my father had a rock garden in the back yard with a birch tree in the corner.  The garden was a lovely refuge.  The “exotic” birch, with its happy little green leaves, reduced the otherwise scorching summer sunlight to a soft glow that dappled the white rocks below. Even though I loved the rock garden, I avoided it like the plague because my Dad would tell me to pick the leaves out of the rocks if he saw me loitering there.  Something about this chore was paralyzing; I hated it.  With much theater, I would sulk, drag my feet, and reluctantly plop down in the littered rocks to begin what, to me, was an insurmountable task.  One day my father said something that, unbeknownst to him, not only would change the way I thought about the rock garden, but also would change the way I approached problem solving for the rest of my life.  He said, “You don’t have to get all of them.”

If you haven’t addressed security in your testing program, the notion of implementing a program to prevent all cheaters from stealing and using your test content can be overwhelming, not to mention impossible.   This idea can be so paralyzing that you might prefer not to address test security at all.  But rather than bury your head in the sand (or, in keeping with the metaphor, bury your head in the rocks), let go of the expectation that your test security program must be perfect; rather, consider taking action to improve your test security, as any security element you choose to implement will be more effective than no security element.

We are frequently asked by testing program managers, “If I can do only one thing to improve test security in my program, what would it be?” The simple answer is data monitoring and analysis.

The Association of Certified Fraud Examiners (ACFE) recently released its 2014 “Report to the Nations” which makes a detailed study of occupational fraud. You can find this report at: In speaking about occupational fraud, the ACFE concluded,

“Many of the most effective anti-fraud controls are being overlooked by a significant portion of organizations. For example, proactive data monitoring and analysis was used by only 35% of the victim organizations in our study, but the presence of this control was correlated with frauds that were 60% less costly and 50% shorter in duration. Other less common controls — including surprise audits, a dedicated fraud department or team and formal fraud risk assessments — showed similar associations with reductions in one or both of these measures of fraud damage. When determining how to invest anti-fraud dollars, management should consider the observed effectiveness of specific control activities and how those controls will enhance potential fraudsters’ perception of detection.” (Italics added.)

Monitoring test security risks through data analysis, or Data Forensics, as we call it at Caveon, provides the following benefits:

  1. You are alerted in a timely manner, before an issue becomes a raging inferno, that you have potential security problems that need to be managed.
  2. You can begin to proactively intervene where potential security problems exist.
  3. You can evaluate the effectiveness of other security measures and procedures that you implement.
  4. You will deter some fraudulent activities because individuals will realize they may be detected and disciplined.
  5. You know where to look, when to look, and how to look for potential security problems.

At Caveon, we like to say that the test security process has four essential elements: Protection, Detection, Response, and Improvement. Each element is important. Routine monitoring and analysis supports each one of these elements.

With today’s advances in technology combined with higher stakes in testing, there is simply no way to prevent all  types of cheating, but implementing consistent monitoring and analysis as part of your testing program will effectively mitigate cheating activity and help keep your rock garden tidy.  Taking small steps that are informed by data analysis will allow you to measure and manage your test security risks.


Jennifer Miller

Data Forensics Coordinator, Caveon Test Security

Leave a Reply