Caveon Data Forensics™ is essential for assessing test security threats to your program, empowering you to manage them, and protecting your exams. These threats leave statistical patterns within individual test results, within groupings of test results (e.g., test sites, countries, test dates, etc.), and across your entire program. In addition to being one of the best sources for discovering test security threats and weaknesses, the test result data provides the evidence you need for taking action after test security has been breached.
Proprietary statistical algorithms analyze test result data and reveal security vulnerabilities as well as evidence of actual test fraud, such as cheating and piracy. The algorithms are specifically designed to find correlations and patterns in the data that identify potential test security violations. These analyses provide critical information regarding where and when suspect activity occurred, by whom, and its effects on your items and tests.
Caveon Data Forensics analysis procedures help answer the three main questions regarding the security of a testing program:
Are there indications that the exams or the items have suffered disclosure or compromise?
Are there indications that the tests were not administered securely? If so, which groups or locations are implicated?
Are there indications that test takers were involved in improper test-taking conduct?
These questions are answered by trained analysts who use statistical patterns of potential test security problems to probe the data, formulate explanations, and present results in a clear and compelling manner. The reports written by our analysts provide needed documentation and evidence to ascertain whether test scores are trustworthy and to support your actions when test security has been breached.
What If I Need More?
You cannot manage what you do not measure. When it comes to test security threats, data forensics is your measuring tape. Using it, you can identify whether your program has been subjected to test security threats. You can see whether potential breaches are increasing, decreasing, or remaining the same. But, when you act upon the results, you go beyond mere measurement. You can:
- Deter test fraud by taking decisive action and publicizing your efforts,
- Prevent test scores from being obtained unfairly by invalidating them in a timely manner,
- Know when to retire or revise disclosed items,
- Design security measures into your exams that will improve your ability to detect potential fraudulent activity, and
- Create presentations that your management team can use for making proactive test security decisions.
There are several questions that you should answer as you consider these options:
- Will commercial software provide the analyses that you need?
- Will commercial software be able to process and handle your unique data sets?
- How much effort will it take to integrate commercial software into your processes?
- Is it appropriate to add data forensics work to the current responsibilities of your psychometricians?
- Do you have software and tools that will support your psychometricians once your data forensics program is in place?
- How will you respond and adapt to changes in test security threats as your program grows?
Caveon’s data forensics team has the expertise, the experience, the tools, and the capability to conduct data forensics analyses on your behalf. They have conducted a wide variety of analyses and have seen many different situations. They provide an independent, objective, third-party analysis of your data. Moreover, with Caveon focused on the security of your tests, you can focus on your core business.
Caveon data forensics can be performed on every high-stakes exam, including both computer and paper-based exams. It does not matter whether your exam has one form or many forms, whether forms are pre-assembled or created on the fly (e.g., LOFT or CAT). In addition to standard multiple-choice and true-false questions, Caveon has successfully analyzed technology-enhanced and performance-based items. Going beyond simple questions, the data forensics team has successfully processed and analyzed multi-part and scenario-based questions. If the necessary test response data are collected, it doesn’t matter when, where, or how the tests were administered.
Data forensics statistics provide evidence concerning the validity of test scores. They also can provide evidence concerning the behavior of individuals when strong statistical patterns are present in the data. When appropriate policies and procedures are in place in a testing program, data forensics results can be used to invalidate scores as well as take action against individuals or groups who may have violated test security or committed fraud. Also, results can support investigations and provide evidence for legal proceedings. Whenever your program acts, you should always make sure that actions are consistent with the evidence and the inferences that are being made.
In legal proceedings, it is essential that your program act consistently and follow the policies that your program has adopted. This is known as acting in good faith. We have acted as expert witnesses concerning the statistical evidence in several legal proceedings.
We can provide guidance concerning the defensibility of each of the statistics as different actions and responses are evaluated.
Data forensics can be conducted on a regular, ongoing basis. This is known as monitoring or screening. When performed this way, the intent of the analysis is to detect potential problems and to measure security threats and potential security violations so that you may learn about test security breaches as quickly as possible. If you wish to monitor for potential test security threats, you should choose a regular frequency for conducting data forensics analysis that is appropriate for your program. This could be monthly, quarterly, or after each testing window.
Additionally, data forensics can be conducted in a targeted manner to assemble evidence with respect to a suspected test security violation. The same statistical tools may be used for both situations, but the methods of statistical inference may differ. If you wish to respond to test security incidents, you should conduct the data forensics analysis so that your response is timely.
Caveon’s data forensics offering is adaptable and flexible. Statistics that are computed depend upon the data elements you provide us. For example, data forensics will use response times when tests are administered by computer and item response times are available. Or, intensity data may be used when test results are entered using scanned documents to analyze wrong-to-right answer changes. The data forensics team will provide you with a document listing the data elements and types that are required, those that are desired, and those that are optional. Because an industry-standard data interchange format does not yet exist, the data forensics team will receive data in your preferred format, and will convert it into a format suitable for processing. This is known as data setup. The data forensics team has a lot of experience receiving and processing data in many different formats.
We would be glad to discuss data forensics implementation with you. We have assisted many clients with this request. We have many resources that provide information about data forensics on the Caveon website. There is also a very good list of resources on the CESPCERT.org website. When we provide and share results of data forensics analyses with our clients, we always have one or more orientation sessions to explain the statistics, the results, the methods, and the interpretation of the data. At times, we have conducted on-site training sessions for clients, and we also have conducted web-based seminars. Here are a few annual conferences which we always attend and where we generally make presentations: Association of Test Publishers’ (ATP) Innovations in Testing; Council of Chief State School Officer’s (CCSSO) National Conference on Student Assessment (NCSA), and the Conference of Test Security (COTS).