We’ve been there. Enter your email to get a permanent copy of this guide sent directly to your inbox.
Writing an excellent (and psychometrically sound) exam is a difficult, time-consuming, and expensive undertaking. And it is devastating when the exam you worked so hard to develop is stolen or exposed, and all that hard work (and money) goes down the drain.
Luckily, there is a solution. You can spare your team this heartache by proactively building security directly into your exam itself during the development process.
Let’s learn how.
Reduced exposure leads to longer-lasting item pools, so you can wait much longer between performing maintenance on your exams. Content is less likely to be breached or even beneficial to test takers who’ve somehow gotten access to it.
The combined power of more accurate measurement and increased security will lead to the restored validity of test scores. Test takers and stakeholders will all be able to respect and trust the scores of your important exams (in case you didn’t know—all of your exams are important!).
New forms of secure test and item designs are primarily guided by the concept of reduced exposure. By reducing the relative amount of content your test takers see, you’ll reduce the risk of item exposure to other groups.
In addition to bolstering security, the newer and more advanced item types (like these) and methods of exam design can measure the abilities of test takers far better than their archaic counterparts. You can learn more about advanced item and test designs in this section.
Now that you know the impact of secure exam design, how will that affect the rest of your choices from this point forward? When thinking about the types of security your exam will need, consider these three questions. Then, perhaps write on a sticky note: “What About the Security?” Place it somewhere prominent to remind you to consider things you can do to improve security within each stage of the exam design & development process—before, during, and after building your exam. (Learn more about the exam development process in this article.)
Your first major security consideration should be “What type of exam will you build?” Not all are created equally. Research different secure test designs and different types of test items that work for your content area. Here’s a great article to get you started.
Do you need to research or vet test development systems, delivery vendors, proctoring solutions, or any other components of your exam workflow before you get started? Make sure your desired security features are supported. You can find a list of the important features to look out for in this article.
Is your team fully on board with the benefits of secure design? What type of training and support might they need to help smooth the implementation process? Would training on how to write test questions be beneficial for your team? You can also check out these three workbooks that have been specifically created to support your team before, during, and after the exam development process.
How will you measure? What is the best way to test the skills you’re trying to measure? Essay questions, performance sections? Should you build or buy your exam?
Who is on your exam development team? Teams usually need a project manager, subject matter experts (SMEs), and other key players when creating exams. Who is responsible for ensuring security? You can view a list of helpful team players on page 15 of this workbook.
What format will you use? What exam format makes the most sense for your testing purposes? Keep in mind that some formats are more secure than others.
What item types should you use? Different item types work best with different content domains. Which forms of test items suit your content best? How can secure item types be best utilized?
What’s your budget? Make a list of your must-have security, development, and delivery needs. Allocate your budget accordingly.
What’s your project deadline? Work backwards from there.
Job Task Analysis. This is when you’ll start scoping the skills that you’ll measure and defining your audience.
Security can (and should) be built into each one of these steps. We recommend implementing security measures that prevent, deter, and detect + react to test security threats (learn more about the test security process in this article).
Some secure test designs (such as the SmartItem™) might change the flow of the test design process, streamlining certain areas and requiring more care and consideration in others. Remember, the end goal is to create a great, secure exam. The best process for you? Well, that is simply the one that gets you there.
This may hurt, but you probably need to hear it: Exam fraud happens all over, all the time, for all testing programs (and it has quite the impact). For many otherwise honest people, the end justifies the means when an opportunity is at stake. In other words, every exam is a high-stakes exam, and yours is at risk. You can read about the most common types of cheating in online exams in this article.
Not all item types are created equal. Some are easier to score, some are easier to implement, and some are far more secure. Consider scalability, budget, accessibility, and (most importantly) security when choosing your item formats.
Wrong-to-right answer changes and similarity analyses can help detect educators and exam administrators who commit test fraud.
Computerized Adaptive Test (CAT) design adapts to you.
Performance-based items have an extra layer of security.
Linear on the Fly Test (LOFT) design is strong and secure.
Utilize multiple forms to reduce the effects of content exposure.
Design exams with scrambled item and response-option orders. Randomizing your items and response options is a good first step in making it more difficult to steal your exam content and use it to cheat. After all, pre-knowledge is only useful when examinees see what other content thieves before them have seen.
The larger the item pool, the less content each individual test taker will see. Additionally, an expansive item pool gives you the option to replace exposed items quickly, avoiding widespread validity issues. Learn more about the benefits of a large item pool below.
A SmartItem uses special technology during development and delivery so that the item renders differently each time it is given on a test. Translation—no two examinees see the same thing during testing, so “stealing answers” to gain an advantage is pointless. To encourage proper studying and instruction, the SmartItem is coded to completely cover the target standard or competency. No more “teaching to the test.” Jump ahead to this section to learn more.
Numbers don’t lie. More items in your item pool mean that fewer eyes see each item. Limit item exposure with Automated Item Generation by rapidly increasing your item pool and staying one step ahead of content thieves and aging exams.
A larger pool of items can prevent test takers from using pre-knowledge. This will make it harder for examinees to predict what their test questions will be and prevent them from guessing what the correct answers are. Nice try!
To combat item theft, it is important to stay one step ahead of prying eyes, and AIG makes it possible to refresh your item pool as often as necessary, as painlessly as possible. And the lifetime cost savings with AIG are pretty impressive as well—view this infographic to learn more.
Maintain an extra layer of security by involving fewer individuals in the item writing process. AIG enables smaller teams to do more with less—efficiency and security all rolled into one.
Secure test designs such as CATs, LOFTs, and the use of multiple forms require fairly large item banks to pull off. That hasn’t always been attainable, but thanks to modern AIG technology, it’s now possible to do more with fewer resources.
SmartItem technology actually enhances the fairness of your exams by completely eliminating the two largest sources of unfairness: testwiseness and cheating.
Designing your exam with SmartItems prevents cheating. With so many renderings, you can no longer cheat by sharing test content, buying questions and answers, asking a friend to take the test and tell you what they saw, etc.
Success on an exam should depend on the depth of an examinee’s knowledge—not their ability to tactically take tests. DOMC items have also been shown to reduce the effects of testwiseness on scores.
SmartItem technology ensures that you don’t need to replace or fix your test until the skills you are measuring are revised. This curbs the draining costs that typically accompany content theft endeavors and security breaches. You can learn more about the cost savings in this booklet.
No matter your question type—multiple-choice, drag-and-drop, short answer, or others—converting them into SmartItems is possible. A SmartItem is not an item type; it’s a treatment for any kind of item. Whatever item types you use, you can make them render in unlimited ways. View SmartItem test item examples here.
Likelihood of Exposure: HIGH
Likelihood of Theft: HIGH
Useable Lifespan: SOMETIMES WEEKS
Likelihood of Exposure: MINIMAL
Likelihood of Theft: MODERATE
Useable Lifespan: SOMETIMES EVERGREEN
You don’t need a disposable lid, bag, or box. Just download here.
Want to get our subscriber-only newsletters and resources?