Fair and Square: DOMC Findings by Demographic Group
Written by Sarah Toton of Caveon Test Security
Fairness is important in testing for ensuring that groups or individual test-takers are not systematically disadvantaged. Often, people ask me about the fairness of DOMC items. In this article, we will explore if the DOMC item format produces items that are biased against specific demographic groups.
Here at Caveon, we have been doing a series of studies on innovative item types. We now have conducted and analyzed data from more than six research projects covering a whole range of topics, including geography, trivia, and technology. When we designed these experiments, we converted multiple-choice items to DOMC items. Thus, we have multiple-choice (MC) items and DOMC items that are equivalent in content and options and differ only in the format. We also asked participants survey questions about their demographics, experience with multiple-choice tests, and preference for multiple-choice versus DOMC. We directly compared the results from both item types across demographic variables such as gender, age, and ethnicity. Over all six of the research projects we’ve completed so far, we see similar results in terms of bias for multiple-choice and DOMC items. Generally, this is a lack of bias in both item types, but in a few cases we have seen a bias in scores for both. For example, age is related to geography knowledge such that older participants score higher. Let’s explore this finding…..