Written by: Jamie Mulkey, Vice President, Caveon Test Development
Recently, I heard a segment on NPR on SMART appliances. You know, those appliances in your home that you manage from a smartphone app? For example, setting your thermostat or viewing footage from a security camera in your house while you’re vacationing in Hawaii. Well it seems that these home appliances are destined to get even SMARTER, eventually allowing you talk to them directly – like telling your coffee pot to brew at 6am the following morning.
I started day dreaming about what a SMART item would be. What if we could talk to the SMART Item Appliance and ask it for the right assessments for any given set of tasks? What would these items look like and how would we get to a point where such a database would exist?
Let’s start backwards at the end result. We would need to begin with a desired skill to be performed. Let’s say, we want to determine if a medical professional has the skills needed to administer a flu shot. If this were a SMART item, the question we’d be asking is, “what are the best items for evaluating readiness to administer a flu shot?” The SMART Item Appliance would come back with a number of different ways this skill could be measured. We might have some multiple-choice items that deal with placement of the needle on the person’s arm. What are good injection areas? What places should be avoided? There may be items that test for the right syringe and sterilizing elements to use. We might also see video-based questions that depict various ways of a shot being administered. We could gleefully point and click on the items we wanted to include in our assessment of administering a flu shot. Then we would administer our test, feeling confident that the right skills and cognitive abilities have been used to test for this performance.
We’d quickly realize our confidence in the quality of the test would depend on the quality of the intelligence entered into the SMART Item Appliance. In order to have the right skills, job task analyses (JTAs) would need to be executed so that we truly understood the performances that needed to be tested. These performances in turn, would generate the desired objectives that would subsequently produce these items.
This is where my dream turns into a nightmare. While JTAs are performed quite routinely today in many professions, we as practitioners often find it difficult to create objectives that truly capture the desired performance that results in great test items. For example, providing a context for the performance and evidence statements of what we expect to see as a result of the performance.
Maybe as we work our way to the SMART Item Appliance, we can work on our processes for developing objectives that truly measure; becoming SMARTER about capturing knowledge, skills, and abilities that enable us to create better items more quickly.
By the way, do you think they will ever come up with a SMART appliance that will make the clothes scattered all over my daughter’s room put themselves away?