Dan Bolt, University of Wisconsin – Madison

Item Complexity: A Neglected Psychometric Feature of Test Items?

Keynote Speaker

2021 Presidential Address

Despite its frequent consideration in test development, item complexity receives little attention in the psychometric modeling of item response data. In this address, I consider how variability in item complexity can be expected to emerge in the form of item characteristic curve (ICC) asymmetry, and how such effects may significantly influence applications of item response theory, especially those that assume interval level properties of the latent proficiency metric. One application is the score gain deceleration phenomenon often observed in vertical scaling contexts involving math or secondary language acquisition. It is demonstrated that the application of symmetric IRT models in the presence of complexity-induced positive ICC asymmetry is a likely cause. A second application concerns the positive correlation between DIF and difficulty commonly seen in verbal proficiency (and other subject area) tests where problem-solving complexity is minimal and proficiency-related guessing effects are likely more pronounced. It is shown how systematic negative ICC asymmetry creates artificial positive correlations when applying either nonparametric (e.g., standardization) or parametric (IRT model based) methods when no DIF is actually present. Unfortunately, the presence of systematic forms of ICC asymmetry is easily missed due to the considerable flexibility afforded by latent trait metrics in IRT. Speculation is provided regarding other applications for which attending to ICC asymmetry may prove useful.

Log in