Leah Feuerstahler, Fordham University

In item response theory, item parameter standard errors are used to characterize the uncertainty associated with individual parameter estimates. These standard errors also can be used to construct confidence bands (Thissen & Wainer, 1990) around estimated item response functions. Whereas early approaches to constructing confidence bands were based on Fisher information, Yang, Hansen, and Cai (2012) recently suggested a multiple imputation (MI) approach that can be used with any approximation to the item parameter covariance matrix. In both the analytic and MI approaches, confidence bands are constructed by treating the latent variable theta as fixed and plotting the variability of response probabilities conditional on theta. However, theta can also be understood as an artifact of the fitted model such that the theta metric itself is determined with error. Specifically, the latent trait metric can be defined as a multidimensional random vector of conditional response probabilities (Ramsay, 1996). Because these multidimensional random vectors will lead to somewhat different predictions across calibrations, item parameter estimation error implies uncertainty about the location of the metric. In this talk, I describe how MI and fully Bayesian approaches can be used to visualize and quantify metric stability, that is, the variability of the theta metric implied by item parameter standard errors. I also clarify how metric stability is related to other item response model evaluation outcomes (e.g., test information, model fit). Overall, I argue that metric stability measures provide unique information that aids in a holistic approach to model evaluation.

about the speaker

Feuerstahler is an assistant professor of psychometrics and quantitative psychology at Fordham University in New York City. Previously, she received her Ph.D. in quantitative psychology from the University of Minnesota and completed a postdoc at the University of California Berkeley’s graduate school of education. Her research interests include applied psychometrics in education and psychology, and she currently focuses on issues surrounding the specification, fit, and interpretation of item response models.