Steffi Pohl, Freie Universität Berlin
Reframing Rankings in Educational Assessments: Disentangling Different Aspects of Test Performance using process data
2020 Early Career Award
Date & Time: Friday, July 23 at 9:50am EST
Country rankings based on competence scores from educational large-scale assessments are widely received and have great impact for decisions in educational policy. However, reported scores in LSAs represent a combination of competence level and test-taking behavior. Even more striking, test-taking behavior is incorporated differently in the overall performance score for different examinees, threatening the fairness of comparisons, such as country rankings. I argue that test-taking behavior should not be viewed as a nuisance factor, but is an important and informative aspect in the assessment of performance. Process data can be used to identify and understand test-taking behavior. I suggest a fundamental change in reporting of assessment data: I propose to disentangle the factors that impact test performance, to report on all of these aspects, and to use all of these when drawing inferences about performance, such as for country rankings. This not only results in fairer comparisons across groups, but also allows for a better understanding and valid assessment of competencies, as well as for more tailored interventions.
About the Speaker
Steffi Pohl is a full professor for research methods in psychology and education at the Freie Universität Berlin, Germany. In her research, she works on approaches for dealing with missing values, estimating causal effects, designing replication studies, and assessments based on the use of log-data. For her work she received the dissertation award of the German Society for Research Methods in Psychology and the early career award of the Psychometric Society. Steffi is part of the executive commitee of the European Association of Methodology and has been speaker of the Method and Evaluation Group of the German Psychological Society. She is or has been associate editor for Psychometrika, British Journal of Mathematical and Statistical Psychology, and Zeitschrift für Psychologie.