Steffi Pohl, Freie Universität Berlin

Reframing Rankings in Educational Assessments: Disentangling Different Aspects of Test Performance using process data

Keynote Speaker

2020 Early Career Award

Country rankings based on competence scores from educational large-scale assessments are widely received and have great impact for decisions in educational policy. However, reported scores in LSAs represent a combination of competence level and test-taking behavior. Even more striking, test-taking behavior is incorporated differently in the overall performance score for different examinees, threatening the fairness of comparisons, such as country rankings. I argue that test-taking behavior should not be viewed as a nuisance factor, but is an important and informative aspect in the assessment of performance. Process data can be used to identify and understand test-taking behavior. I suggest a fundamental change in reporting of assessment data: I propose to disentangle the factors that impact test performance, to report on all of these aspects, and to use all of these when drawing inferences about performance, such as for country rankings. This not only results in fairer comparisons across groups, but also allows for a better understanding and valid assessment of competencies, as well as for more tailored interventions.

Log in