Psychometric Society Webinar Series

Presenters: Yigal Attali, Andrew Runge, Geoff LaFlair, Kevin Yancey & Alina A. von Davier

Moderator: Peter Halpin

Along with the advances in communication and platform technology, it’s become apparent that (quality) content development is at the core of many industries, including the education industry. The development of learning and assessment content has been a craft that has required a high level of expertise, often of the type that was built over the years on the job. In the fast-paced digital education this is difficult to sustain.

Moreover, in traditional assessments, a further complicating factor in the development and maintenance of the quality of the test is that items are pretested with human subjects, which is an expensive and time-consuming activity.

This webinar presents an alternative approach to the development of an assessment based on:

• Creating a large item bank using language-model-based automatic item generation techniques;
• Estimating their preliminary difficulties using natural language processing (NLP) models;
• Piloting the items in the context of an adaptive test and a framework for updating item parameters.

This framework involves a form of concurrent calibration that accounts for the fact that different people take different items. We have investigated several methodologies for this purpose that account for the selection bias inherent in adaptive testing. We also use a Bayesian approach for incorporating the uncertainty of the estimates of the item parameters.

In summary, this webinar illustrates how a psychometric framework combined with ML algorithms can support quality assessments for the 21st century. We will illustrate this approach with the Duolingo English Test.

Registration is free, but you will need to register to attend. Register in advance for this webinar here.

After registering, you will receive a confirmation email containing information about joining the webinar.