Jessica Flake, University of British Columbia

Psychology is in a period of methodological reform. Researchers are rethinking their practices, sharing their data, and trying out registered reports. In this open science era, my work has focused on the role of measurement practices. I’ll provide some background as to how that previous metascience and psychometric research in the context of replication studies led me to discover two related problems. First, even as registration becomes common, we lack practices to analysis plan for latent variable models and to ensure their transparent reporting and reproducibility. Second, current methodological research in the form of simulation studies does not address this because it does not focus on how to navigate the garden of forking paths and quantify result heterogeneity that stems from analytical flexibility. These problems prevent the uptake of open science practices, threaten the validity of research results, and limit the impact of methodological research. I’ll preview my recent work to rethink simulation studies to integrate multiverse analysis and my efforts to consider decision making in analysis pipelines a necessary aspect of methodological development. I’ll close with some ideas for methodologists getting started on their own methodological reform movement and invite ideas from the audience.

about the speaker

Dr. Flake developed an interest in psychology and statistics as an undergraduate at Northern Kentucky University and went on to complete a PhD in Measurement, Evaluation, and Assessment from the University of Connecticut in 2015. She recently joined the faculty in the Quantitative Methods Area of the Department of Psychology at University of British Columbia, after having been an Associate Professor at McGill University. Her lab works on applied problems in psychological measurement and psychometrics with an emphasis on reproducibility and replicability. She was a founding member of the Psychological Science Accelerator, a distributed laboratory network of over 1,000 researchers, serving as the Assistant Director of the Data and Methods Committee from 2018 to 2023. She continues to work on methodological challenges for big team science and psychometric methods for complex data structures. Currently she is an Associate Editor at Advances in Methods and Practices in Psychological Science and Psychological Methods. Her recent and future work is considering how open science practices like registered reports and multiverse analysis can be further developed for latent variable models.