Alina von Davier, ACTNext, Inc.
Miss Marple’s Search for Truth in Big Data: On Design, Causal Inference and Computational Psychometrics
Advances in technology have led to a redesign of the educational experience, from testing to learning, from academic to holistic evaluations. Most of these experiences are in virtual settings or in technology-enhanced real environments that allow for an unprecedented amount of fine-grained data collection and for a merge of various data sources that together have the potential to improve the quality of measurement and of diagnostic and recommendations capabilities. The bounty of big data comes with the realization that not all of it is good data and that extracting signal requires a detective mind that can sift through data patterns, design considerations and computational approaches. In this presentation I will share my experience with computational psychometrics, going back to basics, and address the relevance of causal inference in big data.
Michel Wedel, University of Maryland
Psychometric Analysis of Eye Movements during Search and Choice
In this presentation I provide an overview of eye movement research for search and choice processes, and provide a framework that helps psychometricians collect and analyze eye movement data. I discuss the multiple cognitive processes underlying the eye movements people make during visual search and choice behavior, briefly mention the design of eye tracking studies, and outline which relevant eye movement measures can be collected through modern eye tracking equipment. Then, I describe integrative Bayesian models of search and choice that represent the underlying cognitive processes based on eye movement data. I conclude with some directions for future research.
Bin Yu, Departments of Statistics and EECS, UC Berkeley
Veridical Data Science
Building and expanding on principles of statistics, machine learning, and the sciences, we propose the predictability, computability, and stability (PCS) framework for veridical data science. Our framework is comprised of both a workflow and documentation and aims to provide responsible, reliable, reproducible, and transparent results across the entire data science life cycle. The PCS workflow uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. It augments predictability and computability with an overarching stability principle for the data science life cycle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. We develop inference procedures that build on PCS, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, modeling decisions, and interpretations. Moreover, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS framework will be illustrated through examples in social science and a case study from neuroscience for movie reconstruction using brain signals.