Aleksandar Tomašević, University of Novi Sad, Serbia

Facial Expression Recognition (FER) has traditionally relied on CNN models trained on expert-annotated datasets using systems like Ekman’s Facial Action Coding System (FACS). While these “black box” approaches encode human expertise through transfer learning, they are constrained by predefined emotion categories and single-image analysis paradigms. This talk presents a novel approach toward transformer-based zero-shot emotion classification that generates multivariate emotion time series from video data. Unlike traditional CNN approaches, recent  transformer models such as CLIP operate as even more opaque black boxes because we cannot directly interpret their internal mechanisms nor fully understand their training processes. While they offer unprecedented flexibility in emotion recognition without training data constraints, this flexibility comes at the cost of losing the expert knowledge embedded in traditional training datasets.

To address this interpretability challenge, we propose a comprehensive psychometric validation pipeline: CLIP → dynamic Exploratory Graph Analysis (dynEGA) → Damped Linear Oscillator with Measurement Model (DLO-MM) → parameter interpretation. Our approach treats transformer-generated emotion scores as dynamic psychological constructs requiring rigorous psychometric validation.

Using the MAFW dataset—a large-scale, multi-modal database containing videos from diverse cultural contexts (China, Japan, Korea, Europe, America, India) across various emotional themes—we demonstrate our methodology through the transforEmotion R package. We compare CLIP-derived emotion scores against MAFW annotations, then apply dynamic Exploratory Graph Analysis (dynEGA) to identify latent emotion dimensions and extract network-based time series. The core innovation lies in modeling these dynamics using the Damped Linear Oscillator with Measurement Model (DLO-MM) framework.  In this model, the progression of each latent emotion factor is defined by a descriptive equation. This equation explains an emotion’s trajectory by considering its tendency to stabilize or amplify over time, its natural oscillatory frequency that creates cyclical emotional patterns, and the impact of unpredictable external influences on emotional states.

Our parameter recovery approach now utilizes a hybrid neural network. This network operates as a dual-pathway feature extractor: one path applies a series of sliding filters over the time series to recognize local waveform shapes, while the other reads the data sequentially to capture how the signal evolves over time. These two streams of information are merged into a combined representation, which is then translated by lightweight prediction heads into estimates of the dynamic parameters. The recovered parameters transform opaque transformer outputs into psychologically meaningful constructs, revealing whether emotions stabilize or amplify over time, identifying their natural oscillatory frequencies, and quantifying their sensitivity to environmental perturbations.

Analysis of multi-factor emotion time series demonstrates decent parameter recovery across emotion factors, with results showing coherent dynamic patterns. This complete pipeline enables researchers to extract, validate, and model emotion dynamics from naturalistic video data, establishing video-based emotion measurement as a promising data source for psychological research.

about the speaker

Aleksandar Tomašević is an Assistant Professor at the University of Novi Sad, Serbia, where he completed his PhD in Sociology with a focus on quantitative methods in 2019. His research applies computational methods to various fields of social science, with particular emphasis on network psychometrics and emotion analysis. 

His work spans multiple fields including:

• Statistical modeling and computational methods in behavioral sciences
• Network psychometrics
• Machine learning applications in emotion recognition and political communication
• Evolution of trust and cooperation in online communities

He is one of the developers of the transforEmotion R package for multimodal sentiment analysis. His research network includes collaborations with leading institutions in the United States and Europe through the European Social Survey and the COST Opinion Network Action. He is currently involved in two major research projects: CTRUST, where he works with physicists and mathematicians on computational models of trust in online communities, and IDIOMATIC, a large-scale cross-cultural project focusing on the relationship between identity content and adolescent mental health. This work reflects his commitment to interdisciplinary research bridging psychology, computer science, social sciences, and physics.

His recent publications focus on innovative methodological approaches, including dynamic exploratory graph analysis of emotions, applications of transformer models in behavioral research, and network analysis of psychological constructs. His work has appeared in journals such as Personality and Individual Differences, Current Psychology, and EPJ Data Science.