Language-Based Affect Assessments Capture Experiment-Induced Changes Beyond Rating Scales
Bång, O., Gu, Z., Nilsson, A., Eijsbroek, V., Wiebel, C., Ganesan, A. V., Kjell, K., Schwartz, H. A., & Kjell, O. (2025). Language-Based Affect Assessments Capture Experiment-Induced Changes Beyond Rating Scales. PsyArXiv. https://doi.org/10.31234/osf.io/phgjn_v1
@misc{bang2025languagebased,
title = {{Language-Based Affect Assessments Capture Experiment-Induced Changes Beyond Rating Scales}},
author = {Oskar Bång and Zhuojun Gu and August Nilsson and Veerle Eijsbroek and Clara Wiebel and Adithya V. Ganesan and Katarina Kjell and H. Andrew Schwartz and Oscar Kjell},
year = {2025},
howpublished = {PsyArXiv},
doi = {10.31234/osf.io/phgjn_v1},
url = {https://doi.org/10.31234/osf.io/phgjn_v1}
}
Abstract
Language-based assessments (LBAs) have shown strong convergent and criterion validity cross-sectionally and longitudinally. However, no study has tested the ability of LBAs of affect to capture experiment-induced changes (i.e., causal validity). In this randomised mixed-design experiment, participants (N=892) completed a Mood Induction Procedure comprising different conditions (church, shopping mall, and park) using physical locations (N=153) and online video clips (N=739). Participants reported their pre- and post-induction affect through open-ended responses and ratings (i.e., the Positive and Negative Affect Schedule; PANAS). Using large language models and machine learning, we compared how accurately PANAS and language responses classified the experimental conditions. The LBA derived from the language responses exhibited significantly higher accuracy compared to the PANAS scores in classifying the conditions in both the training set (cross-validated AUC=.74 vs. .63), the online holdout set (AUC = .76 vs. .70), and the offline holdout set (AUC = .67 vs. .53). Apart from showing higher predictive accuracy, we assessed LBAs of specific affective dimensions (valence, arousal, and basic emotions). These analyses revealed that positive valence was most strongly expressed in the park condition, while anger was particularly elevated in the mall condition. Further, we visualized the language differences across conditions, showing more calm and peaceful expressions in the park condition and more tired and bored expressions in the mall condition. These findings support the causal validity of language-based affect assessments and its ability to provide insights into affective states. We discuss the potential of using LBA in experimental psychological research.
