Language-based Assessments for Experienced Well-being: Accuracy and External Validity Across Behaviors, Traits, and States
Nilsson, A., Boyd, R., Ganesan, A. V., Kjell, O. N. E., Mahwish, S., Huang, H., Rosenthal, R. N., Ungar, L. H., & Schwartz, H. A. (2024). Language-based Assessments for Experienced Well-being: Accuracy and External Validity Across Behaviors, Traits, and States. PsyArXiv. https://doi.org/10.31234/osf.io/dgnaf
@misc{nilsson2024languagebased,
title = {{Language-based Assessments for Experienced Well-being: Accuracy and External Validity Across Behaviors, Traits, and States}},
author = {August Nilsson and Ryan Boyd and Adithya V. Ganesan and Oscar N.E. Kjell and Syeda Mahwish and Haitao Huang and Richard N. Rosenthal and Lyle H. Ungar and H. Andrew Schwartz},
year = {2024},
howpublished = {PsyArXiv},
doi = {10.31234/osf.io/dgnaf},
url = {https://doi.org/10.31234/osf.io/dgnaf}
}
Abstract
Self-reported rating scales have been central to social science for decades, even though language is our primary form of communication. We used language analysis with machine learning to compare self-reported ratings with language-based responses of experienced well-being (i.e., daily emotions) linked with human traits, states, and behaviors. In a sample of 764 U.S. service workers who reported their feelings daily for up to 12 weeks, responses were provided in 1) numerical ratings, 2) short essays, and 3) descriptive words. When predicting nine self-rated affective traits, essays were slightly less accurate than the ratings (r = .49 v .55, p < .001), while the word-based responses slightly outperformed the ratings (r = .53 vs .51, p < .001). For external predictive validity across 24 traits, demographics, and behaviors, both language formats doubled the ratings' accuracy, with words reaching an average of r = .18 and essays r = .24 (p < .001). Beyond their greater accuracy, both language formats offered rich psychological insights, with words focusing on emotions and essays encompassing broader life domains. While both language-based formats showed advantages and disadvantages, each demonstrated greater information-richness than traditional rating scales.
