WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning

Rajath Rao, Adithya V Ganesan, Oscar Kjell, Jonah Luby, Akshay Raghavan, Scott Feltman, Whitney Ringwald, Ryan L. Boyd, Benjamin Luft, Camilo Ruggero, Neville Ryant, Roman Kotov, H. Andrew Schwartz

January 2025 · Journal article · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics

@article{rao2025whispa,
  title = {{WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learning}},
  author = {Rajath Rao and Adithya V Ganesan and Oscar Kjell and Jonah Luby and Akshay Raghavan and Scott Feltman and Whitney Ringwald and Ryan L. Boyd and Benjamin Luft and Camilo Ruggero and Neville Ryant and Roman Kotov and H. Andrew Schwartz},
  year = {2025},
  journal = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics}
}

Abstract

Current speech encoding pipelines often relyon an additional text-based LM to get robustrepresentations of human communication, eventhough SotA speech-to-text models often havea LM within. This work proposes an approachto improve the LM within an audio model suchthat the subsequent text-LM is unnecessary. Weintroduce WhiSPA (Whisper with Semanticand Psychological Alignment), which lever-ages a novel audio training objective: con-trastive loss with a language model embeddingas a teacher. Using over 500k speech segmentsfrom mental health audio interviews, we evalu-ate the utility of aligning Whisper’s latent spacewith semantic representations from a text au-toencoder (SBERT) and lexically derived em-beddings of basic psychological dimensions:emotion and personality. Over self-supervisedaffective tasks and downstream psychologicaltasks, WhiSPA surpasses current speech en-coders, achieving an average error reductionof 73.4% and 83.8%, respectively. WhiSPAdemonstrates that it is not always necessaryto run a subsequent text LM on speech-to-textoutput in order to get a rich psychological rep-resentation of human communication

← All publications