A Dynamic Confidence-Weighted Ensemble of Speech and Text Models for Emotion Recognition

Recognizing human emotion in conversation is a complex task because people convey feelings through multiple channels, including vocal tone (prosody) and word choice (lexical content). Systems that rely on a single channel, or modality, such as only speech or only text, often fail to capture the complete emotional context. To address this, we propose the Dynamic Confidence-Weighted Ensemble (DCWE). This novel framework intelligently integrates the outputs from a Speech Emotion Recognition (SER) model and a text-based emotion classifier. Unlike static fusion methods, our approach dynamically adjusts each model's contribution based on two key criteria: the pre-assessed difficulty of each emotion class for the model, and the model's real-time prediction confidence. By leveraging the complementary strengths of both models, DCWE achieves superior performance on multi-party conversational data.