Deep Learning Based Engagement Recognition in Highly Imbalanced Data

Engagement recognition is a growing domain in paralinguistics evaluation due to its importance in many human-computer and human-robot applications. Current requirements for such systems are not only to interact, but also to engage the user into interaction as long as possible. To do so, the machine should differ among different levels of engagement to adjust its behavior properly. However, actual models are still far from it, partially due to data quality – usually, engagement recognition datasets are highly biased towards the high-engagement levels, because they are more often and naturally expressed by humans during interaction context. Thus, currently, the development of a reliable engagement recognition system able to detect all engagement levels is necessary. To facilitate it, we introduce a deep learning engagement recognition framework in the context of the DAiSEE corpus, which is a highly imbalanced dataset. We showed that the metric used formerly for evaluating the performance of the models on the DAiSEE dataset is inadequate due to its imbalance and conducted extensive experiments on DAiSEE, suggesting a new baseline performance based on the Unweighted Average Recall metric.