Facial Recognition and Emotional Expressions Over Video Conferencing Based on Web Real Time Communication and Artificial Intelligence

Since the emergence of the Coronavirus pandemic, relying on video conferencing skyrocketed. The stay home and stay safe slogan increased the need of using Video Conferences in—for example but not limited to—education, work meeting, business deals ,.. etc providing a safer yet effective communication channel. On other hand, online meetings are lacking interaction. Emotional expressions provide a method to determine the Audience’s impressions about the meeting. Demonstrated in this paper, an application for video conferencing capable of detecting the facial emotion like surprise, happy, sad or angry by reading the particepants’ facial expressions also it can detect age and gender. The application is based on WebRTC and TensorFlow.js technologies. WebRTC, Web Real Time Communication, is a collection of standards and protocols which are supported in most of the modern browsers. Using web applications and platforms based on WebRTC technology helps in establishing a peer to peer communication and streaming, transmitting and receiving video, audio and data in real-time. In other hand, TensorFlow.js is opensource models in JavaScript which applied the concept of machine learning. Both WebRTC and TensorFlow.js can be used as regular JavaScript APIs which provide optimization in the performance and simplicity of the applications because the most of processing will be over the client-side and that is the core of our application. In this paper, it’s used the OpenVidu as a videoconference platform based on WebRTC and integrated it to Faceapi.js which is built based on tensorflow.js to can recognition face emotions and expressions over videoconference. We will use four classifications of servers in our application. Hosted Server, signaling server, NAT traversal server, and media server are used for WebRTC to handle the streaming across various types of networks.