Search

This paper tackles the challenges of face detection, a vital computer vision task with wide-ranging applications, particularly in driver monitoring systems, where both accuracy and computational efficiency are crucial. Traditional frame-based methods often suffer from high computational complexity and under-sampling issues. In turn this limits their effectiveness in real-time applications and increase energy budget. To overcome these limitations, this paper explores the benefits of neuromorphic event cameras, which capture asynchronous pixel-level changes, offering lower data processing demands and reduced latency. The paper proposes a hybrid architecture combining Spiking Neural Networks (SNNs) and Artificial Neural Networks (ANNs). This approach leverages the energy efficiency and low-latency of SNNs while maintaining the high accuracy of ANNs, resulting in a highly efficient and accurate face detection system. The proposed Spiking-Face approach demonstrates a 57.76% improvement in mean average precision (mAP) over state-of-the-art methods. Additionally, the paper provides a comprehensive analysis of the system’s performance across different temporal resolutions, showing that the system performs robustly and adapts effectively to varying conditions. This hybrid SNN-ANN architecture achieves up to three times higher computational efficiency compared to equivalent traditional ANN methods, significantly reducing computational complexity while maintaining accuracy. These findings underscore the potential of this hybrid architecture for real-time, energy-constrained applications.

The digitalisation of visual tasks through imaging techniques and Computer Vision has the potential to disrupt the manner in which Advanced Manufacturing processes are deployed. In this study we collaborated with the manufacturing industry to investigate the effective usage of end-to-end convolutional neural networks (CNNs) to enable advanced manufacturing processes by inspecting the seal integrity of sterile barrier packaging in highly regulated products, such as Medical Devices. For this purpose, a novel ‘DS1’ dataset of labelled images representative of production samples was acquired in an industrial-like environment which is an open source for future research work. The core focus of this research is to address the common challenges associated with performing quality inspections in advanced manufacturing environments with the aim of detecting defects with very high impact but very low occurrence rates, by incorporating a set of pre-trained deep learning architectures. The performance of state-of-the-art CNNs is validated on unseen test data when trained in small and imbalanced datasets with low image variation and low pixel complexity. The study indicated that while CNN performance drops when datasets are imbalanced, some architectures are more resilient and capable of successfully classifying defects in small datasets in the order of a few hundred samples wherein as little as 5% of the samples are defective. Furthermore, this study also discusses the marginal impact of training with basic data augmentations and the tendency for models to overfit when trained with manufacturing datasets such as “DS1.”

Event cameras provide a novel imaging technology for high-speed analysis of localized facial motions such as eye gaze, eye-blink and micro-expressions by taking input at the level of an individual pixel. Due to this capability, and lightweight amount of the output data these cameras are being evaluated as a viable option for driver monitoring systems (DMS). This research is the first to investigate the impact of bias modifications on the event-based DMS output and propose an approach for evaluating and comparing DMS performance. The study investigates the impact of pixel-bias alteration on DMS features, which are: face tracking, blink counting, head pose and gaze estimation. In order to do this, new metrics are proposed to evaluate how effectively the DMS performs for each feature and overall. These metrics identify stability as the most important factor for face tracking, head pose estimations, and gaze estimations. The accuracy of the blink counting, which is the key component of this function, is also evaluated. Finally, all of these metrics are used to assess the system’s overall performance. The effects of bias changes on each feature are explored on a number of human subjects with their consent. The newly proposed metrics are used to determine the ideal bias ranges for each DMS feature and the overall performance. The results indicate that the DMS’s functioning is enhanced with proper bias tuning based on the proposed metrics.

Modern end-to-end Automatic Speech Recognition (ASR) systems struggle to recognise children’s speech. This challenge is due to the high acoustic variability in children’s voices and the scarcity of child speech training data, particularly for accented or low-resource languages. This study focuses on improving the performance of ASR on native and non-native English child speech using publicly available datasets. We evaluate how the large-scale whisper models (trained with a large amount of adult speech data) perform with child speech. In addition, we perform finetuning experiments using different child speech datasets to investigate the performance of whisper ASR on non-native English-speaking children’s speech. Our findings indicate relative Word Error Rate (WER) improvements ranging from 29% to 89% over previous benchmarks on the same datasets. Notably, these gains were achieved by finetuning with only a 10% sample of unseen non-native datasets. These results demonstrate the potential of whisper for improving ASR in a low-resource scenario for non-native child speech.

Event cameras (EC) represent a paradigm shift and are emerging as valuable tools in the automotive industry, particularly for in-cabin and out-of-cabin monitoring. These cameras capture pixel intensity changes as ”events” with ultra-low latency, making them suitable for real- time applications. In the context of in-cabin monitoring, EC offer solution for driver and passenger tracking, enhancing safety and comfort. For out-of-cabin monitoring, they excel in tracking objects and detecting potential hazards on the road. This article explores the applications, benefits, and challenges of event cameras in these two critical domains within the automotive industry. This review also highlights relevant datasets and methodologies, enabling researchers to make informed decisions tailored to their specific vehicular-technology and place their work in the broader context of EC sensing. Through an exploration of the hardware, the complexities of data processing, and customized algorithms for both in-cabin and out-of-cabin surveillance, this paper outlines a framework encompassing methodologies, tools, and datasets critical for the implementation of event camera sensing in automotive systems.

While current research predominantly focuses on image-based colorization, the domain of video-based colorization remains relatively unexplored. Many existing video colorization techniques operate frame-by-frame, often overlooking the critical aspect of temporal coherence between successive frames. This approach can result in inconsistencies across frames, leading to undesirable effects like flickering or abrupt color transitions between frames. To address these challenges, we combine the generative capabilities of a fine-tuned latent diffusion model with an autoregressive conditioning mechanism to ensure temporal consistency in automatic speaker video colorization. We demonstrate strong improvements on established quality metrics compared to existing methods, namely, PSNR, SSIM, FID, FVD, NIQE and BRISQUE. Specifically, we achieve an 18% improvement in performance when FVD is employed as the evaluation metric. Furthermore, we performed a subjective study, where users preferred LatentColorization to the existing state-of-the-art DeOldify 80% of the time. Our dataset combines conventional datasets and videos from television/movies. A short demonstration of our results can be seen in some example videos available at https://youtu.be/vDbzsZdFuxM.

Neuromorphic vision sensors, commonly referred to as Event Cameras (ECs), have gained prominence as a field of research in Computer Vision. This popularity stems from the numerous unique characteristics including High Dynamic Range, High Temporal Resolution, and Low Latency. Of particular interest is their temporal resolution, which proves ideal for human monitoring applications. Capturing rapid facial movements and eye gaze can be effectively achieved with ECs. Recent studies involving the use of ECs for object detection and tracking have demonstrated success in tasks involving Eye Motion Analysis such as Eye tracking, Blink detection, Gaze estimation and Pupil tracking. The objective of this study is to provide a comprehensive review of the current research in the aforementioned tasks, focusing on the potential utilization of ECs for future tasks involving rapid eye motion detection, such as detection and classification of saccades. We highlight studies that may serve as a foundation for undertaking such a task, such as pupil tracking and gaze estimation. We also highlight in our review some common challenges encountered such as the availability of datasets and review some of the methods used in solving this problem. Finally, we discuss some limitations of this field of research and conclude with future directions including real-world applications and potential research directions.

Heart rate is a crucial metric in health monitoring. Traditional computer vision solutions estimate cardiac signals by detecting physical manifestations of heartbeats, such as facial discoloration caused by blood oxygenation changes, from subject videos using regression methods. As continuous signals are more complex and expensive to de-noise, this study introduces an alternative approach, employing end-to-end classification models to remotely derive a discrete representation of cardiac signals from face videos. These visual cardiac signal classifiers are trained on discretized cardiac signals, a novel pre-processing method with limited precedent in health monitoring literature. Consequently, various methods to convert continuous cardiac signals into binary form are presented, and their impact on training is evaluated. An implementation of this approach, the temporal shift convolutional attention binary classifier, is presented using the regression-based convolutional attention network architecture. The classifier and a baseline regression model are trained and tested using publicly available and locally collected datasets designed for heart signal detection from face video. The model performance is then assessed based on the heart rate error from the extracted cardiac signals. Results show the proposed method outperforms the baseline on the UBFC-rPPG dataset, reducing cross-dataset root mean square error from 2.33 to 1.63 beats per minute. However, both models struggled to generalize to the PURE dataset, with root mean square errors of 12.40 and 16.29 beats per minute, respectively. Additionally, the proposed approach reduces the computational complexity of model output post-processing, enhancing its suitability for real-time applications and deployment on systems with restricted resources.

Devices that use speech as the communication medium between human and computer have been emerging for the past few years. The technologies behind this interface are called Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). The two are distinct fields in speech signal processing that have independently made great strides in recent years. This paper proposes an architecture that takes advantage of the two modalities present in ASR and TTS, speech and text, while simultaneously training three tasks, adding speaker recognition to the underlying ASR and TTS tasks. This architecture not only reduces the memory footprint required to run all tasks, but also has performance comparable to single-task models. The dataset used to train and evaluate the model is the CSTR VCTK Corpus. Results show a 97.64% accuracy in the speaker recognition task, word and character error rates of 18.18% and 7.95% for the ASR task, a mel cepstral distortion of 4.31 and two predicted MOS of 2.98 and 3.28 for the TTS task. While voice conversion is not part of the training tasks, the architecture is capable of doing this and was evaluated to have 5.22, 2.98, and 2.73 for mel cepstral distortion and predicted MOS, respectively.

For decades, the number of automobiles in urban areas around the world has been increasing. It causes serious challenges such as traffic congestion, accidents, and pollution, which have a social, economic, and environmental impact on widespread urban cities. To overcome these challenges, we need to explore smart AI-based perception systems for vehicular applications. Such types of systems can provide improved situational awareness to the driver and generate early alarm about upcoming obstacles and road incidents. In this study, we have presented the effective use of uncooled thermal IR sensors for designing smart thermal perception systems as an alternative to CMOS visible imaging by presenting state-of-the-art studies for in-cabin and out-cabin vehicular applications with potential long-term benefits for the automotive industry. The key rationale for selecting thermal IR sensors over conventional image sensors is that visible cameras are highly dependent on lighting conditions and performance is degraded significantly in low-lighting scenarios and harsh weather conditions. Contrary to this, thermal sensors remain largely unaffected by external lighting conditions or most environmental conditions, making them a perfect optical sensor choice for all-weather and harsh environmental conditions. This study presents a review of the current state of the art for automotive thermal imaging with a focus on the contributions and advances achieved by the EU-funded project ‘HELIAUS’ in the domain of AI-based thermal imaging pipelines for safer and reliable road journeys.

Due to the real-time acquisition and reasonable cost of consumer cameras, monocular depth maps have been employed in a variety of visual applications. Regarding ongoing research in depth estimation, they continue to suffer from low accuracy and enormous sensor noise. To improve the prediction of depth maps, this paper proposed a lightweight neural facial depth estimation model based on single image frames. Following a basic encoder-decoder network design, the features are extracted by initializing the encoder with a high-performance pre-trained network and reconstructing high-quality facial depth maps with a simple decoder. The model can employ pixel representations and recover full details in terms of facial features and boundaries by employing a feature fusion module. When tested and evaluated across four public facial depth datasets, the suggested network provides more reliable and state-of-the-art results, with significantly less computational complexity and a reduced number of parameters. The training procedure is primarily based on the use of synthetic human facial images, which provide a consistent ground truth depth map, and the employment of an appropriate loss function leads to higher performance. Numerous experiments have been performed to validate and demonstrate the usefulness of the proposed approach. Finally, the model performs better than existing comparative facial depth networks in terms of generalization ability and robustness across different test datasets, setting a new baseline method for facial depth maps.

Despite recent advancements in deep learning technologies, Child Speech Recognition remains a challenging task. Current Automatic Speech Recognition (ASR) models require substantial amounts of annotated data for training, which is scarce. In this work, we explore using the ASR model, wav2vec2, with different pretraining and finetuning configurations for self-supervised learning (SSL) toward improving automatic child speech recognition. The pretrained wav2vec2 models were finetuned using different amounts of child speech training data, adult speech data, and a combination of both, to discover the optimum amount of data required to finetune the model for the task of child ASR. Our trained model achieves the best Word Error Rate (WER) of 7.42 on the MyST child speech dataset, 2.91 on the PFSTAR dataset and 12.77 on the CMU KIDS dataset using cleaned variants of each dataset. Our models outperformed the unmodified wav2vec2 BASE 960 on child speech using as little as 10 hours of child speech data in finetuning. The analysis of different types of training data and their effect on inference is provided by using a combination of custom datasets in pretraining, finetuning and inference. These ‘cleaned’ datasets are provided for use by other researchers to provide comparisons with our results.

Event cameras, unlike traditional frame-based cameras, excel in detecting and reporting changes in light intensity on a per-pixel basis. This unique technology offers numerous advantages, including high temporal resolution, low latency, wide dynamic range, and reduced power consumption. These characteristics make event cameras particularly well-suited for sensing applications such as monitoring drivers or human behavior. This paper presents a feasibility study on the using a multitask neural network with event cameras for real-time facial analytics. Our proposed network simultaneously estimates head pose, eye gaze, and facial occlusions. Notably, the network is trained on synthetic event camera data, and its performance is demonstrated and validated using real event data in real-time driving scenarios. To compensate for global head motion, we introduce a novel event integration method capable of handling both short and long-term temporal dependencies. The performance of our facial analytics method is quantitatively evaluated in both controlled lab environments and unconstrained driving scenarios. The results demonstrate that useful accuracy and computational speed is achieved by the proposed method to determining head pose and relative eye-gaze direction. This shows that neuromorphic facial analytics can be realized in real-time and are well-suited for edge/embedded computing deployments. While the improvement ratio in comparison to existing literature may not be as favorable due to the unique event-based vision approach employed, it is crucial to note that our research focuses specifically on event-based vision, which offers distinct advantages over traditional RGB vision. Overall, this study contributes to the emerging field of event-based vision systems and highlights the potential of multitask neural networks combined with event cameras for real-time sensing of human subjects. These techniques can be applied in practical applications such as driver monitoring systems, interactive human-computer systems and for human behavior analysis.

Driver monitoring systems (DMS) are a key component of vehicular safety and essential for the transition from semi-autonomous to fully autonomous driving. Neuromorphic vision systems, based on event camera technology, provide advanced sensing in motion analysis tasks. In particular, the behaviours of drivers’ eyes have been studied for the detection of drowsiness and distraction. This research explores the potential to extend neuromorphic sensing techniques to analyse the entire facial region, detecting yawning behaviours that give a complimentary indicator of drowsiness. A second proof of concept for the use of event cameras to detect the fastening or unfastening of a seatbelt is also developed. Synthetic training datasets are derived from RGB and Near-Infrared (NIR) video from both private and public datasets using a video-to-event converter and used to train, validate, and test a convolutional neural network (CNN) with a self-attention module and a recurrent head for both yawning and seatbelt tasks. For yawn detection, respective F1-scores of 95.3% and 90.4% were achieved on synthetic events from our test set and the “YawDD” dataset. For seatbelt fastness detection, 100% accuracy was achieved on unseen test sets of both synthetic and real events. These results demonstrate the feasibility to add yawn detection and seatbelt fastness detection components to neuromorphic DMS.

This work presents a study on the effects of aging on the performance and reliability of facial authentication methods. First, a brief review of the literature on the effect of age on face recognition algorithms is presented, followed by a detailed description of the face aging datasets. In contrast with some recent studies, we demonstrate significant variations in authentication robustness between age groups. The second part of this paper focuses on a comprehensive comparative assessment on the effects across age groups. Four different face recognition algorithms are studied of which three are state-of-the-art neural network based models and the fourth one is a conventional machine learning model. Two different age range threshold settings (±3 in Experiment Category A and ±5 in Experiment Category B) of the age groups are adopted in the experimental analysis to get a proper comparison. Moreover, a synthetic aging method has been incorporated to augment the age data. Experimental result shows that the older adults groups are easier to identify with higher levels of accuracy and robustness compared to other age groups, while younger adults are the most challenging and false authentications are more likely to occur.

In this research work, we proposed a novel ChildGAN, a pair of GAN networks for generating synthetic boys and girls facial data derived from StyleGAN2. ChildGAN is built by performing smooth domain transfer using transfer learning. It provides photo-realistic, high-quality data samples. A large-scale dataset is rendered with a variety of smart facial transformations: facial expressions, age progression, eye blink effects, head pose, skin and hair color variations, and variable lighting conditions. The dataset comprises more than 300k distinct data samples. Further, the uniqueness and characteristics of the rendered facial features are validated by running different computer vision application tests which include CNN-based child gender classifier, face localization and facial landmarks detection test, identity similarity evaluation using ArcFace, and lastly running eye detection and eye aspect ratio tests. The results demonstrate that synthetic child facial data of high quality offers an alternative to the cost and complexity of collecting a large-scale dataset from real children. The complete dataset along with the trained model are open-sourced on our GitHub website and GitHub page: https://github.com/MAli-Farooq/ChildGAN.

Technologies such as Text-To-Speech (TTS) synthesis and Automatic Speech Recognition (ASR) have become important in providing speech-based Artificial Intelligence (AI) solutions in today’s AI-centric technology sector. Most current research work and solutions focus largely on adult speech compared to child speech. The main reason for this disparity can be linked to the limited availability of children’s speech datasets that can be used in training modern speech AI systems. In this paper, we propose and validate a speech augmentation pipeline to transform existing adult speech datasets into synthetic child-like speech. We use a publicly available phase vocoder-based toolbox for manipulating sound files to tune the pitch and duration of the adult speech utterances making them sound child-like. Both objective and subjective evaluations are performed on the resulting synthetic child utterances. For the objective evaluation, the similarities of the selected top adults’ speaker embeddings are compared before and after the augmentation to a mean child speaker embedding. The average adult voice is shown to have a cosine similarity of approximately 0.87 (87%) relative to the mean child voice after augmentation, compared to a similarity of approximately 0.74 (74%) before augmentations. Mean Opinion Score (MOS) tests were also conducted for the subjective evaluation, with average MOS scores of 3.7 for how convincing the samples are as child-speech and 4.6 for how intelligible the speech is. Finally, ASR models fine-tuned with the augmented speech are tested against a baseline set of ASR experiments showing some modest improvements over the baseline model finetuned with only adult speech.

Driver Monitoring Systems (DMS) represent a promising approach for enhancing driver safety within vehicular technologies. This research explores the integration of neuromorphic event camera technology into DMS, offering faster and more localized detection of changes due to motion or lighting in an imaged scene. When applied to the observation of a human subject event camera provides a new level of sensing capabilities over conventional imaging systems. The study focuses on the application of DMS by incorporating the event cameras, augmented by submanifold sparse neural network models (SSNN) to reduce computational complexity. To validate the effectiveness of proposed machine learning pipeline built on event data we have opted the Driver Distraction as a critical use case. The SSNN model is trained on synthetic event data generated from the publicly available Drive&Act and Driver Monitoring Dataset (DMD) using a video-to-event conversion algorithm (V2E). The proposed approach yields comparable performance with state-of-the-art approaches, achieving an accuracy of 86.25% on the Drive&Act dataset and 80% on comprehensive DMD dataset while significantly reducing computational complexity. In addition, to demonstrate the generalization of our results the network is also evaluated using locally acquired event dataset gathered from a commercially available neuromorphic event sensor.

The proliferation of multi-lingual content on today’s streaming services has created a need for automated multi-lingual dubbing tools. In this article, current state-of-the-art approaches are discussed with reference to recent works in automatic dubbing and the closely related field of talking head generation. A taxonomy of papers within both fields is presented, and the main challenges of both speech-driven automatic dubbing, and talking head generation are discussed and outlined, together with proposals for future research to tackle these issues.

This article presents C3I-SynFace: a large-scale synthetic human face dataset with corresponding ground truth annotations of head pose and face depth generated using the iClone 7 Character Creator “Realistic Human 100” toolkit with variations in ethnicity, gender, race, age, and clothing. The data is generated from 15 female and 15 male synthetic 3D human models extracted from iClone software in FBX format. Five facial expressions - neutral, angry, sad, happy, and scared are added to the face models to add further variations. With the help of these models, an open-source data generation pipeline in Python is proposed to import these models into the 3D computer graphics tool Blender and render the facial images along with the ground truth annotations of head pose and face depth in raw format. The datasets contain more than 100k ground truth samples with their annotations. With the help of virtual human models, the proposed framework can generate extensive synthetic facial datasets (e.g., head pose or face depths datasets) with a high degree of control over facial and environmental variations such as pose, illumination, and background. Such large datasets can be used for the improved and targeted training of deep neural networks.

Background Level 3 automated driving systems involve the continuous performance of the driving task by artificial intelligence within set environmental conditions, such as a straight highway. The driver's role in Level 3 is to resume responsibility of the driving task in response to any departure from these conditions. As automation increases, a driver's attention may divert towards non-driving-related tasks (NDRTs), making transitions of control between the system and user more challenging. Safety features such as physiological monitoring thus become important with increasing vehicle automation. However, to date there has been no attempt to synthesise the evidence for the effect of NDRT engagement on drivers’ physiological responses in Level 3 automation. Methods A comprehensive search of the electronic databases MEDLINE, EMBASE, Web of Science, PsycINFO, and IEEE Explore will be conducted. Empirical studies assessing the effect of NDRT engagement on at least one physiological parameter during Level 3 automation, in comparison with a control group or baseline condition will be included. Screening will take place in two stages, and the process will be outlined within a PRISMA flow diagram. Relevant physiological data will be extracted from studies and analysed using a series of meta-analyses by outcome. A risk of bias assessment will also be completed on the sample. Conclusion This review will be the first to appraise the evidence for the physiological effect of NDRT engagement during Level 3 automation, and will have implications for future empirical research and the development of driver state monitoring systems.

Robust authentication for low-power consumer devices without a keyboard remains a challenge. The recent availability of low-power neural accelerator hardware, combined with improvements in neural facial recognition algorithms provides enabling technology for low-power, on-device facial authentication. The present research work explores a number of approaches to test the robustness of a state-of-the-art facial recognition (FR) technique, Arcface for such end-to-end applications. As extreme lighting conditions and facial pose are the two more challenging scenarios for FR we focus on these. Due to the general lack of large-scale multiple-identity datasets, GAN-based re-lighting and pose techniques are used to explore the effects on FR performance. These results are further validated on the best available multi-identity datasets - MultiPIE and BIWI. The results show that FR is quite robust to pose variations up to 45–55 degrees, but the outcomes are not definitive for the tested lighting scenarios. For lighting, the tested GAN-based relighting augmentations show significant effects on FR robustness. However, the lighting scenarios from MultiPIE dataset - the best available public dataset - show some conflicting results. It is unclear if this is due to an incorrectly learned GAN relighting transformation or, alternatively, to mixed ambient/directional lighting scenes in the dataset. However, it is shown that the GAN-induced FR errors for extreme lighting conditions can be corrected by fine-tuning the FR network layers. The conclusions support the feasibility of implementing a robust authentication method for low-power consumer devices.

This article contains all of the information needed to conduct a study on monocular facial depth estimation problems. A brief literature review and applications on facial depth map research were offered first, followed by a comprehensive evaluation of publicly available facial depth datasets and widely used loss functions. The key properties and characteristics of each facial depth map dataset are described and evaluated. Furthermore, facial depth maps loss functions are briefly discussed, which will make it easier to train neural facial depth models on a variety of datasets for both short- and long-range depth maps. The network’s design and components are essential, but its effectiveness is largely determined by how it is trained, which necessitates a large dataset and a suitable loss function. Implementation details of how neural depth networks work and their corresponding evaluation matrices are presented and explained. In addition, an SoA neural model for facial depth estimation is proposed, along with a detailed comparison evaluation and, where feasible, direct comparison of facial depth estimation methods to serve as a foundation for a proposed model that is utilized. The model employed shows better performance compared with current state-of-the-art methods when tested across four datasets. The new loss function used in the proposed method helps the network to learn the facial regions resulting in an accurate depth prediction. The network is trained on synthetic human facial depth datasets whereas for validation purposes real as well as synthetic facial images are used. The results prove that the trained network outperforms current state-of-the-art networks performances, thus setting up a new baseline method for facial depth estimations.

“Smart Cities” are a viable option to various issues caused by accelerated urban growth. To make smart cities a reality, smart citizens need to be connected to the “Smart City” through a digital ID. A digital ID enables citizens to utilize smart city facilities such as healthcare, transportation, finance, and energy with ease and efficiency. In this paper, we propose a proof-of-concept of a facial authentication-based end-to-end digital ID system for a smart city. Facial authentication systems are prone to various biometric template attacks and cyber security attacks. Our proposed system is designed to detect the first type of attack, especially deepfake and presentation attacks. Users are authenticated each time they use facilities in a smart city. Facial data is stored in the cloud in a lookup table format with an unidentifiable username. The process is very secure as no data leaves the device during authentication. Our proposed solution achieved 97% accuracy in authentication with a False Rejection Ratio of 2% and False Acceptance Ratio of 3%.

Speech synthesis has come a long way as current text-to-speech (TTS) models can now generate natural human-sounding speech. However, most of the TTS research focuses on using adult speech data and there has been very limited work done on child speech synthesis. This study developed and validated a training pipeline for fine-tuning state-of-the-art (SOTA) neural TTS models using child speech datasets. This approach adopts a multi-speaker TTS retuning workflow to provide a transfer-learning pipeline. A publicly available child speech dataset was cleaned to provide a smaller subset of approximately 19 hours, which formed the basis of our fine-tuning experiments. Both subjective and objective evaluations were performed using a pretrained MOSNet for objective evaluation and a novel subjective framework for mean opinion score (MOS) evaluations. Subjective evaluations achieved the MOS of 3.95 for speech intelligibility, 3.89 for voice naturalness, and 3.96 for voice consistency. Objective evaluation using a pretrained MOSNet showed a strong correlation between real and synthetic child voices. Speaker similarity was also verified by calculating the cosine similarity between the embeddings of utterances. An automatic speech recognition (ASR) model is also used to provide a word error rate (WER) comparison between the real and synthetic child voices. The final trained TTS model was able to synthesize child-like speech from reference audio samples as short as 5 seconds.

A novel neural pipeline allowing one to generate pose aware 3D animated facial landmarks synchronised to a target speech signal is proposed for the task of automatic dubbing. The goal is to automatically synchronize a target actors’ lips and facial motion to an unseen speech sequence, while maintaining the quality of the original performance. Given a 3D facial key point sequence extracted from any reference video, and a target audio clip, the neural pipeline learns how to generate head pose aware, identity aware landmarks and outputs accurate 3D lip motion directly at the inference stage. These generated landmarks can be used to render a photo-realistic video via an additional image to image conversion stage. In this paper, a novel data augmentation technique is introduced that increases the size of the training dataset from N audio/visual pairs up to NxN unique pairs for the task of automatic dubbing. The trained inference pipeline employs a LSTM-based network that takes Mel-coefficients as input from an unseen speech sequence, combined with head pose, and identity parameters extracted from a reference video to generate a new set of pose aware 3D landmarks that are synchronized with the unseen speech.

In many applications, such as robotic perception, scene understanding, augmented reality, 3D reconstruction, and medical image analysis, depth from images is a fundamentally ill-posed problem. The success of depth estimation models relies on assembling a suitably large and diverse training dataset and on the selection of appropriate loss functions. It is critical for researchers in this field to be made aware of the wide range of publicly available depth datasets along with the properties of various loss functions that have been applied to depth estimation. Selection of the right training data combined with appropriate loss functions will accelerate new research and enable better comparison with state-of-the-art. Accordingly, this work offers a comprehensive review of available depth datasets as well as the loss functions that are applied in this problem domain. These depth datasets are categorised into five primary categories based on their application, namely (i) people detection and action recognition, (ii) faces and facial pose, (iii) perception-based navigation (i.e., street signs, roads), (iv) object and scene recognition, and (v) medical applications. The important characteristics and properties of each depth dataset are described and compared. A mixing strategy for depth datasets is presented in order to generalise model results across different environments and use cases. Furthermore, depth estimation loss functions that can help with training deep learning depth estimation models across different datasets are discussed. State-of-the-art deep learning-based depth estimation methods evaluations are presented for three of the most popular datasets. Finally, a discussion about challenges and future research along with recommendations for building comprehensive depth datasets will be presented as to help researchers in the selection of appropriate datasets and loss functions for evaluating their results and algorithms.

AI-based smart thermal perception systems can cater to the limitations of conventional imaging sensors by providing a more reliable data source in low-lighting conditions and adverse weather conditions. This research evaluates and modifies the state-of-the-art object detection and classifier framework for thermal vision with seven key object classes in order to provide superior thermal sensing and scene understanding input for advanced driver-assistance systems (ADAS). The networks are trained on public datasets and is validated on test data with three different test approaches which include test-time augmentation, test-time with no augmentation, and test-time with model ensembling. Additionally, a new model ensemble-based inference engine is proposed, and its efficacy is tested on locally gathered novel test data comprising of 20K thermal frames captured with an uncooled LWIR prototype thermal camera in challenging weather and environmental scenarios. The performance analysis of trained models is investigated by computing precision, recall, and mean average precision scores (mAP). Furthermore, the smaller network variant of thermal-YOLO architecture is optimized using TensorRT inference accelerator, which is then deployed on GPU and resource-constrained edge hardware Nvidia Jetson Nano. This is implemented to explicitly reduce the inference time on GPU as well as on Nvidia Jetson Nano to evaluate the feasibility for added real-time onboard installations.

Accurate head pose estimation from 2D image data is an essential component of applications such as driver monitoring systems, virtual reality technology, and human-computer interaction. It enables a better determination of user engagement and attentiveness. The most accurate head pose estimators are based on Deep Neural Networks that are trained with the supervised approach and rely primarily on the accuracy of training data. The acquisition of real head pose data with a wide variation of yaw, pitch and roll is a challenging task. Publicly available head pose datasets have limitations with respect to size, resolution, annotation accuracy and diversity. In this work, a methodology is proposed to generate pixel-perfect synthetic 2D headshot images rendered from high-quality 3D synthetic facial models with accurate head pose annotations. A diverse range of variations in age, race, and gender are also provided. The resulting dataset includes more than 300k pairs of RGB images with corresponding head pose annotations. A wide range of variations in pose, illumination and background are included. The dataset is evaluated by training a state-of-the-art head pose estimation model and testing against the popular evaluation-dataset Biwi. The results show that training with purely synthetic data generated using the proposed methodology achieves close to state-of-the-art results on head pose estimation which are originally trained on real human facial datasets. As there is a domain gap between the synthetic images and real-world images in the feature space, initial experimental results fall short of the current state-of-the-art. To reduce the domain gap, a semi-supervised visual domain adaptation approach is proposed, which simultaneously trains with the labelled synthetic data and the unlabeled real data. When domain adaptation is applied, a significant improvement in model performance is achieved. Additionally, by applying a data fusion-based transfer learning approach, better results are achieved than previously published work on this topic.

One of the most interesting challenges in Artificial Intelligence is to train conditional generators which are able to provide labeled adversarial samples drawn from a specific distribution. For a successful implementation of conditional generators, the created samples are constrained to a specific class. In this work, a new framework is presented to train a deep conditional generator by placing a classifier or regression model in parallel with the discriminator and back propagate the classification or regression error through the generator network. Special cases for binary classification, multi-class classification, and regression are studied. Experimental results on several data-sets are provided and the results are compared with similar state-of-the-art techniques. The main advantage of the method is that it is versatile and applicable to any variation of Generative Adversarial Network (GAN) implementation but also it is shown to obtain superior results compared to other methods. The mathematical proofs for the proposed scheme for both classification and regression are presented.

This work explores the identity attribute of synthetic face samples derived from Generative Adversarial Networks. The goal is to determine if individual samples are unique in terms of identity, firstly with respect to the seed dataset that trains the GAN model and secondly with respect to other synthetic face samples. Two approaches are introduced to enable the comparative analysis of large sets of synthetic face samples. The first of these uses ROC curves to determine identity uniqueness using a number of large publicly available datasets of real facial samples to provide reference ROCs as a baseline. The second approach uses a thresholding technique utilizing again large publicly available datasets as a reference. For this approach, new metrics are introduced, and a technique is provided to remove the most connected data samples within a large synthetic dataset. The remaining synthetic samples can be considered as unique as data samples gathered from different real individuals. Several StyleGAN models are used to create the synthetic datasets, and variations in key model parameters are explored. It is concluded that the resulting synthetic data samples exhibit excellent uniqueness when compared with the original training dataset, but significantly less uniqueness when comparisons are made within the synthetic dataset. Nevertheless, it is possible to remove the most highly connected synthetic data samples. Thus, in some cases, up to 92% of the data samples in a 20k synthetic dataset can be shown to exhibit similar uniqueness to data samples taken from real public datasets.

As a biometric modality, palmprints have been largely under-utilized, but they offer some advantages over fingerprints and facial biometrics. Recent improvements in imaging capabilities on handheld and wearable consumer devices have re-awakened interest in the use of palmprints. The aim of this paper is to provide a comprehensive review of state-of-the-art methods for palmprint recognition including Region of Interest extraction methods, feature extraction approaches and matching algorithms along with an overview of available palmprint datasets in order to understand the latest trends and research dynamics in the palmprint recognition field.

This paper proposes two major novelties. First, we provide a mathematical framework for hardware resource efficient IP core-based image compression and decompression (CODEC). The framework includes CODEC functions that are capable of determining the pixel intensities of a compressed gray scale image using significantly lesser hardware resources. Digital pixel values of the original image are fed as an input to the functions of proposed IP framework and compressed digital pixel values of compressed image generated. Similarly, digital pixel values of the compressed image are fed into other functions of the proposed framework for image decompression. Second, the second novelty is using the derived IP functions to propose designs of reusable IP cores for complete Haar wavelet transformation (HWT)-based lossy image CODEC. Testing of images from various data sets (NASA, medical applications, and so on) in terms of hardware resources, image quality, and compression efficiency have indicated that the proposed IP core framework was successful in achieving hardware efficient CODEC compared with JPEG and conventional HWT CODECs.

In this paper, a review is presented for the research on eye gaze estimation techniques and applications, which has progressed in diverse ways over the past two decades. Several generic eye gaze use-cases are identified: desktop, TV, head-mounted, automotive, and handheld devices. Analysis of the literature leads to the identification of several platform specific factors that influence gaze tracking accuracy. A key outcome from this review is the realization of a need to develop standardized methodologies for the performance evaluation of gaze tracking systems and achieve consistency in their specification and comparative evaluation. To address this need, the concept of a methodological framework for practical evaluation of different gaze tracking systems is proposed.

A recurring problem faced when training neural networks is that there is typically not enough data to maximize the generalization capability of deep neural networks. There are many techniques to address this, including data augmentation, dropout, and transfer learning. In this paper, we introduce an additional method, which we call smart augmentation and we show how to use it to increase the accuracy and reduce over fitting on a target network. Smart augmentation works, by creating a network that learns how to generate augmented data during the training process of a target network in a way that reduces that networks loss. This allows us to learn augmentations that minimize the error of that network. Smart augmentation has shown the potential to increase accuracy by demonstrably significant measures on all data sets tested. In addition, it has shown potential to achieve similar or improved performance levels with significantly smaller network sizes in a number of tested cases.

Accurate depth estimation is still an important challenge after a decade, particularly from stereo images. The accuracy comes from a good depth level and preserved structure. For this purpose, a depth post-processing framework is proposed in this paper. The framework starts with the “Adaptive Random Walk with Restart (2015)” algorithm. To refine the depth map generated by this method, we introduced a form of median solver/filter based on the concept of the mutual structure, which refers to the structural information in both images. This filter is further enhanced by a joint filter. Next, a transformation in image domain is introduced to remove the artifacts that cause distortion in the image. The proposed post-processing method is then compared with the top eight algorithms in the Middlebury benchmark. To explore how well this method is able to compete with more widely known techniques, a comparison is performed with Google's new depth map estimation method. The experimental results demonstrate the accuracy and efficiency of the proposed post-processing method.

In this paper, a range of open-source tools, datasets, and software that have been developed for quantitative and in-depth evaluation of eye gaze data quality are presented. Eye tracking systems in contemporary vision research and applications face major challenges due to variable operating conditions such as user distance, head pose, and movements of the eye tracker platform. However, there is a lack of open-source tools and datasets that could be used for quantitatively evaluating an eye tracker’s data quality, comparing performance of multiple trackers, or studying the impact of various operating conditions on a tracker’s accuracy. To address these issues, an open-source code repository named GazeVisual-Lib is developed that contains a number of algorithms, visualizations, and software tools for detailed and quantitative analysis of an eye tracker’s performance and data quality. In addition, a new labelled eye gaze dataset that is collected from multiple user platforms and operating conditions is presented in an open data repository for benchmark comparison of gaze data from different eye tracking systems. The paper presents the concept, development, and organization of these two repositories that are envisioned to improve the performance analysis and reliability of eye tracking systems.

Blockchain technology is ready to disrupt nearly every industry and business model, and the energy sector is no exception. Energy businesses across the world have already started exploring the use of blockchain technology in large-scale energy trading systems, peer-to-peer energy trading, project financing, supply chain tracking, and asset management among other applications. Information and Communication Technologies (ICTs) recently started revolutionizing the energy landscape, and now blockchain technology is providing an additional opportunity to make the energy system more intelligent, efficient, transparent, and secure in the longer term. The idea of this paper is to examine more closely the use of blockchain technology for its possible application in the energy efficiency industry and to determine how it could make energy efficiency markets more secure and transparent in the longer term. This paper examines in detail the key benefits and implications of using blockchain in the energy efficiency sector through the presentation and discussion of two case studies as possible blockchain applications—(i) the UK Energy Company Obligation scheme and (ii) the Italian White Certificate Scheme. We have presented how the key issues around trading energy efficiency savings—correctly estimating the savings, data transparency among stakeholders, and inefficient administrative processes—can be solved through the application of a blockchain-based smart contract system. Finally, this paper presents an implementation of a smart contract for trading of energy-saving certificates achieved via execution of smart contract transactions on the Ethereum blockchain.

An eye tracker’s accuracy and system behavior play critical roles in determining the reliability and usability of eye gaze data obtained from them. However, in contemporary eye gaze research, there exists a lot of ambiguity in the definitions of gaze estimation accuracy parameters and lack of well-defined methods for evaluating the performance of eye tracking systems. In this paper, a set of fully defined evaluation metrics are therefore developed and presented for complete performance characterization of generic commercial eye trackers, when they operate under varying conditions on desktop or mobile platforms. In addition, some useful visualization methods are implemented, which will help in studying the performance and data quality of eye trackers irrespective of their design principles and application areas. Also the concept of a graphical user interface software named GazeVisual v1.1 is proposed that would integrate all these methods and enable general users to effortlessly access the described metrics, generate visualizations and extract valuable information from their own gaze datasets. We intend to present these tools as open resources in future to the eye gaze research community for use and further advancement, as a contribution towards standardization of gaze research outputs and analysis.