Search

Facial action unit (AU) detection and face alignment are two highly correlated tasks, since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. However, most existing AU detection works handle the two tasks independently by treating face alignment as a preprocessing, and often use landmarks to predefine a fixed region or attention for each AU. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared feature is learned firstly, and high-level feature of face alignment is fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment feature and global feature for AU detection. Extensive experiments demonstrate that our framework (i) significantly outperforms the state-of-the-art AU detection methods on the challenging BP4D, DISFA, GFT and BP4D+ benchmarks, (ii) can adaptively capture the irregular region of each AU, (iii) achieves competitive performance for face alignment, and (iv) also works well under partial occlusions and non-frontal poses. The code for our method is available at https://github.com/ZhiwenShao/PyTorch-JAANet .

Employing deep learning-based approaches for fine-grained facial expression analysis, such as those involving the estimation of Action Unit (AU) intensities, is difficult due to the lack of a large-scale dataset of real faces with sufficiently diverse AU labels for training. In this paper, we consider how AU-level facial image synthesis can be used to substantially augment such a dataset. We propose an AU synthesis framework that combines the well-known 3D Morphable Model (3DMM), which intrinsically disentangles expression parameters from other face attributes, with models that adversarially generate 3DMM expression parameters conditioned on given target AU labels, in contrast to the more conventional approach of generating facial images directly. In this way, we are able to synthesize new combinations of expression parameters and facial images from desired AU labels. Extensive quantitative and qualitative results on the benchmark DISFA dataset and BP4D dataset demonstrate the effectiveness of our method on 3DMM facial expression parameter synthesis and data augmentation for deep learning-based AU intensity estimation. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore)

Facial expression analysis in the wild is easily vulnerable to the quality of facial images, such as low resolution or occlusion. Existing facial image restoration studies have mostly failed to take full advantage of facial expression prior information, which leads to loss of information related to facial expression in the restoration results. In this paper, we propose a multi-scale feature relation modeling GAN (MFRM-GAN) for facial expression restoration by exploring the multi-scale property and relationship of facial action units. Based on the GAN model, the MFRM-GAN integrates the graph convolution network (GCN) for feature relation modeling and feature pyramid network (FPN) for multi-scale feature extraction. Extensive qualitative and quantitative experiments on BP4D and DISFA datasets demonstrate that our proposed MFRM-GAN (i) can conduct facial expression in-painting and facial image super-resolution jointly, (ii) can recover better facial expression details comparing with state-of-the-art method in both visual effect and AU detection task.

Due to the more robust characteristics compared to unimodal, audio-video multimodal emotion recognition (MER) has attracted a lot of attention. The efficiency of representation fusion algorithm often determines the performance of MER. Although there are many fusion algorithms, information redundancy and information complementarity are usually ignored. In this paper, we propose a novel representation fusion method, Capsule Graph Convolutional Network (CapsGCN). Firstly, after unimodal representation learning, the extracted audio and video representations are distilled by capsule network and encapsulated into multimodal capsules respectively. Multimodal capsules can effectively reduce data redundancy by the dynamic routing algorithm. Secondly, the multimodal capsules with their inter-relations and intra-relations are treated as a graph structure. The graph structure is learned by Graph Convolutional Network (GCN) to get hidden representation which is a good supplement for information complementarity. Finally, the multimodal capsules and hidden relational representation learned by CapsGCN are fed to multihead self-attention to balance the contributions of source representation and relational representation. To verify the performance, visualization of representation, the results of commonly used fusion methods, and ablation studies of the proposed CapsGCN are provided. Our proposed fusion method achieves 80.83% accuracy and 80.23% F1 score on eNTERFACE05’.

To overcome the limitation of the conventional 3D face model on the synthesis of local facial expression movements, this paper proposes an improved 3D face model that combines facial action coding system (FACS) and 3D morphable model (3DMM). Our proposed 3D face model can be used for 3D facial expression synthesis with local action units (AUs). To be specific, AUs are introduced as prior knowledge into the 3D face model to capture the anatomically defined muscle movements of different facial expressions. Our proposed model extracts the parameters of a single AU, which can also be used to represent AU information to facilitate AU recognition. The qualitative and quantitative experimental results demonstrate that our proposed model can generate 3D faces with specific AU labels. These AU parameters of our proposed model perform better in AU classification than the global expression parameters of conventional 3DMM.

Convolutional neural network (CNN) based deep representation learning methods for speech emotion recognition (SER) have demonstrated great success. The basic design of CNN restricts the ability to model only local information well. Capsule network (CapsNet) can overcome the shortages of CNNs to capture the shallow global features from the spectrogram, although CapsNet cannot learn the local and deep global information. In this paper, we propose a local-global aware deep representation learning system that mainly includes two modules. One module contains a multi-scale CNN, time- frequency CNN (TFCNN) to learn the local representation. In the other module, we introduce a structure with dense connections of multiple blocks to learn shallow and deep global information. Every block in this structure is a complete CapsNet improved by a new routing algorithm. The local and global representations are fed to the classifier and achieve an absolute increase of at least 4.25% than benchmarks on IEMOCAP.

Previous studies of speech emotion recognition using either empirical features (e.g., F0, energy, and voice probability) or spectrogram-based statistical features. The empirical features can highlight the human knowledge of emotion recognition, while the statistical features enable a general representation, but they do not emphasize human knowledge sufficiently. However, the use of these two kinds of features together can complement some features that may be unconsciously used by humans in daily life but have not been realized yet. Based on this consideration, this paper proposes a dynamic fusion framework to utilize the potential advantages of the complementary spectrogram-based statistical features and the auditory-based empirical features. In addition, a kernel extreme learning machine (KELM) is adopted as the classifier to distinguish emotions. To validate the proposed framework, we conduct experiments on two public emotional databases, including Emo-DB and IEMOCAP databases. The experimental results demonstrate that the proposed fusion framework significantly outperforms the existing state-of-the-art methods. The results also show that the proposed method, by integrating the auditory-based features with spectrogram-based features, could achieve a notably improved performance over the conventional methods.

Motion segmentation plays a central role in a variety of computer vision and pattern recognition applications. In moving camera videos, the dynamic background caused by camera motion and the object motions caused by multiple moving objects are mixed and dependent with each other making the segmentation of camera motion and object motions quite difficult. This is the known dependent motion segmentation. Although motion segmentation in static scenes is a relative easy work, the dependent motion segmentation in dynamic scenes is very challenging due to the high mixture and dependence between camera motion and object motions. Recently, a large number of works have been developed for the task of dependent motion segmentation. However, their performances are still far behind human perceptions. In this paper, we studied, analyzed, and reviewed the most important and recent developed dependent motion segmentation techniques and proposed a classification strategy to categorize these techniques into different groups according to their characteristics and features. Furthermore, we also discussed the advantages and disadvantages of each kind of techniques and provided suggestions for further research directions.

Motion segmentation in moving camera videos is a very challenging task because of the motion dependence between the camera and moving objects. Camera motion compensation is recognized as an effective approach. However, existing work depends on prior-knowledge on the camera motion and scene structure for model selection. This is not always available in practice. Moreover, the image plane motion suffers from depth variations, which leads to depth-dependent motion segmentation in 3D scenes. To solve these problems, this paper develops a prior-free dependent motion segmentation algorithm by introducing a modified Helmholtz-Hodge decomposition (HHD) based object-motion oriented map (OOM). By decomposing the image motion (optical flow) into a curl-free and a divergence-free component, all kinds of camera-induced image motions can be represented by these two components in an invariant way. HHD identifies the camera-induced image motion as one segment irrespective of depth variations with the help of OOM. To segment object motions from the scene, we deploy a novel spatio-temporal constrained quadtree labeling. Extensive experimental results on benchmarks demonstrate that our method improves the performance of the state-of-the-art by 10%~20% even over challenging scenes with complex background.

With the supervision from source domain only in class-level, existing unsupervised domain adaptation (UDA) methods mainly learn the domain-invariant representations from a shared feature extractor, which causes the source-bias problem. This paper proposes an unsupervised domain adaptation approach with Teacher-Student Competition (TSC). In particular, a student network is introduced to learn the target-specific feature space, and we design a novel competition mechanism to select more credible pseudo-labels for the training of student network. We introduce a teacher network with the structure of existing conventional UDA method, and both teacher and student networks compete to provide target pseudo-labels to constrain every target sample's training in student network. Extensive experiments demonstrate that our proposed TSC framework significantly outperforms the state-of-the-art domain adaptation methods on Office-31 and ImageCLEF-DA benchmarks., Comment: Accepted by ICPR 2020

Most existing AU detection works considering AU relationships are relying on probabilistic graphical models with manually extracted features. This paper proposes an end-to-end deep learning framework for facial AU detection with graph convolutional network (GCN) for AU relation modeling, which has not been explored before. In particular, AU related regions are extracted firstly, latent representations full of AU information are learned through an auto-encoder. Moreover, each latent representation vector is feed into GCN as a node, the connection mode of GCN is determined based on the relationships of AUs. Finally, the assembled features updated through GCN are concatenated for AU detection. Extensive experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for facial AU detection. The proposed framework is also validated through a series of ablation studies.

Facial expression synthesis or editing has recently received increasing attention in the field of affective computing and facial expression modeling. However, most existing facial expression synthesis works are limited in paired training data, low resolution, identity information damaging, and so on. To address those limitations, this paper introduces a novel Action Unit (AU) level facial expression synthesis method called Local Attentive Conditional Generative Adversarial Network (LAC-GAN) based on face action units annotations. Given desired AU labels, LAC-GAN utilizes local AU regional rules to control the status of each AU and attentive mechanism to combine several of them into the whole photo-realistic facial expressions or arbitrary facial expressions. In addition, unpaired training data is utilized in our proposed method to train the manipulation module with the corresponding AU labels, which learns a mapping between a facial expression manifold. Extensive qualitative and quantitative evaluations are conducted on commonly used BP4D dataset to verify the effectiveness of our proposed AU synthesis method.

The use of a convolutional neural network (CNN) for extracting deep acoustic features from spectrograms has become one of the most commonly used methods for speech emotion recognition. In those studies, however, common amplitude information is chosen as input with no special attention to phase-related or speaker-related information. In this paper, we propose a multi-channel method employing amplitude and phase channels for speech emotion recognition. Two separate CNN channels are adopted to extract deep features from amplitude spectrograms and modified group delay (MGD) spectrograms. Then a concatenate layer is used to fuse the features. Furthermore, to gain more robust features, speaker information is considered in the stage of emotional feature extraction. Finally, the fusion features that considering speaker-related information are fed into the extreme learning machine (ELM) to distinguish emotions. Experiments are conducted on the Emo-DB database to evaluate the proposed model. Results demonstrate the recognition performance of average F1 in 94.82%, which significantly outperforms the baseline CNN-ELM model based on amplitude only spectrograms by 39.27% relative error reduction.

Facial expression analysis in the wild is challenging when the facial image is with low resolution or partial occlusion. Considering the correlations among different facial local regions under different facial expressions, this paper proposes a novel facial expression restoration method based on generative adversarial network by integrating an improved graph convolutional network (IGCN) and region relation modeling block (RRMB). Unlike conventional graph convolutional networks taking vectors as input features, IGCN can use tensors of face patches as inputs. It is better to retain the structure information of face patches. The proposed RRMB is designed to address facial generative tasks including inpainting and super-resolution with facial action units detection, which aims to restore facial expression as the ground-truth. Extensive experiments conducted on BP4D and DISFA benchmarks demonstrate the effectiveness of our proposed method through quantitative and qualitative evaluations.

Learning efficient deep representations from spectrogram for speech emotion recognition still represents a significant challenge. Most existing spectrogram feature extraction methods empowered by deep learning have demonstrated great success, but the respective changing information of time and frequency exhibited by the spectrogram is ignored. In this paper, a speech emotion recognition method integrating self-attention is proposed by considering the interactive and respective changing information of time and frequency. To learn the deep representations from spectrogram, a time-frequency convolutional neural network (TFCNN) is proposed at first. After that, a Multi-head Self-attention layer inspired by Transformer proposed by Google is introduced to fuse deep representations more efficiently. Finally, extreme learning machine (ELM) and bidirectional long short term memory (BLSTM) models are adopted as emotion classifiers. Experiments conducted on IEMOCAP dataset demonstrate the effectiveness of our proposed methods showing better visual illustrations and classification results.

In recent years, the deep learning based face inpainting methods have achieved some promising results, mainly related to the generative adversarial networks (GAN). Based on a large number of training samples, GAN generates a false true from a known training sample, and cannot use the trained parameters to generate image other than training samples. In addition, most of the previous works did not take into account high-level facial structure information, such as the co-existence of facial action units. In order to better to exploit facial structural knowledge, this paper proposes a method that combines prior knowledge based on high-level dynamic structural information of facial action units and GAN to complete face images. We primarily validate the effectiveness of our approach in facial expression restoration during face inpainting on the two datasets of BP4D and DISFA.

Attention mechanism has recently attracted increasing attentions in the field of facial action unit (AU) detection. By finding the region of interest of each AU with the attention mechanism, AU-related local features can be captured. Most of the existing attention based AU detection works use prior knowledge to predefine fixed attentions or refine the predefined attentions within a small range, which limits their capacity to model various AUs. In this paper, we propose an end-to-end deep learning based attention and relation learning framework for AU detection with only AU labels, which has not been explored before. In particular, multi-scale features shared by each AU are learned firstly, and then both channel-wise and spatial attentions are adaptively learned to select and extract AU-related local features. Moreover, pixel-level relations for AUs are further captured to refine spatial attentions so as to extract more relevant local features. Without changing the network architecture, our framework can be easily extended for AU intensity estimation. Extensive experiments show that our framework (i) soundly outperforms the state-of-the-art methods for both AU detection and AU intensity estimation on the challenging BP4D, DISFA, FERA 2015 and BP4D+ benchmarks, (ii) can adaptively capture the correlated regions of each AU, and (iii) also works well under severe occlusions and large poses., Comment: This paper is accepted by IEEE Transactions on Affective Computing

Facial action unit (AU) detection and face alignment are two highly correlated tasks since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. Most existing AU detection works often treat face alignment as a preprocessing and handle the two tasks independently. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared features are learned firstly, and high-level features of face alignment are fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment features and global features for AU detection. Experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for AU detection.

Recently, increasing attention has been directed to the study of the speech emotion recognition, in which global acoustic features of an utterance are mostly used to eliminate the content differences. However, the expression of speech emotion is a dynamic process, which is reflected through dynamic durations, energies, and some other prosodic information when one speaks. In this paper, a novel local dynamic pitch probability distribution feature, which is obtained by drawing the histogram, is proposed to improve the accuracy of speech emotion recognition. Compared with most of the previous works using global features, the proposed method takes advantage of the local dynamic information conveyed by the emotional speech. Several experiments on Berlin Database of Emotional Speech are conducted to verify the effectiveness of the proposed method. The experimental results demonstrate that the local dynamic information obtained with the proposed method is more effective for speech emotion recognition than the traditional global features.

Facial expression recognition (FER) for emotion inference has become one of the most important research fields in human-computer interaction. Existing study on FER mainly focuses on visible images, whereas varying lighting conditions may influence their performances. Recent studies have demonstrated the advantages of infrared thermal images reflecting the temperature distributions, which are robust to lighting changes. In this paper, a novel infrared image sequence based FER method is proposed using spatiotemporal feature analysis and deep Boltzmann machines (DBM). Firstly, a dense motion field among infrared image sequences is generated using optical flow algorithm. Then, PCA is applied for dimension reduction and a three-layer DBM structure is designed for final expression classification. Finally, the effectiveness of the proposed method is well demonstrated based on several experiments conducted on NVIE database.

In this paper, a novel approach of implicit multiple emotional video tagging is proposed, which considers the relations between the users' facial expressions and emotions as well as the relations among multiple expressions. First, the audiences' expressions are inferred through a multi-expression recognition model, which consists of an image-driven expression measurement recognition and a Bayesian network representing the co-existence and mutual exclusion relations among multi-expressions. Second, the videos' multi-emotion tags are obtained from the recognized expressions by another Bayesian Network, capturing the relations between expressions and emotions. Results of the experiments conducted on the JAFFE and NVIE databases demonstrate that the performance of expression recognition is improved by considering the relations among multiple expressions. Furthermore, the relations between expressions and emotions help improve emotional tagging, as our approach outperforms the traditional expression-based or image-driven implicit tagging methods.

In this paper, we propose a novel implicit video emotion tagging approach by exploring the relationships between videos' common emotions, subjects' individualized emotions and subjects' outer facial expressions. First, head motion and face appearance features are extracted. Then, the spontaneous facial expressions of subjects are recognized by Bayesian networks. After that, the relationships between the outer facial expressions, the inner individualized emotions and the video's common emotions are captured by another Bayesian network, which can be used to infer the emotional tags of videos. To validate the effectiveness of our approach, an emotion tagging experiment is conducted on the NVIE database. The experimental results show that head motion features improve the performance of both facial expression recognition and emotion tagging, and that the captured relations between the outer facial expressions, the inner individualized emotions and the common emotions improve the performance of common and individualized emotion tagging.

By using the knowledge of facial structure and temperature distribution, this paper proposes an automatic eye localization method from long wave infrared thermal images both with eyeglasses and without eyeglasses. First, with the help of support vector machine classifier, three gray-projection features are defined to determine whether a subject is with eyeglasses. For subjects with eyeglasses, the locations of valleys in the projection curve are used to perform eye localization. For subjects without eyeglasses, a facial structure consisting of 15 sub-regions is proposed to extract Haar-like features. Eight classifiers are learned from the features selected by Adaboost algorithm for left and right eye, respectively. A vote strategy is employed to find the most likely eyes. To evaluate the effectiveness of our approach, experiments are performed on NVIE and Equinox databases. The eyeglass detection results on NVIE database and Equinox database are 99.36% and 95%, respectively, which demonstrate the effectiveness and robustness of our eyeglass detection method. Eye localization results of within-database experiments and cross-database experiments on these two databases are very comparable with the previous results in this field, verifying the effectiveness and the generalization ability of our approach.

Directions into Velocities of Articulators (DIVA) model is a kind of self-adaptive neural network model which controls movements of a simulated vocal tract to produce words, syllables or phonemes. However, DIVA model lacks of emotion functions. To implement the emotion function in DIVA model, we investigate the process of affective speech production based on the combination of fundamental frequency (F0) and formant frequencies, as well as the relations between F0 and formants of emotional speech. The relations between F0 and formants of the speech with different emotions are investigated using the logistic regression (LR) models on the emotional databases. For a given emotion-related F0, the formants can be predicted correctly using the LR models. An affective speech synthesizer was constructed by implementing the relation of F0 and formants in an improved formant synthesis method. Experiments on affective speech synthesis were conducted on three different emotional speech datasets, and affective speech with negative or positive emotion can also be effectively synthesized from neutral speech.

To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In this paper, we propose and establish a natural visible and infrared facial expression database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible and an infrared thermal camera, with illumination provided from three different directions. The posed database includes the apex expressional images with and without glasses. As an elementary assessment of the usability of our spontaneous database for expression recognition and emotion inference, we conduct visible facial expression recognition using four typical methods, including the eigenface approach [principle component analysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the AAM-based + LDA. We also use PCA and PCA+LDA to recognize expressions from infrared thermal images. In addition, we analyze the relationship between facial temperature and emotion through statistical analysis. Our database is available for research purposes.

Kinematic analysis of the digits using optical motion capture systems relies on defining accurate coordinate systems for the individual segments. Limitations of previous digit kinematic protocols include marker placement errors, marker occlusion and superimposition, and skin movement artifact. The purpose of this study was to develop a protocol utilizing a digit alignment device (DAD) and nail marker clusters to overcome these limitations. Ten subjects underwent 10 static calibration trials for validation. The orientation of the thumb distal phalange relative to the index finger distal phalange was described using Euler angles of pitch( x ), yaw( y ′), and roll( z ′′). The digit calibration protocol demonstrated high accuracy (0.5°, 1.9° and 2.2° for x , y ′, z ′′) and precision (1.4°, 2.3° and 3.1° for x , y ′, z ′′). The developed protocol provided convenient identification of transformations that determine anatomically relevant coordinate systems for the distal phalanges of the digits. The potential of utilizing this protocol as a standardized tool for digit kinematics was demonstrated using a dynamic task of precision pinching.

In multi-label learning, each sample can be assigned to multiple class labels simultaneously. In this work, we focus on the problem of multi-label learning with missing labels (MLML), where instead of assuming a complete label assignment is provided for each sample, only partial labels are assigned with values, while the rest are missing or not provided. The positive (presence), negative (absence) and missing labels are explicitly distinguished in MLML. We formulate MLML as a transductive learning problem, where the goal is to recover the full label assignment for each sample by enforcing consistency with available label assignments and smoothness of label assignments. Along with an exact solution, we also provide an effective and efficient approximated solution. Our method shows much better performance than several state-of-the-art methods on several benchmark data sets.

In this paper, a novel implicit video multi-emotion tagging method is proposed, which considers the relations between the users' outer facial expressions and inner emotions as well as the relations among multiple expressions. First, the audiences' expressions are inferred through a multi-expression recognition model, which consists of an image driven expression measurement recognition and a Bayesian network representing the co-existence and mutual exclusion relations among multi-expressions. Second, the videos' multi-emotion tags are obtained from the recognized expressions by another Bayesian Network, capturing the relations between outer expressions and inner emotions. Results of the experiments conducted on the NVIE database demonstrate that multi-expression recognition considering the relations among expressions improves the recognition performance. Furthermore, the performance of our proposed emotion tagging method considering the relations between outer expressions and inner emotions outperforms the traditional expression-based or image-based implicit video emotion tagging methods.

By using the knowledge of facial structure and temperature distribution, this paper proposes an automatic eye localization method from infrared thermal images. A facial structure consisting of 15 sub-regions is proposed to extract Haar-like features. Eight classifiers are learned from the features selected by Adaboost algorithm for left and right eye, respectively. A vote strategy is used to find the most likely eyes. Experimental results on the NVIE and Equinox databases show the effectiveness of our approach.

This paper proposes an approach of facial expression recognition from infrared thermal images by using temperature difference features and voting strategy. Firstly, three kinds of temperature features named horizontal, vertical and sequential difference grid features are introduced and extracted from the thermal images of four facial regions. Secondly, K-Nearest Neighbor is used as a classifier in each facial region. After that, a voting strategy is used as the decision-level fusion. Experiments on a large scale infrared thermal expression database achieve around 61.62% recognition rate. The comparative experiment results suggest that face-region-based facial expression classification using the temperature difference features is feasible, and demonstrate that difference grid features are independent and insensitive to individual or environment. The voting results provide the evidence that our face-region-based voting strategy using infrared thermal images for facial expression recognition is reliable and effective.

Purpose: To create digital phantoms using paired planning and replanning CTs from patients with H'N cancer and to comprehensively evaluate three commonly-used deformable image registration (DIR) algorithms with known ROIs and volumetric doses from the established phantoms. Methods: We have identified five H'N patients with adaptive-IMRT. For each patient, manually-adjusted meshes were created for selected physicianapproved ROIs. These meshes were applied to drive the thin-plate spline algorithm to deform the planning CT to match the replanning CT. The digital phantom consisted of the planning CT and simulated replanning CT, which were linked by a known deformation vector field (DVF). Three DIR algorithms (Fast-Symmetric Demons, B-Spline, and intensity-based) were applied to these five digital phantoms. The images, ROIs, and volumetric doses were mapped from the planning CT to the simulated replanning CT using the DVFs computed by these three DIRs and compared to those using the known DVF. Results: The average normalized cross-correlation (NCC) value between planning and replanning CTs with rigid registration was 0.924, indicating large anatomical changes. The NCC values were improved to 0.998, 0.974, and 0.996, with Demons, B-Spline, and intensity-based DIRs, respectively. The mean Dice coefficients for selected ROIs (e.g. PTV, brainstem, spinal cord, mandible, parotids) were in the range of 0.87-0.95, 0.84–0.94, and 0.85–0.95 for Demons, B-Spline, and intensity-based DIRs, respectively. The mean slice-wise Hausdorff distances for selected ROIs were in the range of 1.8–4.9 mm, 2.2–5.4 mm, and 2.4–6.6 mm for Demons, B-Spline, and intensity-based DIRs, respectively. The average percentage of voxels in the external contours that have dose errors

The 2010 Silent Speech Challenge benchmark is updated with new results obtained in a Deep Learning strategy, using the same input features and decoding strategy as in the original article. A Word Error Rate of 6.4% is obtained, compared to the published value of 17.4%. Additional results comparing new auto-encoder-based features with the original features at reduced dimensionality, as well as decoding scenarios on two different language models, are also presented. The Silent Speech Challenge archive has been updated to contain both the original and the new auto-encoder features, in addition to the original raw data., Comment: 25 pages, 6 pages