Search

Steganographer detection aims to identify guilty users who conceal secret information in a number of images for the purpose of covert communication in social networks. Existing steganographer detection methods focus on designing discriminative features but do not explore relationship between image features or effectively represent users based on features. In these methods, each image is recognized as an equivalent, and each user is regarded as the distribution of all images shared by the corresponding user. However, the nuances of guilty users and innocent users are difficult to recognize with this flattened method. In this paper, the steganographer detection task is formulated as a multiple-instance learning problem in which each user is considered to be a bag, and the shared images are multiple instances in the bag. Specifically, we propose a similarity accumulation graph convolutional network to represent each user as a complete weighted graph, in which each node corresponds to features extracted from an image and the weight of an edge is the similarity between each pair of images. The constructed unit in the network can take advantage of the relationships between instances so that common patterns of positive instances can be enhanced via similarity accumulations. Instead of operating on a fixed original graph, we propose a novel strategy for reconstructing and pooling graphs based on node features to iteratively operate multiple convolutions. This strategy can effectively address oversmoothing problems that render nodes indistinguishable although they share different instance-level labels. Compared with the state-of-the-art method and other representative graph-based models, the proposed framework demonstrates its effectiveness and reliability ability across image domains, even in the context of large-scale social media scenarios. Moreover, the experimental results also indicate that the proposed network can be generalized to other multiple-instance learning problems.

While current research on multimedia is essentially dealing with the information derived from our observations of the world, internal activities inside human brains, such as imaginations and memories of past events etc., could become a brand new concept of multimedia, for which we coin as “brain-media”. In this paper, we pioneer this idea by directly applying natural images to stimulate human brains and then collect the corresponding electroencephalogram (EEG) sequences to drive a deep framework to learn and visualize the corresponding brain activities. By examining the relevance between the visualized image and the stimulation image, we are able to assess the performance of our proposed deep framework in terms of not only the quality of such visualization but also the feasibility of introducing the new concept of “brain-media”. To ensure that our explorative research is meaningful, we introduce a dually conditioned learning mechanism in the proposed deep framework. One condition is analyzing EEG sequences through deep learning to extract a more compact and class-dependent brain features via exploiting those unique characteristics of human brains such as hemispheric lateralization and biological neurons myelination (neurons importance), and the other is to analyze the content of images via computing approaches and extract representative visual features to exploit artificial intelligence in assisting our automated analysis of brain activities and their visualizations. By combining the brain feature space with the associated visual feature space of those images that are candidates of the stimuli, we are able to generate a combined-conditional space to support the proposed dual-conditioned and lateralization-supported GAN framework. Extensive experiments carried out illustrate that our proposed deep framework significantly outperforms the existing relevant work, indicating that our proposed does provide a good potential for further research upon the introduced concept of “brain-media”, a new member for the big family of multimedia. To encourage more research along this direction, we make our source codes publicly available for downloading at GitHub. 1 1 https://github.com/aneeg/LS-GAN .

BackgroundAs a physiological signal, EEG data cannot be subjectively changed or hidden. Compared with other physiological signals, EEG signals are directly related to human cortical activities with excellent temporal resolution. After the rapid development of machine learning and artificial intelligence, the analysis and calculation of EEGs has made great progress, leading to a significant boost in performances for content understanding and pattern recognition of brain activities across the areas of both neural science and computer vision. While such an enormous advance has attracted wide range of interests among relevant research communities, EEG-based classification of brain activities evoked by images still demands efforts for further improvement with respect to its accuracy, generalization, and interpretation, yet some characters of human brains have been relatively unexplored.MethodsWe propose a region-level stacked bi-directional deep learning framework for EEG-based image classification. Inspired by the hemispheric lateralization of human brains, we propose to extract additional information at regional level to strengthen and emphasize the differences between two hemispheres. The stacked bi-directional long short-term memories are used to capture the dynamic correlations hidden from both the past and the future to the current state in EEG sequences.ResultsExtensive experiments are carried out and our results demonstrate the effectiveness of our proposed framework. Compared with the existing state-of-the-arts, our framework achieves outstanding performances in EEG-based classification of brain activities evoked by images. In addition, we find that the signals of Gamma band are not only useful for achieving good performances for EEG-based image classification, but also play a significant role in capturing relationships between the neural activations and the specific emotional states.ConclusionsOur proposed framework provides an improved solution for the problem that, given an image used to stimulate brain activities, we should be able to identify which class the stimuli image comes from by analyzing the EEG signals. The region-level information is extracted to preserve and emphasize the hemispheric lateralization for neural functions or cognitive processes of human brains. Further, stacked bi-directional LSTMs are used to capture the dynamic correlations hidden in EEG data. Extensive experiments on standard EEG-based image classification dataset validate that our framework outperforms the existing state-of-the-arts under various contexts and experimental setups.

Over the past decade, “content-based” multimedia systems have realized success. By comparison, brain imaging and classification systems demand more efforts for improvement with respect to accuracy, generalization, and interpretation. The relationship between electroencephalogram (EEG) signals and corresponding multimedia content needs to be further explored. In this paper, we integrate implicit and explicit learning modalities into a context-supported deep learning framework. We propose an improved solution for the task of brain imaging classification via EEG signals. In our proposed framework, we introduce a consistency test by exploiting the context of brain images and establishing a mapping between visual-level features and cognitive-level features inferred based on EEG signals. In this way, a multimodal approach can be developed to deliver an improved solution for brain imaging and its classification based on explicit learning modalities and research from the image processing community. In addition, a number of fusion techniques are investigated in this work to optimize individual classification results. Extensive experiments have been carried out, and their results demonstrate the effectiveness of our proposed framework. In comparison with the existing state-of-the-art approaches, our proposed framework achieves superior performance in terms of not only the standard visual object classification criteria, but also the exploitation of transfer learning. For the convenience of research dissemination, we make the source code publicly available for downloading at GitHub ( https://github.com/aneeg/dual-modal-learning ).

Steganographer detection aims to identify the guilty user who utilizes steganographic methods to hide secret information in the spread of multimedia data, especially image data, from a large amount of innocent users on social networks. A true embedding probability map illustrates the probability distribution of embedding secret information in the corresponding images by specific steganographic methods and settings, which has been successfully used as the guidance for content-adaptive steganographic and steganalytic methods. Unfortunately, in real-world situation, the detailed steganographic settings adopted by the guilty user cannot be known in advance. It thus becomes necessary to propose an automatic embedding probability estimation method. In this article, we propose a novel content-adaptive steganographer detection method via embedding probability estimation. The embedding probability estimation is first formulated as a learning-based saliency detection problem and the multi-scale estimated map is then integrated into the CNN to extract steganalytic features. Finally, the guilty user is detected via an efficient Gaussian vote method with the extracted steganalytic features. The experimental results prove that the proposed method is superior to the state-of-the-art methods in both spatial and frequency domains.

Steganographer detection is to detect culprit users, who attempt to hide confidential information with steganography, among many innocent users. By incorporating the knowledge of true embedding probability map that illustrates the probability distribution of embedding messages in the corresponding image, content-adaptive steganography and steganalysis have made great progress. Unfortunately, true embedding probability map is inappropriate for steganographer detection method due to the significant challenges that the steganographic algorithm and the embedding payload are usually unknown in the task of steganographer detection. In this paper, we propose a novel content-adaptive selective steganographer detection method incorporated with learning-based embedding probability estimation. The embedding probability estimation is first formulated as a pixel-wise segmentation and recognition problem and is integrated into multi-class dilated residual learning model to extract the discriminative features. In the end, the steganographer is identified by local factor outlier with the selective strategy. Extensive experiments demonstrate that the estimated embedding probability map shows robustness against different steganographic algorithms and different payloads. From our experiments, we also find that the proposed content-adaptive selective steganographer detection framework integrated by the estimated embedding probability map achieves low detection error rates in both spatial and frequency domains.

Extracting an effective representation to analyze affection in videos is an inherent challenge. To address this problem, we present a novel emotion recognition system that can intelligently analyze and automatically recognize video emotions. We observe that most of the emotions in a video are closely related to the roles, especially those of the protagonist, which motivates us to explore how to utilize the protagonist’s information to help recognize video emotions. By analyzing the traits of the protagonists, we suggest a new solution to detect the protagonists to adapt not only to the whole video but also to a video clip. Moreover, a new keyframe selection strategy based on the protagonist is designed to select a set of representative frames from video clips. Furthermore, the scale invariant feature transform (SIFT) features matrix, built from each keyframe, is fed into a convolutional neural network (CNN) to learn the discriminative representations, which makes the CNN and local features complement each other. Considering that emotions are usually continuous, we introduce temporal information into the CNN by using optical flow images. Additionally, we extract some handcrafted visual and audio features as a supplement. Finally, all the features, including the features learned from the CNN and the handcrafted features, are fused and input into a support vector machine (SVM) and a support vector regression (SVR) for video emotion recognition. The proposed system is validated on a public dataset (LIRIS-ACCEDE) and a new dataset (PMSZU). The experimental results demonstrate the promising performance of the proposed system, which achieves better performance than the compared methods. Our designed system for recognizing video emotions could also facilitate the development of similar video expert systems, for applications such as video recommendations, video classification and video retrieval.

Video summarization has unprecedented importance to help us overview current ever-growing amount of video collections. In this paper, we propose a novel dynamic video summarization model based on deep learning architecture. We are the first to solve the imbalanced class distribution problem in video summarization. The over-sampling algorithm is used to balance the class distribution on training data. The novel two-stream deep architecture with the cost-sensitive learning is proposed to handle the class imbalance problem in feature learning. In the spatial stream, RGB images are used to represent the appearance of video frames, and in the temporal stream, multi-frame motion vectors with deep learning framework is firstly introduced to represent and extract temporal information of the input video. The proposed method is evaluated on two standard video summarization datasets and a standard emotional dataset. Empirical validations for video summarization demonstrate that our model achieves performance improvement over the existing and state-of-the-art methods. Moreover, the proposed method is able to highlight the video content with the active level of arousal in affective computing task. In addition, the proposed frame-based model has another advantage. It can automatically preserve the connection between consecutive frames. Although the summary is constructed based on the frame level, the final summary is comprised of informative and continuous segments instead of individual separate frames.

As a key component in simultaneous localization and mapping (SLAM) system, loop closure detection (LCD) eliminates the accumulated errors by recognizing previously visited places. In recent years, deep learning methods have been proved effective in LCD. However, most of the existing methods do not make good use of the useful information provided by monocular images, which tends to limit their performance in challenging dynamic scenarios with partial occlusion by moving objects. To this end, we propose a novel workflow, which is able to combine multiple information provided by images. We first introduce semantic information into LCD by developing a local-aware Class Activation Maps (CAMs) weighting method for extracting features, which can reduce the adverse effects of moving objects. Compared with previous methods based on semantic segmentation, our method has the advantage of not requiring additional models or other complex operations. In addition, we propose two effective temporal constraint strategies, which utilize the relationship of image sequences to improve the detection performance. Moreover, we propose to use the keypoint matching strategy as the final detector to further refuse false positives. Experiments on four publicly available datasets indicate that our approach can achieve higher accuracy and better robustness than the state-of-the-art methods.

Video summarization technique is a promising approach to process large-scale video by shortening the video content into a compact version. Most previous methods used recurrent networks such as Long Short-Term Memory (LSTM) even combined with attention mechanism to achieve state-of-the-art results. However, these networks are complex to implement and cannot be easily parallelized in computation. In this paper, we propose a novel deep summarization framework named Bi-Directional Self-Attention with Relative Positional Encoding for Video Summarization (BiDAVS) that can be highly parallelized. Our proposed BiDAVS considers position information of input sequence and effectively capture long-range temporal dependencies of sequential frames by computing bi-directional attention. Extensive experiments on two popular benchmark datasets, i.e., SumMe and TVSum, show that our proposed model outperforms state-of-the-art approaches.

With the proliferation of video data, video summarization is an ideal tool for users to browse video content rapidly. In this paper, we propose a novel foveated convolutional neural networks for dynamic video summarization. We are the first to integrate gaze information into a deep learning network for video summarization. Foveated images are constructed based on subjects’ eye movements to represent the spatial information of the input video. Multi-frame motion vectors are stacked across several adjacent frames to convey the motion clues. To evaluate the proposed method, experiments are conducted on two video summarization benchmark datasets. The experimental results validate the effectiveness of the gaze information for video summarization despite the fact that the eye movements are collected from different subjects from those who generated summaries. Empirical validations also demonstrate that our proposed foveated convolutional neural networks for video summarization can achieve state-of-the-art performances on these benchmark datasets.

Saliency cuts aims to segment salient objects from a given saliency map. The existing saliency cuts methods are fixed to the input cues. It limits their performance when the input cues are changed. In this paper, we propose a novel saliency cuts method named adaptive saliency cuts, which takes advantage of all the input cues in a unified framework and adjusts its components adaptively. Given a saliency map, we first generate segmentation seeds with adaptive triple thresholding. Next, we extend GrabCut by combining different input cues, and use it to generate a rough-labeled map of salient objects. Finally, we refine the boundaries of the salient objects with adaptive initialized segmentation, and produce an accurate binary mask. To the best of our knowledge, this method is the first adaptive saliency cuts method for different input cues. We validated the proposed method on MSRA10K and NJU2000. The experimental results demonstrate that our method outperforms the state-of-the-art methods.

As a fundamental preprocessing of various multimedia applications, object proposal aims to detect the candidate windows possibly containing arbitrary objects in images with two typical strategies, window scoring and grouping. In this paper, we first analyze the feasibility of improving object proposal performance by integrating window scoring and grouping strategies. Then, we propose a novel object proposal method for RGB-D images, named elastic edge boxes. The initial bounding boxes of candidate object regions are efficiently generated by edge boxes, and further adjusted by grouping the super-pixels within elastic range to obtain more accurate candidate windows. To validate the proposed method, we construct the largest RGB-D image data set NJU1800 for object proposal with balanced object number distribution. The experimental results show that our method can effectively and efficiently generate the candidate windows of object regions and it outperforms the state-of-the-art methods considering both accuracy and efficiency.

Image steganalysis is to discriminate innocent images and those suspected images with hidden messages. This task is very challenging for modern adaptive steganography, since modifications due to message hiding are extremely small. Recent studies show that Convolutional Neural Networks (CNN) have demonstrated superior performances than traditional steganalytic methods. Following this idea, we propose a novel CNN model for image steganalysis based on residual learning. The proposed Deep Residual learning based Network (DRN) shows two attractive properties than existing CNN based methods. First, the model usually contains a large number of network layers, which proves to be effective to capture the complex statistics of digital images. Second, the residual learning in DRN preserves the stego signal coming from secret messages, which is extremely beneficial for the discrimination of cover images and stego images. Comprehensive experiments on standard dataset show that the DRN model can detect the state of arts steganographic algorithms at a high accuracy. It also outperforms the classical rich model method and several recently proposed CNN based methods.

Aspect category sentiment analysis (ACSA) aims to predict the sentiment polarities of the aspect categories discussed in sentences. Since a sentence usually discusses one or more aspect categories and expresses different sentiments toward them, various attention-based methods have been developed to allocate the appropriate sentiment words for the given aspect category and obtain promising results. However, most of these methods directly use the given aspect category to find the aspect category-related sentiment words, which may cause mismatching between the sentiment words and the aspect categories when an unrelated sentiment word is semantically meaningful for the given aspect category. To mitigate this problem, we propose a Sentence Constituent-Aware Network (SCAN) for aspect-category sentiment analysis. SCAN contains two graph attention modules and an interactive loss function. The graph attention modules generate representations of the nodes in sentence constituency parse trees for the aspect category detection (ACD) task and the ACSA task, respectively. ACD aims to detect aspect categories discussed in sentences and is a auxiliary task. For a given aspect category, the interactive loss function helps the ACD task to find the nodes which can predict the aspect category but can’t predict other aspect categories. The sentiment words in the nodes then are used to predict the sentiment polarity of the aspect category by the ACSA task. The experimental results on five public datasets demonstrate the effectiveness of SCAN (Data and code can be found at https://github.com/l294265421/SCAN).

Multimedia stimulation of brain activities is not only becoming an emerging area for intensive research, but also achieved significant progresses towards classification of brain activities and interpretation of brain understanding of multimedia content. To exploit the characteristics of EEG signals in capturing human brain activities, we propose a region-dependent and attention-driven bi-directional LSTM network (RA-BiLSTM) for image evoked brain activity classification. Inspired by the hemispheric lateralization of human brains, the proposed RA-BiLSTM extracts additional information at regional level to strengthen and emphasize the differences between two hemispheres. In addition, we propose a new attentional-LSTM by adding an extra attention gate to: (i) measure and seize the importance of channel-based spatial information, and (ii) support the proposed RA-BiLSTM to capture the dynamic correlations hidden from both the past and the future in the current state across EEG sequences. Extensive experiments are carried out and the results demonstrate that our proposed RA-BiLSTM not only achieves effective classification of brain activities on evoked image categories, but also significantly outperforms the existing state of the arts.

Multi-video summarization, which tries to generate a single summary for a collection of video, is an important task in dealing with ever-growing video data. In this paper, we are the first to propose a graph convolutional network for multi-video summarization. The novel network measures the importance and relevance of each video shot in its own video as well as in the whole video collection. The important node sampling method is proposed to emphasize the effective features which are more possible to be selected as the final video summary. Two strategies are proposed to integrate into the network to solve the inherent class imbalance problem in the task of video summarization. The loss regularization for diversity is used to encourage a diverse summary to be generated. Extensive experiments are carried out, and in comparison with traditional and recent graph models and the state-of-the-art video summarization methods, our proposed model is effective in generating a representative summary for multiple videos with good diversity. It also achieves state-of-the-art performance on two standard video summarization datasets.

According to consciousness involvement, human’s learning can be roughly classified into explicit learning and implicit learning. Contrasting strongly to explicit learning with clear targets and rules, such as our school study of mathematics, learning is implicit when we acquire new information without intending to do so. Research from psychology indicates that implicit learning is ubiquitous in our daily life. Moreover, implicit learning plays an important role in human visual perception. But in the past 60 years, most of the well-known machine-learning models aimed to simulate explicit learning while the work of modeling implicit learning was relatively limited, especially for computer vision applications. This article proposes a novel unsupervised computational model for implicit visual learning by exploring dissipative system, which provides a unifying macroscopic theory to connect biology with physics. We test the proposed Dissipative Implicit Learning Model (DILM) on various datasets. The experiments show that DILM not only provides a good match to human behavior but also improves the explicit machine-learning performance obviously on image classification tasks.

Human visual system actively seeks salient regions and movements in video sequences to reduce the search effort. Computational visual saliency detection model provides important information for semantic understanding in many real world applications. In this paper, we propose a novel perception-oriented video saliency detection model to detect the attended regions for both interesting objects and dominant motions in video sequences. Based on the visual orientation inhomogeneity of human perception, a novel spatial saliency detection technique called visual orientation inhomogeneous saliency model is proposed. In temporal saliency detection, a novel optical flow model is created based on the dynamic consistency of motion. We fused the spatial and the temporal saliency maps together to build the spatio-temporal attention analysis model toward a uniform framework. The proposed model is evaluated on three typical video datasets with six visual saliency detection algorithms and achieves remarkable performance. Empirical validations demonstrate the salient regions detected by the proposed model highlight the dominant and interesting objects effectively and efficiently. More importantly, the saliency regions detected by the proposed model are consistent with human subjective eye tracking data.

Image recognition with incomplete data is a well-known hard problem in computer vision and machine learning. This article proposes a novel deep learning technique called Field Effect Bilinear Deep Networks (FEBDN) for this problem. To address the difficulties of recognizing incomplete data, we design a novel second-order deep architecture with the Field Effect Restricted Boltzmann Machine, which models the reliability of the delivered information according to the availability of the features. Based on this new architecture, we propose a new three-stage learning procedure with field effect bilinear initialization, field effect abstraction and estimation, and global fine-tuning with missing features adjustment. By integrating the reliability of features into the new learning procedure, the proposed FEBDN can jointly determine the classification boundary and estimate the missing features. FEBDN has demonstrated impressive performance on recognition and estimation tasks in various standard datasets.

Static video summarization is recognized as an effective way for users to quickly browse and comprehend large numbers of videos. In this paper, we formulate static video summarization as a clustering problem. Inspired by the idea from high density peaks search clustering algorithm, we propose an effective clustering algorithm by integrating important properties of video to gather similar frames into clusters. Finally, all clusters' center will be collected as static video summarization. Compared with existing clustering-based video summarization approaches, our work can detect frames which are highly relevant and generate representative clusters automatically. We evaluate our proposed work by comparing it with several state-of-the-art clustering-based video summarization methods and some classical clustering algorithms. The experimental results evidence that our proposed method has better performance and efficiency.

Multi-video summarization is an effective tool for users to browse multiple videos. In this paper, multi-video summarization is formulated as a graph analysis problem and a dynamic graph convolutional network is proposed to measure the importance and relevance of each video shot in its own video as well as in the whole video collection. Two strategies are proposed to solve the inherent class imbalance problem of video summarization task. Moreover, we propose a diversity regularization to encourage the model to generate a diverse summary. Extensive experiments are conducted, and the comparisons are carried out with the state-of-the-art video summarization methods, the traditional and novel graph models. Our method achieves state-of-the-art performances on two standard video summarization datasets. The results demonstrate the effectiveness of our proposed model in generating a representative summary for multiple videos with good diversity.

Despite many deep learning models are proposed for content understanding or pattern recognition of brain activities via EEGs, EEG-based object classification still demands efforts for the improvement with respect to its accuracy, generalization, and interpretation, yet some characters of human brains have been relatively unexplored. In this paper, we propose a regionlevel bi-directional deep learning framework for EEG-based object classification. Inspired by the hemispheric lateralization of human brain, we propose to extract additional information at regional level to strengthen and emphasize the differences between two hemispheres. The bi-directional long short-term memory is used to capture the dynamic correlations hidden from both the past and the future to the current state in EEG sequence. Extensive experiments are carried out and our results demonstrate the effectiveness of our proposed framework. Compared with the existing work, our framework achieves outstanding performances in EEG-based object classification task.

Steganographer detection task is to identify criminal users, who attempt to conceal confidential information by steganography methods, among a large number of innocent users. The significant challenge of the task is how to collect the evidences to identify the guilty user with suspicious images, which are embedded with secret messages generating by unknown steganography and payload. Unfortunately, existing methods for steganalysis were served for the binary classification. It makes them harder to classify the images with different kinds of payloads, especially when the payloads of images in test dataset have not been provided in advance. In this paper, we propose a novel steganographer detection method based on multiclass deep neural networks. In the training stage, the networks are trained to classify the images with six types of payloads. The networks can preserve even strengthen the weak stego signals from secret messages in much larger receptive filed by virtue of residual and dilated residual learning. In the inference stage, the learnt model is used to extract the discriminative features, which can capture the difference between guilty users and innocent users. A series of empirical experimental results demonstrate that the proposed method achieves good performance in spatial and frequency domains even though the embedding payload is low. The proposed method achieves a higher level of robustness of inter-steganographic algorithms and can provide a possible solution to address the payload mismatch problem

Object proposal aims to detect category-independent object candidates with a limited number of bounding boxes. In this paper, we propose a novel object proposal method on RGB-D images with the constraint of depth connectivity, which can improve the key techniques in grouping based object proposal effectively, including segment generation, hypothesis expansion and candidate ranking. Given an RGB-D image, we first generate segments using depth aware hierarchical segmentation. Next, we combine the segments into hypotheses hierarchically on each level, and further expand these hypotheses to object candidates using depth connectivity constrained region growing. Finally, we score the object candidates based on their color and depth features, and select the ones with the highest scores as the object proposal result. We validated the proposed method on the largest RGB-D image data set for object proposal, and our method is superior to the state-of-the-art methods.

Emotion recognition is the task of recognizing a person’s emotional state. EEG, as a physiological signal, can provide more detailed and complex information for emotion recognition task. Meanwhile, EEG can’t be changed and hidden intentionally makes EEG-based emotion recognition achieve more effective and reliable result. Unfortunately, due to the cost of data collection, most EEG datasets have small number of EEG data. The lack of data makes it difficult to predict the emotion states with the deep models, which requires enough number of training data. In this paper, we propose to use a simple data augmentation method to address the issue of data shortage in EEG-based emotion recognition. In experiments, we explore the performance of emotion recognition with the shallow and deep computational models before and after data augmentation on two standard EEG-based emotion datasets. Our experimental results show that the simple data augmentation method can improve the performance of emotion recognition based on deep models effectively.

The psychological research found that human eyes could serve as a sensitive indicator of emotional response. Pupillary response has been used to analyze the affective video content in previous studies, but the performance is not good enough. In this paper, we propose a novel method for implicit affective video tagging using pupillary response. The issue of pupil size difference between subjects has not been effectively solved, which seriously affected the performance of the implicit affective video tagging. In our method, we first define the pupil diameter baseline of each subject to diminish individual difference on pupil size. Besides, the probabilistic support vector machine (SVM) and long short term memory (LSTM) network are used to extract valuable information and output the probability estimates based on the proposed global features and sequence features obtained from the pupil dilation ratio time-series data, respectively. The final decision is made by combining the probability estimates from these two models based on the sum rule. In empirical validation, we evaluate the proposed method on a standard dataset MAHNOB-HCI. The experimental results show that the proposed method achieves better classification accuracy compared with the existing method.

Remote sensing images or images collected by unmanned aerial vehicles in the hazy weather are easily interfered by scattering effect generated by atmospheric particulate matter. The terrible interference will not only lead to the images quality seriously degraded, but also result in a bad effect on the process of images feature extraction and images feature matching. In this paper, by proposing an effective adaptive dehaze method, we compare the statistical results of feature detection and matching based on Scale-invariant feature transform (SIFT) detector and descriptor before and after haze removal. And we also provide the comparisons of image stitching task. The experimental results show that, after the haze removal is implemented on hazy images, more SIFT feature keypoints and SIFT matching keypoints will be extracted, which is also beneficial to images stitching. Moreover, the proposed adaptive method performs better than the original dehaze method.

Automatic fine-art painting classification is an important task to assist the analysis of fine-art paintings. In this paper, we propose a novel two-channel deep residual network to classify fine-art painting images. In detail, we take the advantage of the ImageNet to pre-train the deep residual network. Our two channels include the RGB channel and the brush stroke information channel. The gray-level co-occurrence matrix is used to detect the brush stroke information, which has never been considered in the task of fine-art painting classification. Experiments demonstrate that the proposed model achieves better classification performance than other models. Moreover, each stage of our model is effective for the image classification.

Video summarization is an ideal tool for skimming videos. Previous computational models extract explicit information from the input video, such as visual appearance, motion or audio information, in order to generate informative summaries. Eye gaze information, which is an implicit clue, has proved useful for indicating important content and the viewer’s interest. In this paper, we propose a novel gaze-aware deep learning model for video summarization. In our model, the position and velocity of the observers’ raw eye movements are processed by the deep neural network to indicate the users’ preferences. Experiments on two widely used video summarization datasets show that our model is more proficient than state-of-the-art methods in summarizing video for characterizing general preferences as well as for personal preferences. The results provide an innovative and improved algorithm for using gaze information in video summarization.

Deep learning based image steganalysis has attracted increasing attentions in recent years. Several Convolutional Neural Network (CNN) models have been proposed and achieved state-of-the-art performances on detecting steganography. In this paper, we explore an important technique in deep learning, the batch normalization, for the task of image steganalysis. Different from natural image classification, steganalysis is to discriminate cover images and stego images which are the result of adding weak stego signals into covers. This characteristic makes a cover image is more statistically similar to its stego than other cover images, requiring steganalytic methods to use paired learning to extract effective features for image steganalysis. Our theoretical analysis shows that a CNN model with multiple normalization layers is hard to be generalized to new data in the test set when it is well trained with paired learning. To hand this difficulty, we propose a novel normalization technique called Shared Normalization (SN) in this paper. Unlike the batch normalization layer utilizing the mini-batch mean and standard deviation to normalize each input batch, SN shares same statistics for all training and test batches. Based on the proposed SN layer, we further propose a novel neural network model for image steganalysis. Extensive experiments demonstrate that the proposed network with SN layers is stable and can detect the state of the art steganography with better performances than previous methods., submitted to IEEE Transactions on Multimedia

Image steganalysis is to discriminate innocent images and those suspected images with hidden messages. In this paper, we propose a unified Convolutional Neural Network (CNN) model for this task. In order to reliably detect modern steganographic algorithms, we design the proposed model from two aspects. For the first, different from existing CNN based steganalytic algorithms that use a predefined highpass kernel to suppress image content, we integrate the highpass filtering operation into the proposed network by building a content suppression subnetwork. For the second, we propose a novel sub-network to actively preserve the weak stego signal generated by secret messages based on residual learning, making the successive network capture the difference between cover images and stego images. Extensive experiments demonstrate that the proposed model can detect states-of-the-art steganography with much lower detection error rates than previous methods.

Recent studies have demonstrated that a well designed deep convolutional neural network (CNN) model achieves competitive performances on detecting the presence of secret message in digital images, compared with the classical rich model based steganalysis. In this paper, we propose to investigate a category of very deep CNN model–the deep residual network (DRN), for steganalysis. DRN is suitable for steganalysis from two aspects. For the first, the DRN model usually contains a large number of network layers, which proves to be effective to capture the complex statistics of digital images. For the second, DRN's residual learning (ResL) method actively strengthens the signal coming from secret messages, which is extremely beneficial for the discrimination between cover images and stego images. Comprehensive experiments on standard dataset show that the DRN model achieves very low detection error rates for the state of arts steganographic algorithms. It also outperforms the classical rich model method and several recently proposed CNN based methods.

The details of oriented visual stimuli are better resolved when they are horizontal or vertical rather than oblique. This "oblique effect" has been researched and confirmed in numerous research studies, including behavioral studies and neurophysiological and neuroimaging findings. Although the "oblique effect" has influence in many fields, little research integrated it into computational models. In this paper, we try to explore this inhomogeneity of visual orientation based on Convolutional neural networks (CNNs) in image recognition. We validate that visual orientation inhomogeneity CNNs can achieve comparable performance with higher computational efficiency on various datasets. We can also get the conclusion that, compared with the cardinal information, oblique information is indeed less useful in natural color image recognition. Through the exploration of the proposed model on image recognition, we gain more understanding of the inhomogeneity of visual orientation. It also illuminates a wide range of opportunities for integrating the inhomogeneity of visual orientation with other computational models.

Affective recognition is an important and challenging task for video content analysis. Affective information in videos is closely related to the viewer's feelings and emotions. Thus, video affective content analysis has a great potential value. However, most of the previous methods are focused on how to effectively extract features from videos for affective analysis. There are several issues are worth to be investigated. For example, what information is used to express emotions in videos, and which information is useful to affect audiences' emotions. Taking into account these issues, in this paper, we proposed a new video affective content analysis method based on protagonist information via Convolutional Neural Network CNN. The proposed method is evaluated on the largest video emotion dataset and compared with some previous work. The experimental results show that our proposed affective analysis method based on protagonist information achieves best performance in emotion classification and prediction.

Example learning-based super-resolution (SR) methods are effective to generate a high-resolution (HR) image from a single low-resolution (LR) input. And these SR methods have shown a great potential for many practical applications. Unfortunately, most of popular example learning-based approaches extract features from limited training images. These training images are insufficient for super resolution task. Our work is to transfer some supplemental information from other domains. Therefore, in this paper, a new algorithm Transfer Learning based on A+ (TLA) is proposed for image super-resolution task. First, we transfer supplemental information from other datasets to construct a new dictionary. Then, in sample selection, more training samples are supplemented to the basic training samples. In experiments, we seek to explore what types of images can provide more appropriate information for super-resolution task. Experimental results indicate that our approach is superior to A+ when transferring images containing similar content with original data.