Search

Emotion recognition from EEG signals has attracted much attention in affective computing. Recently, a novel dynamic graph convolutional neural network (DGCNN) model was proposed, which simultaneously optimized the network parameters and a weighted graph G characterizing the strength of functional relation between each pair of two electrodes in the EEG recording equipment. In this paper, we propose a sparse DGCNN model which improves the DGCNN by imposing a sparseness constraint on G. Our work is based on an important observation: the tomography study reveals that different brain regions sampled by EEG electrodes may be related to different functions of the brain and then the functional relations among electrodes are possibly highly localized and sparse. However, introducing sparseness constraint into the graph G makes the loss function of sparse DGCNN non-differentiable at some singular points. To ensure that the training process of sparse DGCNN converges, we apply the forward-backward splitting method. To evaluate the performance of sparse DGCNN, we compare it with four representative recognition methods as well as different features and spectral bands. The results show that (1) sparse DGCNN has consistently better accuracy than representative methods and has a good scalability, and (2) DE, PSD and ASM features on γ bands convey most discriminative emotional information, and fusion of separate features and frequency bands can improve recognition performance.

Compared with the widely studied negative emotions in which different classes are easy to distinguish, nowadays less attention is paid to the recognition of positive emotions that are not fully independent. In this paper, we propose to recognize multiple positive emotions by analyzing brain activities and explore the neural representation of different positive emotions. Thirty-seven participants volunteered to participate in our study, in which their brain activities were recorded when watching five selected film clips. First, 150 well-known power features extracted from Electroencephalography (EEG) signals and 105 multimedia content analysis features were collected as the pool of candidate features. Second, based on the collected features, we propose to use a linear model and a nonlinear model to predict the percentage of five positive emotions. Then, percentage values were converted to ranking numbers and Kendall rank correlation coefficients were calculated. Our results showed that (1) ensemble of regressor chains using LSTM as unit regressor obtained both the best regression results and the best Kendall rank correlation coefficient on EEG features merely, and (2) top features from alpha frequency bands of EEG signals could represent different positive emotions. These results demonstrate the effectiveness of selective EEG features on recognizing different positive emotions.

Most supervised learning-based pose estimation methods for stacked scenes are trained on massive synthetic datasets. In most cases, the challenge is that the learned network on the training dataset is no longer optimal on the testing dataset. To address this problem, we propose a pose regression network PPR-Net++. It transforms each scene point into a point in the centroid space, followed by a clustering process and a voting process. In the training phase, a mapping function between the network's critical parameter (i.e., the bandwidth of the clustering algorithm) and the compactness of the centroid distributions is obtained. This function is used to adapt the bandwidth between centroid distributions of two different domains. In addition, to further improve the pose estimation accuracy, the network also predicts the confidence of each point, based on its visibility and pose error. Only the points with high confidence have the right to vote for the final object pose. In experiments, our method is trained on the IPA synthetic dataset and compared with the state-of-the-art algorithm. When tested with the public synthetic Sileane dataset, our method is better in all eight objects, where five of them are improved by more than 5% in average precision (AP). On IPA real dataset, our method outperforms a large margin by 20%. This lays a solid foundation for robot grasping in industrial scenarios.

Cartoon is a common form of art in our daily life and automatic generation of cartoon images from photos is highly desirable. However, state-of-the-art single-style methods can only generate one style of cartoon images from photos and existing multi-style image style transfer methods still struggle to produce high-quality cartoon images due to their highly simplified and abstract nature. In this article, we propose a novel multi-style generative adversarial network (GAN) architecture, called MS-CartoonGAN, which can transform photos into multiple cartoon styles. MS-CartoonGAN uses only unpaired photos and cartoon images of multiple styles for training. To achieve this, we propose to use (1) a hierarchical semantic loss with sparse regularization to retain semantic content and recover flat shading in different abstract levels, (2) a new edge-promoting adversarial loss for producing fine edges, and (3) a style loss to enhance the difference between output cartoon styles and make training process more stable. We also develop a multi-domain architecture, where the generator consists of a shared encoder and multiple decoders for different cartoon styles, along with multiple discriminators for individual styles. By observing that cartoon images drawn by different artists have their unique styles while sharing some common characteristics, our shared network architecture exploits the common characteristics of cartoon styles, achieving better cartoonization and being more efficient than single-style cartoonization. We show that our multi-domain architecture can theoretically guarantee to output desired multiple cartoon styles. Through extensive experiments including a user study, we demonstrate the superiority of the proposed method, outperforming state-of-the-art single-style and multi-style image style transfer methods.

Most previous EEG-based emotion recognition methods studied hand-crafted EEG features extracted from different electrodes. In this paper, we study the relation among different EEG electrodes and propose a deep learning method to automatically extract the spatial features that characterize the functional relation between EEG signals at different electrodes. Our proposed deep model is called ATtention-based LSTM with Domain Discriminator (ATDD-LSTM) that can characterize nonlinear relations among EEG signals of different electrodes. To achieve state-of-the-art emotion recognition performance, the architecture of ATDD-LSTM has two distinguishing characteristics: (1) By applying the attention mechanism to the feature vectors produced by LSTM, ATDD-LSTM automatically selects suitable EEG channels for emotion recognition, which makes the learned model concentrate on the emotion related channels in response to a given emotion; (2) To minimize the significant feature distribution shift between different sessions and/or subjects, ATDD-LSTM uses a domain discriminator to modify the data representation space and generate domain-invariant features. We evaluate the proposed ATDD-LSTM model on three public EEG emotional databases (DEAP, SEED and CMEED) for emotion recognition. The experimental results demonstrate that our ATDD-LSTM model achieves superior performance on subject-dependent (for the same subject), subject-independent (for different subjects) and cross-session (for the same subject) evaluation.

This article presents a simple yet effective algorithm for automatically transferring face colors in portrait videos. We extract the facial features and vectorize the faces in the input video using Poisson vector graphics, which encodes the low-frequency colors as the boundary colors of diffusion curves, and the high-frequency colors as Poisson regions. Then, we transfer the face color of a reference image/video to the first frame of the input video by applying optimal mass transport between the boundary colors of diffusion curves. Next the boundary color of the first frame is transferred to the subsequent frames by matching the curves. Finally, with the original or modified Poisson regions, we render the video using an efficient random-access Poisson solver. Thanks to our efficient diffusion curve matching algorithm, transferring colors for the vectorized video takes less than 1 millisecond per frame. Our method is particularly desired for frequent transfer from multiple references due to its information reuse nature. The simple diffusion curve matching also greatly improves the performance of video vectorization, since we only need to solve an optimization problem for the first frame. Since our method does not require correspondence between the reference image/video and the input video, it is flexible and robust to handle faces with significantly different geometries and postures, which often pose challenges to the existing methods. Moreover, by manipulating Poisson regions, we can enhance or reduce the highlight and contrast so that the reference color can fit into the input video naturally. We demonstrate the efficacy of our method on image-to-video transfer and color swap in videos.

Caricature is a type of artistic style of human faces that attracts considerable attention in the entertainment industry. So far a few 3D caricature generation methods exist and all of them require some caricature information (e.g., a caricature sketch or 2D caricature) as input. This kind of input, however, is difficult to provide by non-professional users. In this paper, we propose an end-to-end deep neural network model that generates high-quality 3D caricatures directly from a normal 2D face photo. The most challenging issue for our system is that the source domain of face photos (characterized by normal 2D faces) is significantly different from the target domain of 3D caricatures (characterized by 3D exaggerated face shapes and textures). To address this challenge, we: (1) build a large dataset of 5,343 3D caricature meshes and use it to establish a PCA model in the 3D caricature shape space; (2) reconstruct a normal full 3D head from the input face photo and use its PCA representation in the 3D caricature shape space to establish correspondences between the input photo and 3D caricature shape; and (3) propose a novel character loss and a novel caricature loss based on previous psychological studies on caricatures. Experiments including a novel two-level user study show that our system can generate high-quality 3D caricatures directly from normal face photos., Comment: Accepted by IEEE Transactions on Visualization and Computer Graphics

Video over-segmentation into supervoxels is an important pre-processing technique for many computer vision tasks. Videos are an order of magnitude larger than images. Most existing methods for generating supervovels are either memory- or time-inefficient, which limits their application in subsequent video processing tasks. In this paper, we present an anisotropic supervoxel method, which is memory-efficient and can be executed on the graphics processing unit (GPU). Therefore, our algorithm achieves good balance among segmentation quality, memory usage and processing time. In order to provide accurate segmentation for moving objects in video, we use the optical flow information to design a brand new non-Euclidean metric to calculate the anisotropic distances between seeds and voxels. To efficiently compute the anisotropic metric, we adjust the classic jump flooding algorithm (which is designed for parallel execution on the GPU) to generate anisotropic Voronoi tessellation in the combined color and spatio-temporal space. We evaluate our method and the representative supervoxel algorithms for their capability on segmentation performance, computation speed and memory efficiency. We also apply supervoxel results to the application of foreground propagation in videos to test the performance on solving practical problems. Experiments show that our algorithm is much faster than the existing methods, and achieves good balance on segmentation quality and efficiency.

Despite significant effort and notable success of neural style transfer, it remains challenging for highly abstract styles, in particular line drawings. In this paper, we propose APDrawingGAN++, a generative adversarial network (GAN) for transforming face photos to artistic portrait drawings (APDrawings), which addresses substantial challenges including highly abstract style, different drawing techniques for different facial features, and high perceptual sensitivity to artifacts. To address these, we propose a composite GAN architecture that consists of local networks (to learn effective representations for specific facial features) and a global network (to capture the overall content). We provide a theoretical explanation for the necessity of this composite GAN structure by proving that any GAN with a single generator cannot generate artistic styles like APDrawings. We further introduce a classification-and-synthesis approach for lips and hair where different drawing styles are used by artists, which applies suitable styles for a given input. To capture the highly abstract art form inherent in APDrawings, we address two challenging operations — (1) coping with lines with small misalignments while penalizing large discrepancy and (2) generating more continuous lines — by introducing two novel loss terms: one is a novel distance transform loss with nonlinear mapping and the other is a novel line continuity loss, both of which improve the line quality. We also develop dedicated data augmentation and pre-training to further improve results. Extensive experiments, including a user study, show that our method outperforms state-of-the-art methods, both qualitatively and quantitatively.

What makes speeches effective has long been a subject for debate, and until today there is broad controversy among public speaking experts about what factors make a speech effective as well as the roles of these factors in speeches. Moreover, there is a lack of quantitative analysis methods to help understand effective speaking strategies. In this paper, we propose E-ffective, a visual analytic system allowing speaking experts and novices to analyze both the role of speech factors and their contribution in effective speeches. From interviews with domain experts and investigating existing literature, we identified important factors to consider in inspirational speeches. We obtained the generated factors from multi-modal data that were then related to effectiveness data. Our system supports rapid understanding of critical factors in inspirational speeches, including the influence of emotions by means of novel visualization methods and interaction. Two novel visualizations include E-spiral (that shows the emotional shifts in speeches in a visually compact way) and E-script (that connects speech content with key speech delivery information). In our evaluation we studied the influence of our system on experts' domain knowledge about speech factors. We further studied the usability of the system by speaking novices and experts on assisting analysis of inspirational speech effectiveness., Comment: IEEE Transactions of Visualization and Computer Graphics (TVCG, Proc. VIS 2021), to appear

Motion style transfer is an important problem in many computer graphics and computer vision applications, including human animation, games, and robotics. Most existing deep learning methods for this problem are supervised and trained by registered motion pairs. In addition, these methods are often limited to yielding a deterministic output, given a pair of style and content motions. In this paper, we propose an unsupervised approach for motion style transfer by synthesizing stylized motions autoregressively using a generative flow model $\mathcal{M}$. $\mathcal{M}$ is trained to maximize the exact likelihood of a collection of unlabeled motions, based on an autoregressive context of poses in previous frames and a control signal representing the movement of a root joint. Thanks to invertible flow transformations, latent codes that encode deep properties of motion styles are efficiently inferred by $\mathcal{M}$. By combining the latent codes (from an input style motion S) with the autoregressive context and control signal (from an input content motion C), $\mathcal{M}$ outputs a stylized motion which transfers style from S to C. Moreover, our model is probabilistic and is able to generate various plausible motions with a specific style. We evaluate the proposed model on motion capture datasets containing different human motion styles. Experiment results show that our model outperforms the state-of-the-art methods, despite not requiring manually labeled training data.

Background and Aim: The temporizing treatment for spontaneous pneumothorax had been established methodically, but thoracoscopic surgical methods has been not yet. The objective of the study is to show the treatment effect of intra- and post-operation by using rational thoracoscopic surgery for the patients with spontaneous pneumothorax. Methods: We had performed Video Assisted Thoracoscopic Surgery (VATS) and Mini Invasive Thoracoscopic Surgery (MITS) for 322 patients with spontaneous pneumothorax and established rational surgery approach between January 2011 and December 2017. And then we evaluated the outcome of surgery compared to common thoracotomy. Results: Compared to common thoracotomy, VATS was an operation with large economic efficacy because of light operation associated injury, fast recovery and the ability to decrease temporary disability rate. And also it saved different drugs and provided Quality of life (QOL) because of low operative cicatrix. Conclusion: VATS not only gave light operative burden to patients but also decreased surgeons’ mental and physical fatigue. In the MITS automatic suture tool with expensive cost and limited indications was not used and the range of indications were wide and operating time was shortened because of procedure under direct visual field associated with thoracoscopy.

Most industrial parts are parametric and their special properties are not fully explored yet. This paper proposes a new 6DoF pose estimation network for parametric shapes in stacked scenarios (ParametricNet). It treats a parametric shape, instead of a part object, as a category. The keypoints of individual instances are learned with point- wise regression and Hough voting scheme, from which specific parameter values are calculated. Then, the template keypoints are obtained based on the computed parameter values and the parametric shape templates. Finally, the 6DoF pose is estimated by least-square fitting between the individual instance’s and the template’s keypoints & centroid. On the public Sileane dataset, the average of APs of ParametricNet is 96%, compared with 82% for the state-of-the-art method. In addition, a new parametric dataset with four shape templates is constructed, in which the evaluated learning and generalization abilities of ParametricNet outperform the state-of-the-art methods. In particular, for the less symmetric shape, the mAP is improved by over 20%, which is an obvious improvement. Real-world experiments show that our method can grasp parametric shapes with unknown parameter values in stacked scenarios.

Motion planning for robots of high degrees-of-freedom (DOFs) is an important problem in robotics with sampling-based methods in configuration space $\mathcal{C}$ as one popular solution. Recently, machine learning methods have been introduced into sampling-based motion planning methods, which train a classifier to distinguish collision free subspace from in-collision subspace in $\mathcal{C}$. In this paper, we propose a novel configuration space decomposition method and show two nice properties resulted from this decomposition. Using these two properties, we build a composite classifier that works compatibly with previous machine learning methods by using them as the elementary classifiers. Experimental results are presented, showing that our composite classifier outperforms state-of-the-art single-classifier methods by a large margin. A real application of motion planning in a multi-robot system in plant phenotyping using three UR5 robotic arms is also presented.

Portrait drawing is a common form of art with high abstraction and expressiveness. Due to its unique characteristics, existing methods achieve decent results only with paired training data, which is costly and time-consuming to obtain.In this paper, we address the problem of automatic transfer from face photos to portrait drawings with unpaired training data. We observe that due to the significant imbalance of information richness between photos and drawings, existing unpaired transfer methods such as CycleGAN tends to embed invisible reconstruction information indiscriminately in the whole drawings, leading to important facial features partially missing in drawings. To address this problem, we propose a novel asymmetric cycle mapping that enforces the reconstruction information to be visible (by a truncation loss) and only embedded in selective facial regions (by a relaxed forward cycle-consistency loss). Along with localized discriminators for the eyes, nose and lips, our method well preserves all important facial features in the generated portrait drawings. By introducing a style classifier and taking the style vector into account, our method can learn to generate portrait drawings in multiple styles using a single network. Extensive experiments show that our model outperforms state-of-the-art methods.

Over-segmenting a video into supervoxels has strong potential to reduce the complexity of downstream computer vision applications. Content-sensitive supervoxels (CSSs) are typically smaller in content-dense regions (i.e., with high variation of appearance and/or motion) and larger in content-sparse regions. In this paper, we propose to compute feature-aware CSSs (FCSSs) that are regularly shaped 3D primitive volumes well aligned with local object/region/motion boundaries in video. To compute FCSSs, we map a video to a 3D manifold embedded in a combined color and spatiotemporal space, in which the volume elements of video manifold give a good measure of the video content density. Then any uniform tessellation on video manifold can induce CSS in the video. Our idea is that among all possible uniform tessellations on the video manifold, FCSS finds one whose cell boundaries well align with local video boundaries. To achieve this goal, we propose a novel restricted centroidal Voronoi tessellation method that simultaneously minimizes the tessellation energy (leading to uniform cells in the tessellation) and maximizes the average boundary distance (leading to good local feature alignment). Theoretically our method has an optimal competitive ratio $O(1)$ O ( 1 ) , and its time and space complexities are $O(NK)$ O ( N K ) and $O(N+K)$ O ( N + K ) for computing $K$ K supervoxels in an $N$ N -voxel video. We also present a simple extension of FCSS to streaming FCSS for processing long videos that cannot be loaded into main memory at once. We evaluate FCSS, streaming FCSS and ten representative supervoxel methods on four video datasets and two novel video applications. The results show that our method simultaneously achieves state-of-the-art performance with respect to various evaluation criteria.

In this work, we tackle the essential problem of scale inconsistency for self-supervised joint depth-pose learning. Most existing methods assume that a consistent scale of depth and pose can be learned across all input samples, which makes the learning problem harder, resulting in degraded performance and limited generalization in indoor environments and long-sequence visual odometry application. To address this issue, we propose a novel system that explicitly disentangles scale from the network estimation. Instead of relying on PoseNet architecture, our method recovers relative pose by directly solving fundamental matrix from dense optical flow correspondence and makes use of a two-view triangulation module to recover an up-to-scale 3D structure. Then, we align the scale of the depth prediction with the triangulated point cloud and use the transformed depth map for depth error computation and dense reprojection check. Our whole system can be jointly trained end-to-end. Extensive experiments show that our system not only reaches state-of-the-art performance on KITTI depth and flow estimation, but also significantly improves the generalization ability of existing self-supervised depth-pose learning methods under a variety of challenging scenarios, and achieves state-of-the-art results among self-supervised learning-based methods on KITTI Odometry and NYUv2 dataset. Furthermore, we present some interesting findings on the limitation of PoseNet-based relative pose estimation methods in terms of generalization ability. Code is available at https://github.com/B1ueber2y/TrianFlow., Comment: To appear in CVPR 2020

Micro-expression is one of important clues for detecting lies. Its most outstanding characteristics include short duration and low intensity of movement. Therefore, video clips of high spatial-temporal resolution are much more desired than still images to provide sufficient details. On the other hand, owing to the difficulties to collect and encode micro-expression data, it is small sample size. In this paper, we use only 560 micro-expression video clips to evaluate the proposed network model: Transferring Long-term Convolutional Neural Network (TLCNN). TLCNN uses Deep CNN to extract features from each frame of micro-expression video clips, then feeds them to Long Short Term Memory (LSTM) which learn the temporal sequence information of micro-expression. Due to the small sample size of micro-expression data, TLCNN uses two steps of transfer learning: (1) transferring from expression data and (2) transferring from single frame of micro-expression video clips, which can be regarded as “big data”. Evaluation on 560 micro-expression video clips collected from three spontaneous databases is performed. The results show that the proposed TLCNN is better than some state-of-the-art algorithms.

We present an intuitive method that seamlessly integrates the whole process of garment design including 3D modeling, pattern development, garment simulation, and grading. Firstly, a sketch-based modeling method is proposed to design a 3D garment using editable control curves. Secondly, 2D patterns are developed by flattening 3D surface patches, then a 3D fine garment is formed directly based on the information of sewing relations and correspondence between 3D surfaces and 2D patterns. Lastly, garment templates are introduced to realize automatic design-preserving transfer of garments between human bodies with different sizes and shapes. The method is able to design 3D garments and 2D patterns efficiently and accurately. Our study provides an easy-to-learn and easy-to-use tool for both fashion designers and computer artists.

Font design has become an essential part of multimedia. It has the ability to convey the mood and intention of the designer. However, creating a new font for Chinese characters takes a lot of effort because the language contains over 20 thousand character with complex morphological structures. Current font completion methods have many disadvantages. To address this problem, we propose StarFont, a font completion system that can automatically complete a whole font using a few-shot learning method. Our model takes several examples of a new font, learns the design style and applies it to the remaining characters to complete the font. Unlike existing models proposed for font generation, we treat each character not the font as a class and abandon reconstruction loss because the font's ground truth is easier to obtain. Moreover, we combine multiple input images to generate new images, while existing methods use a one-to-one approach. Compared to other deep learning-based font completion methods, our model requires fewer examples of the new font and generates better results. Both qualitative and quantitative methods show that our method is more advanced.

State-of-the-art researches model the data of images and videos as low-dimensional manifolds and generate superpixels/supervoxels in a content-sensitive way, which is achieved by computing geodesic centroidal Voronoi tessellation (GCVT) on manifolds. However, computing exact GCVTs is slow due to computationally expensive geodesic distances. In this paper, we propose a much faster queue-based graph distance (called q-distance). Our key idea is that for manifold regions in which q-distances are different from geodesic distances, GCVT is prone to placing more generators in them, and therefore after few iterations, the q-distance-induced tessellation is an exact GCVT. This idea works well in practice and we also prove it theoretically under moderate assumption. Our method is simple and easy to implement. It runs 6-8 times faster than state-of-the-art GCVT computation, and has an optimal approximation ratio O(1) and a linear time complexity O(N) for N-pixel images or N-voxel videos. A thorough evaluation of 31 superpixel methods on five image datasets and 8 supervoxel methods on four video datasets shows that our method consistently achieves the best over-segmentation accuracy. We also demonstrate the advantage of our method on one image and two video applications.

Recently some pioneering works such as PointNet and Point-Net++ successfully introduce deep learning architectures into point cloud analysis. These novel networks take irregular point cloud (i.e., a set of unordered points) as input, in which each point is represented by (x, y, z) coordinates plus some attributes including color, normal, and other local or global features. Despite of their success on various tasks, the computational cost of these networks become extremely high for large-scale point clouds, e.g., containing hundreds of thousands or millions of points. Instead of uniform downsam-pling, in this paper, we propose a simple and novel filter that can efficiently filter any large-scale point cloud into thousands of representative points embedded in a high dimensional feature space, such that without changing the existing deep learning networks, simply using our filter as a preprocess, these existing models can work with large-scale point clouds. Experimental results show that by using our proposed filter, the computational cost (measured by floating-point operation) of PointNet and PointNet++ is reduced 30-60 times and the accuracy of semantic segmentation (measured by mean IoU) on ScanNet dataset is improved 5%-15% averagely.

Image retargeting techniques adjust images into different sizes and have attracted much attention recently. Objective quality assessment (OQA) of image retargeting results is often desired to automatically select the best results. Existing OQA methods train a model using some benchmarks (e.g., RetargetMe), in which subjective scores evaluated by users are provided. Observing that it is challenging even for human subjects to give consistent scores for retargeting results of different source images (diff-source-results), in this paper we propose a learning-based OQA method that trains a General Regression Neural Network (GRNN) model based on relative scores - which preserve the ranking - of retargeting results of the same source image (same-source-results). In particular, we develop a novel training scheme with provable convergence that learns a common base scalar for same-source-results. With this source specific offset, our computed scores not only preserve the ranking of subjective scores for same-source-results, but also provide a reference to compare the diff-source-results. We train and evaluate our GRNN model using human preference data collected in RetargetMe. We further introduce a subjective benchmark to evaluate the generalizability of different OQA methods. Experimental results demonstrate that our method outperforms ten representative OQA methods in ranking prediction and has better generalizability to different datasets.

Hand-drawn sketch recognition is a fundamental problem in computer vision, widely used in sketch-based image\ud and video retrieval, editing, and reorganization. Previous methods often assume that a complete sketch is used as input; however, hand-drawn sketches in common application scenarios are often incomplete, which makes sketch recognition a challenging problem. In this paper, we propose SketchGAN, a new generative adversarial network (GAN) based approach that jointly completes and recognizes a sketch, boosting the performance of both tasks. Specifically, we use a cascade Encode-Decoder network to complete the input sketch in an iterative manner, and employ an auxiliary sketch recognition task to recognize the completed sketch. Experiments on the Sketchy database benchmark demonstrate that our joint learning approach achieves competitive sketch completion and recognition performance compared with the state-of-the-art methods. Further experiments using several sketch-based applications also validate the performance of our method.

Significant progress has been made with image stylization using deep learning, especially with generative adversarial networks (GANs). However, existing methods fail to produce high quality artistic portrait drawings. Such drawings have a highly abstract style, containing a sparse set of continuous graphical elements such as lines, and so small artifacts are much more exposed than for painting styles. Moreover, artists tend to use different strategies to draw different facial features and the lines drawn are only loosely related to obvious image features. To address these challenges, we propose APDrawingGAN, a novel GAN based architecture that builds upon hierarchical generators and discriminators combining both a global network (for images as a whole) and local networks (for individual facial regions). This allows dedicated drawing strategies to be learned for different facial features. Since artists' drawings may not have lines perfectly aligned with image features, we develop a novel loss to measure similarity between generated and artists' drawings based on distance transforms, leading to improved strokes in portrait drawing. To train APDrawingGAN, we construct an artistic drawing dataset containing high-resolution portrait photos and corresponding professional artistic drawings. Extensive experiments, including a user study, show that APDrawingGAN produces significantly better artistic drawings than state-of-the-art methods.

This paper introduces a method for transferring colors between portrait images. Using a trained neural network to extract facial mask, we vectorize each image with a set of sparse diffusion curves to encode the low-frequency colors, and use the Laplacian of residual colors to represent the high-frequency details. Then we apply optimal mass transport to transfer the boundary colors between the diffusion curves of the source and reference images. Finally, the original or modified Laplacians of colors are added to the transferred diffusion curve image. Unlike the existing methods that either require 3D information or assume the source and reference images have similar poses and dense correspondence, our method is computationally efficient and flexible, which can work for portrait images with large pose and color differences. Ministry of Education (MOE) Accepted version This project was partially supported by Singapore Ministry of Education Grant RG26/17, NTU-Alibaba Joint Research Institute, Singapore, National Natural Science Foundation of China Grants (61872347, 61672481, 61725204, and U1736220), Special Plan for the Development of Distinguished Young Scientists of ISCAS, China (Y8RC535018), Youth Innovation Promotion Association CAS, China (No. 2018495) and the Royal Society-Newton Advanced Fellowship, China (NA150431).

Neural style transfer has drawn considerable attention from both academic and industrial field. Although visual effect and efficiency have been significantly improved, existing methods are unable to coordinate spatial distribution of visual attention between the content image and stylized image, or render diverse level of detail via different brush strokes. In this paper, we tackle these limitations by developing an attention-aware multi-stroke style transfer model. We first propose to assemble self-attention mechanism into a style-agnostic reconstruction autoencoder framework, from which the attention map of a content image can be derived. By performing multi-scale style swap on content features and style features, we produce multiple feature maps reflecting different stroke patterns. A flexible fusion strategy is further presented to incorporate the salient characteristics from the attention map, which allows integrating multiple stroke patterns into different spatial regions of the output image harmoniously. We demonstrate the effectiveness of our method, as well as generate comparable stylized images with multiple stroke patterns against the state-of-the-art methods.

Tangible interaction with customized products integrating sensors and actuators recently grows into an interdisciplinary research area in computer graphics and human-robot interaction (e.g., [Groeger et al. 2016; Yu et al. 2018]). In this paper, we introduce tangible interaction into 3D printed modular robots. Our user study demonstrates that interacting with our robots can effectively improve human spatial ability, which plays an important role during a person's development in science, technology, engineering or math (STEM).

Visualizing and communicating insights through maps offers an intuitive and familiar way to explore large-scale dynamic relational data. In this paper, we present VideoMap, which is a novel approach for presenting and interacting with relational video content by taking advantage of the map metaphor. VideoMap employs a metaphor to visualize video content by elements of a map with the aim of enabling exploration of video content as if reading a map. Video content is visualized in a hierarchal structure from a very large scale to a small scale of finely detailed representation. VideoMap recognizes a small set of sketch gestures for semantic zooming in and out, annotating the map, and automatically completing path navigation. To achieve this, VideoMap synthesizes map-derived visuals and binds them to the underlying data by operating the map with sketch interaction to facilitate interactive exploration. Extensive user studies were conducted to evaluate VideoMap, and the results demonstrated the effectiveness of VideoMap for facilitating the exploration and understanding of large video content.

Micro-expressions are brief facial movements characterized by short duration, involuntariness and low intensity. Recognition of spontaneous facial micro-expressions is a great challenge. In this paper, we propose a simple yet effective Main Directional Mean Optical-flow (MDMO) feature for micro-expression recognition. We apply a robust optical flow method on micro-expression video clips and partition the facial area into regions of interest (ROIs) based partially on action units. The MDMO is a ROI-based, normalized statistic feature that considers both local statistic motion information and its spatial location. One of the significant characteristics of MDMO is that its feature dimension is small. The length of a MDMO feature vector is $36 \times 2=72$ , where $36$ is the number of ROIs. Furthermore, to reduce the influence of noise due to head movements, we propose an optical-flow-driven method to align all frames of a micro-expression video clip. Finally, a SVM classifier with the proposed MDMO feature is adopted for micro-expression recognition. Experimental results on three spontaneous micro-expression databases, namely SMIC, CASME and CASME II, show that the MDMO can achieve better performance than two state-of-the-art baseline features, i.e., LBP-TOP and HOOF.

A majority of video summarization systems use linear representations, such as rectangular storyboards and timelines at linear scales. In this paper, we propose a novel nonlinear dynamic representation called SpiralTape that summarizes a video in a smooth spiral pattern. SpiralTape provides an unusual and fresh activity suitable for stimulating environments such as science and technology museums, in which children or young individuals can have enjoyable experiences that create meaningful learning outcomes. In addition, SpiralTape provides an uninterrupted overall structure of video content and takes design principles including compactness, continuity, efficient overview, and interactivity into consideration. A working SpiralTape system was developed and deployed in pilot applications and exhibitions. Elaborate user studies with evaluation benchmarks on multiple metrics were conducted to compare SpiralTape with two representative linear video summarization methods and a state-of-the-art radial video visualization. The evaluation results demonstrate the effectiveness and natural interaction performance of SpiralTape.

We present an evolution method for designing the styling curves of garments. The procedure of evolution is driven by aesthetics-inspired scores to evaluate the quality of styling designs, where the aesthetic considerations are represented in the form of streamlines on human bodies. A dual representation is introduced in our platform to process the styling curves of designs, based on which robust methods for realizing the operations of evolution are developed. Starting from a given set of styling designs on human bodies, we demonstrate the effectiveness of set evolution inspired by aesthetic factors. The evolution is adaptive to the change of aesthetic inspirations. By this adaptation, our platform can automatically generate new designs fulfilling the demands of variations in different human bodies and poses.

Humans' remarkable capacity on rapid natural scene categorization has been widely studied in neuroscience. Recently, a functional MRI (fMRI) study showed that in human brain, decoding of natural scenes from line drawings was very similar to those from color photographs. In this paper, based on recently proposed computational cognition model of Perception, Memory and Judgement (PMJ model), we investigate the computational model of line drawings and propose a PMJ-inspired cognitive framework for natural scene categorization in line drawings. The Ohio State University (OSU) dataset was used, which included 475 color photographs in six categories, i.e., beaches, city streets, forests, highways, mountains and offices, as well as 475 corresponding line drawings produced by trained artists. Experimental results show that our proposed cognitive framework achieves 48.4% recognition rate in leave-one-out cross-validation, which is much higher than fMRI-data-driven decoding accuracy in the visual-processing hierarchy (29% in V1, 27% in V2+VP, 26% in V4, 29% in PPA and 23% in RSC).

Micro-expression (ME) is a special type of human expression\ud which can reveal the real emotion that people want to\ud conceal. Spontaneous ME (SME) spotting is to identify the\ud subsequences containing SMEs from a long facial video. The\ud study of SME spotting has a significant importance, but is also\ud very challenging due to the fact that in real-world scenarios,\ud SMEs may occur along with normal facial expressions and\ud other prominent motions such as head movements. In this\ud paper, we improve a state-of-the-art SME spotting method\ud called feature difference analysis (FD) in the following two\ud aspects. First, FD relies on a partitioning of facial area into\ud uniform regions of interest (ROIs) and computing features of\ud a selected sequence. We propose a novel evaluation method\ud by utilizing the Fisher linear discriminant to assign a weight\ud for each ROI, leading to more semantically meaningful ROIs.\ud Second, FD only considers two features (LBP and HOOF)\ud independently. We introduce a state-of-the-art MDMO feature\ud into FD and propose a simple yet efficient collaborative\ud strategy to work with two complementary features, i.e., LBP\ud characterizing texture information and MDMO characterizing\ud motion information. We call our improved FD method\ud collaborative feature difference (CFD). Experimental results\ud on two well-established SME datasets SMIC-E and CASME\ud II show that CFD significantly improves the performance of\ud the original FD.

International audience; This paper presents a new method to fabricate 3D models on a robotic printing system equipped with multi-axis motion. Materials are accumulated inside the volume along curved tool-paths so that the need of supporting structures can be tremendously reduced - if not completely abandoned - on all models. Our strategy to tackle the challenge of tool-path planning for multi-axis 3D printing is to perform two successive decompositions, first volume-to-surfaces and then surfaces-to-curves. The volume-to-surfaces decomposition is achieved by optimizing a scalar field within the volume that represents the fabrication sequence. The field is constrained such that its iso-values represent curved layers that are supported from below, and present a convex surface affording for collision-free navigation of the printer head. After extracting all curved layers, the surfaces-to-curves decomposition covers them with tool-paths while taking into account constraints from the robotic printing system. Our method successfully generates tool-paths for 3D printing models with large overhangs and high-genus topology. We fabricated several challenging cases on our robotic platform to verify and demonstrate its capabilities.

In this paper, we propose a solution to transforming photos\ud of real-world scenes into cartoon style images, which is\ud valuable and challenging in computer vision and computer\ud graphics. Our solution belongs to learning based methods,\ud which have recently become popular to stylize images in\ud artistic forms such as painting. However, existing methods\ud do not produce satisfactory results for cartoonization,\ud due to the fact that (1) cartoon styles have unique characteristics\ud with high level simplification and abstraction, and\ud (2) cartoon images tend to have clear edges, smooth color\ud shading and relatively simple textures, which exhibit significant\ud challenges for texture-descriptor-based loss functions\ud used in existing methods. In this paper, we propose CartoonGAN,\ud a generative adversarial network (GAN) framework\ud for cartoon stylization. Our method takes unpaired\ud photos and cartoon images for training, which is easy to\ud use. Two novel losses suitable for cartoonization are proposed:\ud (1) a semantic content loss, which is formulated as\ud a sparse regularization in the high-level feature maps of\ud the VGG network to cope with substantial style variation\ud between photos and cartoons, and (2) an edge-promoting\ud adversarial loss for preserving clear edges. We further introduce\ud an initialization phase, to improve the convergence\ud of the network to the target manifold. Our method is also\ud much more efficient to train than existing methods. Experimental\ud results show that our method is able to generate\ud high-quality cartoon images from real-world photos (i.e.,\ud following specific artists’ styles and with clear edges and\ud smooth shading) and outperforms state-of-the-art methods.

In this article, we present a human experience–inspired path planning algorithm for service robots. In addition to considering the path distance and smoothness, we emphasize the safety of robot navigation. Specifically, we build a speed field in accordance with several human driving experiences, like slowing down or detouring at a narrow aisle, and keeping a safe distance to the obstacles. Based on this speed field, the path curvatures, path distance, and steering speed are all integrated to form an energy function, which can be efficiently solved by the A* algorithm to seek the optimal path by resorting to an admissible heuristic function estimated from the energy function. Moreover, a simple yet effective fast path smoothing algorithm is proposed so as to ease the robots steering. Several examples are presented, demonstrating the effectiveness of our human experience–inspired path planning method.

Offsetting-based hollowing is a solid modeling operation widely used in 3D printing, which can change the model's physical properties and reduce the weight by generating voids inside a model. However, a hollowing operation can lead to additional supporting structures for fabrication in interior voids, which cannot be removed. As a consequence, the result of a hollowing operation is affected by these additional supporting structures when applying the operation to optimize physical properties of different models. This paper proposes a support-free hollowing framework to overcome the difficulty of fabricating voids inside a solid. The challenge of computing a support-free hollowing is decomposed into a sequence of shape optimization steps, which are repeatedly applied to interior mesh surfaces. The optimization of physical properties in different applications can be easily integrated into our framework. Comparing to prior approaches that can generate support-free inner structures, our hollowing operation can reduce more volume of material and thus provide a larger solution space for physical optimization. Experimental tests are taken on a number of 3D models to demonstrate the effectiveness of this framework.

Line drawings, as a concise form, can be recognized by infants and even chimpanzees. Recently, how the visual system processes line-drawings attracts more and more attention from psychology, cognitive science and computer science. The neuroscientific studies revealed that line drawings generate similar neural actions as color photographs, which give insights on how to efficiently process big media data. In this paper, we present a comprehensive survey on line drawing studies, including cognitive mechanism of visual perception, computational models in computer vision and intelligent process in diverse media applications. Major debates, challenges and solutions that have been addressed over the years are discussed. Finally some of the ensuing challenges in line drawing studies are outlined.

In this paper, inspired by Gatys’s recent work, we propose\ud a novel approach that transforms photos to comics using\ud deep convolutional neural networks (CNNs). While Gatys’s\ud method that uses a pre-trained VGG network generally works\ud well for transferring artistic styles such as painting from a\ud style image to a content image, for more minimalist styles\ud such as comics, the method often fails to produce satisfactory\ud results. To address this, we further introduce a dedicated\ud comic style CNN, which is trained for classifying comic images\ud and photos. This new network is effective in capturing\ud various comic styles and thus helps to produce better comic\ud stylization results. Even with a grayscale style image, Gatys’s\ud method can still produce colored output, which is not desirable\ud for comics. We develop a modified optimization framework\ud such that a grayscale image is guaranteed to be synthesized.\ud To avoid converging to poor local minima, we further\ud initialize the output image using grayscale version of the content\ud image. Various examples show that our method synthesizes\ud better comic images than the state-of-the-art method.

This paper presents a robotic system — RoboFDM that targets at printing 3D models without support-structures, which is considered as the major restriction to the flexibility of 3D printing. The hardware of RoboFDM consists of a robotic arm providing 6-DOF motion to the platform of material accumulation and an extruder forming molten filaments of polylactic acid (PLA). The fabrication of 3D models in this system follows the principle of fused decomposition modeling (FDM). Different from conventional FDM, an input model fabricated by RoboFDM is printed along different directions at different places. A new algorithm is developed to decompose models into support-free parts that can be printed one by one in a collision-free sequence. The printing directions of all parts are also determined during the computation of model decomposition. Experiments have been successfully taken on our RoboFDM system to print general freeform objects in a support-free manner.

Self-reconfigurable modular robots (SRRobot) that can change their shape and function in different environments according to different tasks have caught a lot of attention recently. Most existing prototypes use professional electronic components with relatively expensive cost and high barrier of fabrication. In this paper, we present a low-cost SRRobot with double-cube modules. Our system is easy-to-build even for novices as all electric components are off-the-shelf and the structural components in plastics are made by 3D printing. To have a better design of interior structures, we first construct a design space for all feasible solutions that satisfy the constraints of fabrication. Then, an optimized solution is found by an objective function incorporating the factors of space utilization, structural sound-ness and assembly complexity. Thirty EasySRRobot modules are manufactured and assembled. The functionality of our algorithm is demonstrated by comparing an optimized interior design with other two feasible designs and realizing different motions on an EasySRRobot with four modules.

Superpixels are perceptually meaningful atomic regions that can effectively capture image features. Among various methods for computing uniform superpixels, simple linear iterative clustering (SLIC) is popular due to its simplicity and high performance. In this paper, we extend SLIC to compute content-sensitive superpixels, i.e., small superpixels in content-dense regions with high intensity or colour variation and large superpixels in content-sparse regions. Rather than using the conventional SLIC method that clusters pixels in $\mathbb {R}^5$ , we map the input image $I$ to a 2-dimensional manifold $\mathcal {M}\subset \mathbb {R}^5$ , whose area elements are a good measure of the content density in $I$ . We propose a simple method, called intrinsic manifold SLIC (IMSLIC), for computing a geodesic centroidal Voronoi tessellation (GCVT)—a uniform tessellation—on $\mathcal {M}$ , which induces the content-sensitive superpixels in $I$ . In contrast to the existing algorithms, IMSLIC characterizes the content sensitivity by measuring areas of Voronoi cells on $\mathcal {M}$ . Using a simple and fast approximation to a closed-form solution, the method can compute the GCVT at a very low cost and guarantees that all Voronoi cells are simply connected. We thoroughly evaluate IMSLIC and compare it with eleven representative methods on the BSDS500 dataset and seven representative methods on the NYUV2 dataset. Computational results show that IMSLIC outperforms existing methods in terms of commonly used quality measures pertaining to superpixels such as compactness, adherence to boundaries, and achievable segmentation accuracy. We also evaluate IMSLIC and seven representative methods in an image contour closure application, and the results on two datasets, WHD and WSD, show that IMSLIC achieves the best foreground segmentation performance.

The 2.1D sketch is a layered image representation, which assigns a partial depth ordering of over-segmented regions in a monocular image. This paper presents a global optimization framework for inferring the 2.1D sketch from a monocular image. Our method only uses over-segmented image regions (i.e., superpixels) as input, without any information of objects in the image, since (1) segmenting objects in images is a difficult problem on its own and (2) the objective of our proposed method is to be generic as an initial module useful for downstream high-level vision tasks. This paper formulates the inference of the 2.1D sketch using a global energy optimization framework. The proposed energy function consists of two components: (1) one is defined based on the local partial ordering relations (i.e., figure-ground) between two adjacent over-segmented regions, which captures the marginal information of the global partial depth ordering and (2) the other is defined based on the same depth layer relations among all the over-segmented regions, which groups regions of the same object to account for the over-segmentation issues. A hybrid evolution algorithm is utilized to minimize the global energy function efficiently. In experiments, we evaluated our method on a test data set containing 100 diverse real images from Berkeley segmentation data set (BSDS500) with the annotated ground truth. Experimental results show that our method can infer the 2.1D sketch with high accuracy.

With the rapid growth of video resources, techniques for efficient organization of video clips are becoming appealing in the multimedia domain. In this article, a sketch-based approach is proposed to intuitively organize video clips by: (1) enhancing their narrations using sketch annotations and (2) structurizing the organization process by gesture-based free-form sketching on touch devices. There are two main contributions of this work. The first is a sketch graph, a novel representation for the narrative structure of video clips to facilitate content organization. The second is a method to perform context-aware sketch recommendation scalable to large video collections, enabling common users to easily organize sketch annotations. A prototype system integrating the proposed approach was evaluated on the basis of five different aspects concerning its performance and usability. Two sketch searching experiments showed that the proposed context-aware sketch recommendation outperforms, in terms of accuracy and scalability, two state-of-the-art sketch searching methods. Moreover, a user study showed that the sketch graph is consistently preferred over traditional representations such as keywords and keyframes. The second user study showed that the proposed approach is applicable in those scenarios where the video annotator and organizer were the same person. The third user study showed that, for video content organization, using sketch graph users took on average 1/3 less time than using a mass-market tool Movie Maker and took on average 1/4 less time than using a state-of-the-art sketch alternative. These results demonstrated that the proposed sketch graph approach is a promising video organization tool.

Green ecological landscape design represents a sustainable design and development model, aims to build an environment which fits peoples health and living. Starting with its meanings, this paper makes a full analysis of its guiding ideology, characteristics and composition, and takes a practical application of the above analysis through elements and methods of residential landscape design. Finally, we hope this paper can play a guiding role in the process of our countrys landscape design toward to sustainable green and eco-design.

Sketch-based modeling has attracted considerable attention in recent years. In this paper, we propose a semantic feature modeling system for sketch-based jewelry design, called blue Sketch2Jewelry. The newly devised semantic feature class encodes specific domain knowledge (jewelry design knowledge in this paper) and supplies prolific semantic information. The advantage of using semantic features is to narrow down the searching space in sketch-based feature retrieval and benefits the parameter selection from input sketches for feature instantiation and feature placement. Thus, the inaccuracy and ambiguity problems of freehand sketch inputs are alleviated within Sketch2Jewelry, compared to previous commercial feature-based modeling tools, e.g. SolidWorks, which are limited to fake sketches (i.e. not real freehand inputs). Since semantic features are high-level building blocks, together with sketch inputs, the proposed Sketch2Jewelry system can significantly improve the jewelry design efficiency. In addition, Sketch2Jewelry allows non-experts to sketch a complex jewelry model naturally and efficiently with design-by-feature. Examples are provided to demonstrate its usefulness.