Search

The revolution of artificial intelligence (AI) is transforming major industries worldwide. With accurate inferencing, AI has caught the attention of many engineers and scientists. Promisingly, hardware-in the-loop (HIL) emulation can adopt this type of modeling method as one of the alternatives after comprehensive investigation. This paper proposes an approach for emulating power electronic motor drive transients for advanced transportation applications (ATAs) using machine learning building blocks (MLBBs) without any traditional circuit-oriented transient solver. The more electric aircraft (MEA) power system is chosen as a case study to validate the real-time emulation performance of MLBBs. Inside MLBBs, neural networks (NNs) have been applied to build component-level, device-level, and system-level models for various equipment. These models are well trained in a cluster and transplanted into the field-programmable gate array (FPGA) based hardware platform. Finally, MLBB emulation results are compared with PSCAD/EMTDC for system-level and SaberRD for device-level, which showed high consistency for model accuracy and high speed-up for hardware execution.

It is still a challenging task to provide satisfactory quality of experience (QoE) to users by traditional routing mechanism in current public Internet for real-time multimedia services with high bandwidth and strict end-to-end latency constraint. Multipath transmission is a promising way to improve the performance of data delivery, due to its advantage in bandwidth aggregation capacity over multiple paths to speed up high volume transfers between endpoints. However, existing multipath transmission has many disadvantages such as the requirements for end-host support (i.e. the necessary multi-homed network setting, and operating system or upper applications updates of user devices), and blindingly pursuing maximum of transmission throughput. Path allocation and end-to-end transmission control are two important factors affecting the quality of multipath transmission. As an emerging network architecture, software-defined networking (SDN) is flexible, manageable, and responsive to rapid changes in traffic requirements. In this paper, we propose a multipath transport scheme based on SDN with segment routing (SR) which can meet the bandwidth and low end-to-end transmission latency requirements of real-time interactive multimedia services. The centralized SDN controller is extended to allocate multiple disjoint paths meeting bandwidth requirement meanwhile balancing the network load. The proprietary multipath media server (MMS) dynamically controls the assignment between subflows and routing paths to reduce the end-to-end transmission latency and thus improve QoE of end-users. To show the feasibility of our approach, we construct an SDN platform using Mininet. Simulation results show that our approach performs better compared to the conventional approaches in terms of traffic balancing and end-to-end transmission delay.

Transferring the sentiment of an image is an unexplored research topic in the area of computer vision. This work proposes a novel framework consisting of a reference image retrieval step and a global sentiment transfer step to transfer sentiments of images according to a given sentiment tag. The proposed image retrieval algorithm is based on the SSIM index. The retrieved reference images by the proposed algorithm are more content-related against the algorithm based on the perceptual loss. Therefore can lead to a better image sentiment transfer result. In addition, we propose a global sentiment transfer step, which employs an optimization algorithm to iteratively transfer sentiment of images based on feature maps produced by the Densenet121 architecture. The proposed sentiment transfer algorithm can transfer the sentiment of images while ensuring the content structure of the input image intact. The qualitative and quantitative experiments demonstrate that the proposed sentiment transfer framework outperforms existing artistic and photorealistic style transfer algorithms in making reliable sentiment transfer results with rich and exact details.

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Short-form video social media shifts away from the traditional media paradigm by telling the audience a dynamic story to attract their attention. In particular, different combinations of everyday objects can be employed to represent a unique scene that is both interesting and understandable. Offered by the same company, TikTok and Douyin are popular examples of such new media that has become popular in recent years, while being tailored for different markets (e.g. the United States and China). The hypothesis that they express cultural differences together with media fashion and social idiosyncrasy is the primary target of our research. To that end, we first employ the Faster Regional Convolutional Neural Network (Faster R-CNN) pre-trained with the Microsoft Common Objects in COntext (MS-COCO) dataset to perform object detection. Based on a suite of objects detected from videos, we perform statistical analysis including label statistics, label similarity, and label-person distribution. We further use the Two-Stream Inflated 3D ConvNet (I3D) pre-trained with the Kinetics dataset to categorize and analyze human actions. By comparing the distributional results of TikTok and Douyin, we uncover a wealth of similarity and contrast between the two closely related video social media platforms along the content dimensions of object quantity, object categories, and human action categories., Accepted by IEEE Big Data 2020

Learning and processing of signals over hypergraph models have gained substantial traction owing to the ability of hypergraphs in characterizing multilateral interactions. In this work, we explore hypergraph spectral analysis and provide alternative definitions of frequency domain operations that are practically useful in image processing. We analyze hypergraph spectral properties and present several application examples, including compression, edge detection and segmentation. Successful experiment results demonstrate the effectiveness and the future prospect of the proposed hypergraph frequency operations in image processing.

Recent skeleton-based action recognition approaches achieve great improvement by using recurrent neural network (RNN) models. Currently, these approaches build an end-to-end network from coordinates of joints to class categories and improve accuracy by extending RNN to spatial domains. First, while such well-designed models and optimization strategies explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, according to the evolution of previous work, we select a set of simple geometric features, and then seperately feed each type of features to a three-layer LSTM framework. Second, we propose a multistream LSTM architecture with a new smoothed score fusion technique to learn classification from different geometric feature streams. Furthermore, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and the fusion results achieve the state-of-the-art performance on four datasets. We also show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art., This paper is accepted by AAAI 2020

Establishing semantic correspondence is a core problem in computer vision and remains challenging due to large intra-class variations and lack of annotated data. In this paper, we aim to incorporate global semantic context in a flexible manner to overcome the limitations of prior work that relies on local semantic representations. To this end, we first propose a context-aware semantic representation that incorporates spatial layout for robust matching against local ambiguities. We then develop a novel dynamic fusion strategy based on attention mechanism to weave the advantages of both local and context features by integrating semantic cues from multiple scales. We instantiate our strategy by designing an end-to-end learnable deep network, named as Dynamic Context Correspondence Network (DCCNet). To train the network, we adopt a multi-auxiliary task loss to improve the efficiency of our weakly-supervised learning procedure. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including PF-Pascal, PF-Willow, and TSS, demonstrating its effectiveness and generality., ICCV 2019

Signal processing over graphs has recently attracted significant attention for dealing with the structured data. Normal graphs, however, only model pairwise relationships between nodes and are not effective in representing and capturing some high-order relationships of data samples, which are common in many applications, such as Internet of Things (IoT). In this article, we propose a new framework of hypergraph signal processing (HGSP) based on the tensor representation to generalize the traditional graph signal processing (GSP) to tackle high-order interactions. We introduce the core concepts of HGSP and define the hypergraph Fourier space. We then study the spectrum properties of hypergraph Fourier transform (HGFT) and explain its connection to mainstream digital signal processing. We derive the novel hypergraph sampling theory and present the fundamentals of hypergraph filter design based on the tensor framework. We present HGSP-based methods for several signal processing and data analysis applications. Our experimental results demonstrate significant performance improvement using our HGSP framework over some traditional signal processing solutions.

Multipath diversity significantly impacts multipath transmission quality. Enough multipath diversity would minimize the negative influence brought by an individual path, thus improving tolerance capability of network congestion and failure. However, multipath diversity is hard to guarantee on overlay networks because of inaccurate awareness of underlay network and multipath generating methods considering little about underlay diversity. In this paper, we design a multi-dimension spatial method for topology awareness and multipath generating. Analyzing that the complicated underlay networks with multiple autonomous systems reduce the accuracy of network positioning for topology awareness, we decompose the underlay networks into multiple dimensions, namely intra and inter autonomous system dimensions. We generate independent views for each autonomous system and merge views by embedding exchange points. Then, we design some spatial mechanisms to evaluate link diversity and to constrain multipath generating. Based on the multi-dimensional view, multipath generating is conducted in inter and intra autonomous system phases. Experiments demonstrate that the proposed method improves topology awareness accuracy and guarantees multipath diversity better and the transmission quality is improved.

We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses on grounding the query as a whole, temporal dependencies and reasoning between events within the text are not fully considered. In this paper, we propose a novel Temporal Compositional Modular Network (TCMN) where a tree attention network first automatically decomposes a sentence into three descriptions with respect to the main event, context event and temporal signal. Two modules are then utilized to measure the visual similarity and location similarity between each segment and the decomposed descriptions. Moreover, since the main event and context event may rely on different modalities (RGB or optical flow), we use late fusion to form an ensemble of four models, where each model is independently trained by one combination of the visual input. Experiments show that our model outperforms the state-of-the-art methods on the TEMPO dataset.

Although the recent success of convolutional neural network (CNN) advances state-of-the-art saliency prediction in static images, few work has addressed the problem of predicting attention in videos. On the other hand, we find that the attention of different subjects consistently focuses on a single face in each frame of videos involving multiple faces. Therefore, we propose in this paper a novel deep learning (DL) based method to predict salient face in multiple-face videos, which is capable of learning features and transition of salient faces across video frames. In particular, we first learn a CNN for each frame to locate salient face. Taking CNN features as input, we develop a multiple-stream long short-term memory (M-LSTM) network to predict the temporal transition of salient faces in video sequences. To evaluate our DL-based method, we build a new eye-tracking database of multiple-face videos. The experimental results show that our method outperforms the prior state-of-the-art methods in predicting visual attention on faces in multiple-face videos.

RNN-based approaches have achieved outstanding performance on action recognition with skeleton inputs. Currently these methods limit their inputs to coordinates of joints and improve the accuracy mainly by extending RNN models to spatial domains in various ways. While such models explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, we select a set of simple geometric features, motivated by the evolution of previous work. With experiments on a 3-layer LSTM framework, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and achieve state-of-art results on four datasets. Further, we show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction., This paper is accepted by NAACL'21