Search

Three-dimensional (3D) point clouds are important data representations in visualization applications. The rapidly growing utility and popularity of point cloud processing strongly motivate a plethora of research activities on large-scale point cloud processing and feature extraction. In this work, we investigate point cloud resampling based on hypergraph signal processing (HGSP). We develop a novel method to extract sharp object features and reduce the data size of point cloud representation. By directly estimating hypergraph spectrum based on hypergraph stationary processing, we design a spectral kernel-based filter to capture high-dimensional interactions among point signal nodes and to better preserve object surface outlines. Experimental results validate the effectiveness of hypergraph in representing point clouds, and demonstrate the robustness of the proposed algorithm under noise.

The revolution of artificial intelligence (AI) is transforming major industries worldwide. With accurate inferencing, AI has caught the attention of many engineers and scientists. Promisingly, hardware-in the-loop (HIL) emulation can adopt this type of modeling method as one of the alternatives after comprehensive investigation. This paper proposes an approach for emulating power electronic motor drive transients for advanced transportation applications (ATAs) using machine learning building blocks (MLBBs) without any traditional circuit-oriented transient solver. The more electric aircraft (MEA) power system is chosen as a case study to validate the real-time emulation performance of MLBBs. Inside MLBBs, neural networks (NNs) have been applied to build component-level, device-level, and system-level models for various equipment. These models are well trained in a cluster and transplanted into the field-programmable gate array (FPGA) based hardware platform. Finally, MLBB emulation results are compared with PSCAD/EMTDC for system-level and SaberRD for device-level, which showed high consistency for model accuracy and high speed-up for hardware execution.

It is still a challenging task to provide satisfactory quality of experience (QoE) to users by traditional routing mechanism in current public Internet for real-time multimedia services with high bandwidth and strict end-to-end latency constraint. Multipath transmission is a promising way to improve the performance of data delivery, due to its advantage in bandwidth aggregation capacity over multiple paths to speed up high volume transfers between endpoints. However, existing multipath transmission has many disadvantages such as the requirements for end-host support (i.e. the necessary multi-homed network setting, and operating system or upper applications updates of user devices), and blindingly pursuing maximum of transmission throughput. Path allocation and end-to-end transmission control are two important factors affecting the quality of multipath transmission. As an emerging network architecture, software-defined networking (SDN) is flexible, manageable, and responsive to rapid changes in traffic requirements. In this paper, we propose a multipath transport scheme based on SDN with segment routing (SR) which can meet the bandwidth and low end-to-end transmission latency requirements of real-time interactive multimedia services. The centralized SDN controller is extended to allocate multiple disjoint paths meeting bandwidth requirement meanwhile balancing the network load. The proprietary multipath media server (MMS) dynamically controls the assignment between subflows and routing paths to reduce the end-to-end transmission latency and thus improve QoE of end-users. To show the feasibility of our approach, we construct an SDN platform using Mininet. Simulation results show that our approach performs better compared to the conventional approaches in terms of traffic balancing and end-to-end transmission delay.

Hypergraph spectral analysis has emerged as an effective tool processing complex data structures in data analysis. The surface of a three-dimensional (3D) point cloud, and the multilateral relationship among their points can be naturally captured by the high-dimensional hyperedges. This work investigates the power of hypergraph spectral analysis in unsupervised segmentation of 3D point clouds. We estimate, and order the hypergraph spectrum from observed point cloud coordinates. By trimming the redundancy from the estimated hypergraph spectral space based on spectral component strengths, we develop a clustering-based segmentation method. We apply the proposed method to various point clouds, and analyze their respective spectral properties. Our experimental results demonstrate the effectiveness and efficiency of the proposed segmentation method.

We address the problem of retrieving a specific moment from an untrimmed video by natural language. It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they do not fully consider the temporal contexts between temporal moments. In this paper, we model the temporal context between video moments by a set of predefined two-dimensional maps under different temporal scales. For each map, one dimension indicates the starting time of a moment and the other indicates the duration. These 2D temporal maps can cover diverse video moments with different lengths, while representing their adjacent contexts at different temporal scales. Based on the 2D temporal maps, we propose a Multi-Scale Temporal Adjacency Network (MS-2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal contexts at each scale, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed MS-2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our MS-2D-TAN outperforms the state of the art.

Interpreting model knowledge is an essential topic to improve human understanding of deep black-box models. Traditional methods contribute to providing intuitive instance-wise explanations which allocating importance scores for low-level features (e.g, pixels for images). To adapt to the human way of thinking, one strand of recent researches has shifted its spotlight to mining important concepts. However, these concept-based interpretation methods focus on computing the contribution of each discovered concept on the class level and can not precisely give instance-wise explanations. Besides, they consider each concept as an independent unit, and ignore the interactions among concepts. To this end, in this paper, we propose a novel COncept-based NEighbor Shapley approach (dubbed as CONE-SHAP) to evaluate the importance of each concept by considering its physical and semantic neighbors, and interpret model knowledge with both instance-wise and class-wise explanations. Thanks to this design, the interactions among concepts in the same image are fully considered. Meanwhile, the computational complexity of Shapley Value is reduced from exponential to polynomial. Moreover, for a more comprehensive evaluation, we further propose three criteria to quantify the rationality of the allocated contributions for the concepts, including coherency, complexity, and faithfulness. Extensive experiments and ablations have demonstrated that our CONE-SHAP algorithm outperforms existing concept-based methods and simultaneously provides precise explanations for each instance and class.

Few-shot video classification aims to learn new video categories with only a few labeled examples, alleviating the burden of costly annotation in real-world applications. However, it is particularly challenging to learn a class-invariant spatial-temporal representation in such a setting. To address this, we propose a novel matching-based few-shot learning strategy for video sequences in this work. Our main idea is to introduce an implicit temporal alignment for a video pair, capable of estimating the similarity between them in an accurate and robust manner. Moreover, we design an effective context encoding module to incorporate spatial and feature channel context, resulting in better modeling of intra-class variations. To train our model, we develop a multi-task loss for learning video matching, leading to video features with better generalization. Extensive experimental results on two challenging benchmarks, show that our method outperforms the prior arts with a sizable margin on Something-Something-V2 and competitive results on Kinetics.

Real time video traffic has stringent delay requirement and is bandwidth demanding when streaming over the internet. Single path transmission may provide unsatisfactory picture quality to users due to packet loss, jitter, and low bandwidth. Multipath transmission is another option to achieve better service by aggregating bandwidth and exploiting path diversity. As utility maximization theory shows, to achieve better satisfaction is to provide higher throughput and lower frame delivery delay for real time video traffic. The congestion control is responsible to probe maximum available bandwidth and packet scheduling is to take advantages of path diversity, which are two key research points in multipath context and are also the main concern in this paper. Given the BBR congestion control algorithm causes high-packet loss rate and large transmission delay when BBR flows competing for bandwidth resource, a delay response BBR (Delay-BBR) algorithm is proposed for real time video transmission. The main idea is that the sender will actively reduce the rate to let intermediate routers drain the excess occupied buffer once round trip delay signal exceeds a certain threshold. The results show that the delay-BBR can achieve lower transmission delay and lower packet loss rate when compared with QUIC-BBR and WebRTC-BBR. The rate stable feature makes delay-BBR appropriate for real time video transmission. Moreover, a packet scheduling algorithm works in coordination with the rate control algorithm achieves lower frame delivery delay further compared with the benchmark algorithms.

Despite the recent success of deep neural networks, it remains challenging to effectively model the long-tail class distribution in visual recognition tasks. To address this problem, we first investigate the performance bottleneck of the two-stage learning framework via ablative study. Motivated by our discovery, we propose a unified distribution alignment strategy for long-tail visual recognition. Specifically, we develop an adaptive calibration function that enables us to adjust the classification scores for each data point. We then introduce a generalized re-weight method in the two-stage learning to balance the class prior, which provides a flexible and unified solution to diverse scenarios in visual recognition tasks. We validate our method by extensive experiments on four tasks, including image classification, semantic segmentation, object detection, and instance segmentation. Our approach achieves the state-of-the-art results across all four recognition tasks with a simple and unified framework. The code and models will be made publicly available at: https://github.com/Megvii-BaseDetection/DisAlign, Accepted by CVPR 2021

Transferring the sentiment of an image is an unexplored research topic in the area of computer vision. This work proposes a novel framework consisting of a reference image retrieval step and a global sentiment transfer step to transfer sentiments of images according to a given sentiment tag. The proposed image retrieval algorithm is based on the SSIM index. The retrieved reference images by the proposed algorithm are more content-related against the algorithm based on the perceptual loss. Therefore can lead to a better image sentiment transfer result. In addition, we propose a global sentiment transfer step, which employs an optimization algorithm to iteratively transfer sentiment of images based on feature maps produced by the Densenet121 architecture. The proposed sentiment transfer algorithm can transfer the sentiment of images while ensuring the content structure of the input image intact. The qualitative and quantitative experiments demonstrate that the proposed sentiment transfer framework outperforms existing artistic and photorealistic style transfer algorithms in making reliable sentiment transfer results with rich and exact details.

Scene graph generation is an important visual understanding task with a broad range of vision applications. Despite recent tremendous progress, it remains challenging due to the intrinsic long-tailed class distribution and large intra-class variation. To address these issues, we introduce a novel confidence-aware bipartite graph neural network with adaptive message propagation mechanism for unbiased scene graph generation. In addition, we propose an efficient bi-level data resampling strategy to alleviate the imbalanced data distribution problem in training our graph network. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including Visual Genome, Open Images V4/V6, demonstrating its effectiveness and generality., Comment: Accepted by CVPR2021

This article is mainly aimed at defining and verifying whether the project involves environmentally sensitive areas in the process of environmental intelligent survey, calculating the position coordinate information of the target GPS position in the aerial image map under the condition of Gaussian projection, and further calculating the four corners of the image and the position of other target points under the WGS84 coordinate system. First of all, based on the POS data information of aerial equipment, this paper calculates the corresponding Gaussian projection and POS data information, then the pseudo-position information of image center coordinates is further calculated; Second, Based on the coordinates information of gauss projection, this paper calculates the heading open angle and side open angle of the camera, then calculate the width and height of the image for pose correction, and finally get the coordinate position of the corrected image center point. Finally, the GPS positions of other target points can be calculated by pixel coordinates of images, and the calculated results are pasted on Google earth for precision comparison. Experimental results show that the proposed method can accurately calculate the GPS positions of targets in aerial images.

Incremental learning of semantic segmentation has emerged as a promising strategy for visual scene interpretation in the open- world setting. However, it remains challenging to acquire novel classes in an online fashion for the segmentation task, mainly due to its continuously-evolving semantic label space, partial pixelwise ground-truth annotations, and constrained data availability. To ad- dress this, we propose an incremental learning strategy that can fast adapt deep segmentation models without catastrophic forgetting, using a streaming input data with pixel annotations on the novel classes only. To this end, we develop a uni ed learning strategy based on the Expectation-Maximization (EM) framework, which integrates an iterative relabeling strategy that lls in the missing labels and a rehearsal-based incremental learning step that balances the stability-plasticity of the model. Moreover, our EM algorithm adopts an adaptive sampling method to select informative train- ing data and a class-balancing training strategy in the incremental model updates, both improving the e cacy of model learning. We validate our approach on the PASCAL VOC 2012 and ADE20K datasets, and the results demonstrate its superior performance over the existing incremental methods., Comment: Accepted by ACM MM'21

The transmeridian flight required pilots to perform tasks that are high in mental workload for long periods. Consequently, they are likely to experience fatigue and have difficulty in concentrating, which could lead to flight safety risks since pilots’ emergency response ability would be weakened. This study proposes a system to help pilots maintain attention on tasks over time to reduce air accidents. The neuroimaging method functional near-infrared spectroscopy (fNIRS) would be used to assess the attention level and mental workload in this system. Based on the results, appropriate feedback, such as a reminder to re-engage the pilot if disengagement detected, would be provided. For testing the effectiveness of this system on sustained attention, an experiment was designed to examine whether using this system could reduce pilots’ missed instructions when communicating with ground control.

Along with increasingly popular virtual reality applications, the three-dimensional (3D) point cloud has become a fundamental data structure to characterize 3D objects and surroundings. To process 3D point clouds efficiently, a suitable model for the underlying structure and outlier noises is always critical. In this work, we propose a hypergraph-based new point cloud model that is amenable to efficient analysis and processing. We introduce tensor-based methods to estimate hypergraph spectrum components and frequency coefficients of point clouds in both ideal and noisy settings. We establish an analytical connection between hypergraph frequencies and structural features. We further evaluate the efficacy of hypergraph spectrum estimation in two common applications of sampling and denoising of point clouds for which we provide specific hypergraph filter design and spectral properties. Experimental results demonstrate the strength of hypergraph signal processing as a tool in characterizing the underlying properties of 3D point clouds.

Short-form video social media shifts away from the traditional media paradigm by telling the audience a dynamic story to attract their attention. In particular, different combinations of everyday objects can be employed to represent a unique scene that is both interesting and understandable. Offered by the same company, TikTok and Douyin are popular examples of such new media that has become popular in recent years, while being tailored for different markets (e.g. the United States and China). The hypothesis that they express cultural differences together with media fashion and social idiosyncrasy is the primary target of our research. To that end, we first employ the Faster Regional Convolutional Neural Network (Faster R-CNN) pre-trained with the Microsoft Common Objects in COntext (MS-COCO) dataset to perform object detection. Based on a suite of objects detected from videos, we perform statistical analysis including label statistics, label similarity, and label-person distribution. We further use the Two-Stream Inflated 3D ConvNet (I3D) pre-trained with the Kinetics dataset to categorize and analyze human actions. By comparing the distributional results of TikTok and Douyin, we uncover a wealth of similarity and contrast between the two closely related video social media platforms along the content dimensions of object quantity, object categories, and human action categories., Accepted by IEEE Big Data 2020

Attention-based models have made tremendous progress on end-to-end automatic speech recognition(ASR) recently. However, the conventional transformer-based approaches usually generate the sequence results token by token from left to right, leaving the right-to-left contexts unexploited. In this work, we introduce a bidirectional speech transformer to utilize the different directional contexts simultaneously. Specifically, the outputs of our proposed transformer include a left-to-right target, and a right-to-left target. In inference stage, we use the introduced bidirectional beam search method, which can not only generate left-to-right candidates but also generate right-to-left candidates, and determine the best hypothesis by the score. To demonstrate our proposed speech transformer with a bidirectional decoder(STBD), we conduct extensive experiments on the AISHELL-1 dataset. The results of experiments show that STBD achieves a 3.6\% relative CER reduction(CERR) over the unidirectional speech transformer baseline. Besides, the strongest model in this paper called STBD-Big can achieve 6.64\% CER on the test set, without language model rescoring and any extra data augmentation strategies., Comment: Accepted by InterSpeech 2020

Even video telephony services have been pervasively applied, it is still a challenge to provide satisfactory quality of experience to users. Multipath transmission is a promising solution ...

Learning and processing of signals over hypergraph models have gained substantial traction owing to the ability of hypergraphs in characterizing multilateral interactions. In this work, we explore hypergraph spectral analysis and provide alternative definitions of frequency domain operations that are practically useful in image processing. We analyze hypergraph spectral properties and present several application examples, including compression, edge detection and segmentation. Successful experiment results demonstrate the effectiveness and the future prospect of the proposed hypergraph frequency operations in image processing.

Few-shot semantic segmentation aims to learn to segment new object classes with only a few annotated examples, which has a wide range of real-world applications. Most existing methods either focus on the restrictive setting of one-way few-shot segmentation or suffer from incomplete coverage of object regions. In this paper, we propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes, capable of capturing diverse and fine-grained object features. In addition, we propose to leverage unlabeled data to enrich our part-aware prototypes, resulting in better modeling of intra-class variations of semantic objects. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes based on labeled and unlabeled images. Extensive experimental evaluations on two benchmarks show that our method outperforms the prior art with a sizable margin (Code is available at: https://github.com/Xiangyi1996/PPNet-PyTorch).

Recent skeleton-based action recognition approaches achieve great improvement by using recurrent neural network (RNN) models. Currently, these approaches build an end-to-end network from coordinates of joints to class categories and improve accuracy by extending RNN to spatial domains. First, while such well-designed models and optimization strategies explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, according to the evolution of previous work, we select a set of simple geometric features, and then seperately feed each type of features to a three-layer LSTM framework. Second, we propose a multistream LSTM architecture with a new smoothed score fusion technique to learn classification from different geometric feature streams. Furthermore, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and the fusion results achieve the state-of-the-art performance on four datasets. We also show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art., This paper is accepted by AAAI 2020

Currently RNN-based methods achieve excellent performance on action recognition using skeletons. But the inputs of these approaches are limited to coordinates of joints, and they improve the performance mainly by extending RNN models in different ways and exploring relations of body parts directly from joint coordinates. Our method utilizes a universal spatial model perpendicular to the RNN model enhancement. Specifically, we propose two simple geometric features, inspired by previous work. With experiments on a 3-layer LSTM (Long Short-Term Memory) framework, we find that the geometric relational features based on vectors and normal vectors outperform other methods and achieve state-of-art results on two datasets. Moreover, we show that utilizing our features as input requires less data for training.

Establishing semantic correspondence is a core problem in computer vision and remains challenging due to large intra-class variations and lack of annotated data. In this paper, we aim to incorporate global semantic context in a flexible manner to overcome the limitations of prior work that relies on local semantic representations. To this end, we first propose a context-aware semantic representation that incorporates spatial layout for robust matching against local ambiguities. We then develop a novel dynamic fusion strategy based on attention mechanism to weave the advantages of both local and context features by integrating semantic cues from multiple scales. We instantiate our strategy by designing an end-to-end learnable deep network, named as Dynamic Context Correspondence Network (DCCNet). To train the network, we adopt a multi-auxiliary task loss to improve the efficiency of our weakly-supervised learning procedure. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including PF-Pascal, PF-Willow, and TSS, demonstrating its effectiveness and generality., ICCV 2019

Signal processing over graphs has recently attracted significant attention for dealing with the structured data. Normal graphs, however, only model pairwise relationships between nodes and are not effective in representing and capturing some high-order relationships of data samples, which are common in many applications, such as Internet of Things (IoT). In this article, we propose a new framework of hypergraph signal processing (HGSP) based on the tensor representation to generalize the traditional graph signal processing (GSP) to tackle high-order interactions. We introduce the core concepts of HGSP and define the hypergraph Fourier space. We then study the spectrum properties of hypergraph Fourier transform (HGFT) and explain its connection to mainstream digital signal processing. We derive the novel hypergraph sampling theory and present the fundamentals of hypergraph filter design based on the tensor framework. We present HGSP-based methods for several signal processing and data analysis applications. Our experimental results demonstrate significant performance improvement using our HGSP framework over some traditional signal processing solutions.

Multipath diversity significantly impacts multipath transmission quality. Enough multipath diversity would minimize the negative influence brought by an individual path, thus improving tolerance capability of network congestion and failure. However, multipath diversity is hard to guarantee on overlay networks because of inaccurate awareness of underlay network and multipath generating methods considering little about underlay diversity. In this paper, we design a multi-dimension spatial method for topology awareness and multipath generating. Analyzing that the complicated underlay networks with multiple autonomous systems reduce the accuracy of network positioning for topology awareness, we decompose the underlay networks into multiple dimensions, namely intra and inter autonomous system dimensions. We generate independent views for each autonomous system and merge views by embedding exchange points. Then, we design some spatial mechanisms to evaluate link diversity and to constrain multipath generating. Based on the multi-dimensional view, multipath generating is conducted in inter and intra autonomous system phases. Experiments demonstrate that the proposed method improves topology awareness accuracy and guarantees multipath diversity better and the transmission quality is improved.

We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses on grounding the query as a whole, temporal dependencies and reasoning between events within the text are not fully considered. In this paper, we propose a novel Temporal Compositional Modular Network (TCMN) where a tree attention network first automatically decomposes a sentence into three descriptions with respect to the main event, context event and temporal signal. Two modules are then utilized to measure the visual similarity and location similarity between each segment and the decomposed descriptions. Moreover, since the main event and context event may rely on different modalities (RGB or optical flow), we use late fusion to form an ensemble of four models, where each model is independently trained by one combination of the visual input. Experiments show that our model outperforms the state-of-the-art methods on the TEMPO dataset.

The brushless doubly-fed induction generator (BDFIG) is demonstrating great potentiality in variable speed constant frequency power generation applications owing to the absence of the brushes and slip rings. To take full advantage of the brushless structure, the sensorless direct voltage control method is an attractive method. However, due to the harmonics caused by nonlinear load, the performance of conventional control method may be degraded and the efficiency of the generators would be reduced. This paper proposes an improved sensorless direct voltage control of the stand-alone BDFIG feeding nonlinear loads. On the one hand, to cancel the effect of the harmonic components, the voltage decomposition has been employed in sensorless control to get the fundamental components. On the other hand, the harmonic voltage elimination with reduced load current sensors has been applied to the load side converter. Experimental results with 30 kVA BDFIG are presented to verify the proposed control method.

Cascading failure in multilayer complex systems draws significant attentions from both industry and academia nowadays. In this paper, we propose a scalable tensor-based framework to represent the interdependent multilayer network, and use this framework to analyze the failure propagation based on a susceptible-infectious-susceptible (SIS) epidemic model. Specifically, the transition equations and transition tensor are derived to characterize the behavior of failure propagation. We show that the spectral radius of transition tensor is a failure indicator with an explicit failure threshold to measure the system reliability. Moreover, to make the failure indicator analytically tractable and computationally efficient, we derive its upper and lower bounds, as well as its approximated expressions in special cases as functions of the adjacency tensor and epidemic parameters. Our analytical results are evaluated by simulations in a set of multilayer networks generated by random graphs, which show that our results can achieve the desired performance compared with other benchmark approximation methods.

Congestion control provides the endpoint the ability to probe available bandwidth and prevents the network falling into severe congestion. But loss based congestion control algorithms designed for ...

Although the recent success of convolutional neural network (CNN) advances state-of-the-art saliency prediction in static images, few work has addressed the problem of predicting attention in videos. On the other hand, we find that the attention of different subjects consistently focuses on a single face in each frame of videos involving multiple faces. Therefore, we propose in this paper a novel deep learning (DL) based method to predict salient face in multiple-face videos, which is capable of learning features and transition of salient faces across video frames. In particular, we first learn a CNN for each frame to locate salient face. Taking CNN features as input, we develop a multiple-stream long short-term memory (M-LSTM) network to predict the temporal transition of salient faces in video sequences. To evaluate our DL-based method, we build a new eye-tracking database of multiple-face videos. The experimental results show that our method outperforms the prior state-of-the-art methods in predicting visual attention on faces in multiple-face videos.

RNN-based approaches have achieved outstanding performance on action recognition with skeleton inputs. Currently these methods limit their inputs to coordinates of joints and improve the accuracy mainly by extending RNN models to spatial domains in various ways. While such models explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, we select a set of simple geometric features, motivated by the evolution of previous work. With experiments on a 3-layer LSTM framework, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and achieve state-of-art results on four datasets. Further, we show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

With the explosive growth of semantic data on the Web over the past years, many large-scale RDF knowledge bases with billions of facts are generating. This poses significant challenges for the storage and query of big RDF graphs. Current systems still have many limitations in processing big RDF graphs including scalability and real-time. In this paper, we introduce the SparkRDF, an elastic discreted RDF graph processing engine with distributed memory. To reduce the high I/O and communication cost in distributed processing platforms, SparkRDF implements SPARQL query based on Spark, a novel in-memory distributed computing framework for big data processing. All the intermediate results are modeled as Resilient Discreted SubGraph, which are cached in the distributed memory to support fast iterative join operations. To cut down the search space and avoid the overhead of memory, we split the RDF graph into the small Multi-layer Elastic SubGraph based on the relations and classes. For SPARQL query optimization, SparkRDF deploys a serials of optimization strategies, leading to effective reduction on the size of intermediate results, the number of joins and the cost of communication. Our extensive evaluation demonstrates that SparkRDF can efficiently implement non-selective joins faster than both current state-of-the-art distributed and centralized stores, while being able to process other queries in real time, scaling linearly to the amount of data.

An effective communication protocol for wireless sensor networks should provide a good quality of service (QoS) for network data transfer, but the traditional layering protocol stack can't sufficiently meet this complex requirement. So a new cross-layer communication protocol - directed spanning tree-based power control (PCDST) is proposed. In the algorithm, a directed spanning tree routing algorithm is discussed to resolve unnecessary energy consumption during data transmission. Regarded as a key factor, transmission power of node should be controlled to decrease the intensity of communication contention. A NUM model is constructed to control power. Based on Prim-dual Algorithm, a Lagrange dual function is designed and optimal solution set of transmission power is deduced. Make use of these cross-layer optimal schemes, WSNs achieve energy efficiency and smooth communication mode. The simulation results show that the performance of the proposed protocol is much better in energy efficiency and improvement of communication throughput. Index Terms - Wireless sensor networks; communication protocol; cross-layer optimization; power control; spanning tree.

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction., This paper is accepted by NAACL'21