Search

In this work, we have experimentally demonstrated the refractive index (RI) sensing characteristics of a S-shape taper refractometer (STR) based on modal interference theory. Our preliminary theoretical analysis reveals that there exists a critical cladding mode, which is essential for understanding the sensing characteristics. When the dominant cladding mode involved in a core-cladding interference is close to the critical cladding mode, the resulting RI sensitivity tends to reach a maximum value. Moreover, both the critical and the dominant cladding mode are dependent on ambient RI. Our sensor achieves a high RI sensitivity of 2109.7 nm/RIU for a transmission dip at around 1505 nm with a measurement range of 1.36 to 1.39.

A systematic comparison of fiber-based gas pressure sensors using different types of hollow-core photonic crystal fibers (PCFs) is conducted. The sensor was fabricated by splicing a segment of hollow-core silica capillary between a single mode fiber (SMF) and a hollow-core PCF. A Fabry-Perot (FP) cavity configuration is thus formed with the capillary tube as the sensing cavity, while the PCF acts as gas passage to the external environment. External pressure change would lead to the variation in the refractive index of air in the sensing cavity, resulting in a wavelength shift of the interference dips. The pressure sensitivity is measured to be around 4 nm/MPa, with a high linearity of 99.7% or above, regardless of the type of the PCFs we used as gas inlet. Among the four different PCFs, the large-mode-area (LMA) PCF shows the highest fringe contrast in the reflection spectrum. Modal analysis reveals that this is due to the high reflectivity caused by the solid core of LMA-PCF. Our experimental results also indicate that the length of the sensing cavity, as well as the offset fusion splice will influence the fringe contrast. The sensor has potential application in gas pressure sensing for advantages of high sensitivity, compact size and ease of fabrication.

The revolution of artificial intelligence (AI) is transforming major industries worldwide. With accurate inferencing, AI has caught the attention of many engineers and scientists. Promisingly, hardware-in the-loop (HIL) emulation can adopt this type of modeling method as one of the alternatives after comprehensive investigation. This paper proposes an approach for emulating power electronic motor drive transients for advanced transportation applications (ATAs) using machine learning building blocks (MLBBs) without any traditional circuit-oriented transient solver. The more electric aircraft (MEA) power system is chosen as a case study to validate the real-time emulation performance of MLBBs. Inside MLBBs, neural networks (NNs) have been applied to build component-level, device-level, and system-level models for various equipment. These models are well trained in a cluster and transplanted into the field-programmable gate array (FPGA) based hardware platform. Finally, MLBB emulation results are compared with PSCAD/EMTDC for system-level and SaberRD for device-level, which showed high consistency for model accuracy and high speed-up for hardware execution.

It is still a challenging task to provide satisfactory quality of experience (QoE) to users by traditional routing mechanism in current public Internet for real-time multimedia services with high bandwidth and strict end-to-end latency constraint. Multipath transmission is a promising way to improve the performance of data delivery, due to its advantage in bandwidth aggregation capacity over multiple paths to speed up high volume transfers between endpoints. However, existing multipath transmission has many disadvantages such as the requirements for end-host support (i.e. the necessary multi-homed network setting, and operating system or upper applications updates of user devices), and blindingly pursuing maximum of transmission throughput. Path allocation and end-to-end transmission control are two important factors affecting the quality of multipath transmission. As an emerging network architecture, software-defined networking (SDN) is flexible, manageable, and responsive to rapid changes in traffic requirements. In this paper, we propose a multipath transport scheme based on SDN with segment routing (SR) which can meet the bandwidth and low end-to-end transmission latency requirements of real-time interactive multimedia services. The centralized SDN controller is extended to allocate multiple disjoint paths meeting bandwidth requirement meanwhile balancing the network load. The proprietary multipath media server (MMS) dynamically controls the assignment between subflows and routing paths to reduce the end-to-end transmission latency and thus improve QoE of end-users. To show the feasibility of our approach, we construct an SDN platform using Mininet. Simulation results show that our approach performs better compared to the conventional approaches in terms of traffic balancing and end-to-end transmission delay.

Hypergraph spectral analysis has emerged as an effective tool processing complex data structures in data analysis. The surface of a three-dimensional (3D) point cloud, and the multilateral relationship among their points can be naturally captured by the high-dimensional hyperedges. This work investigates the power of hypergraph spectral analysis in unsupervised segmentation of 3D point clouds. We estimate, and order the hypergraph spectrum from observed point cloud coordinates. By trimming the redundancy from the estimated hypergraph spectral space based on spectral component strengths, we develop a clustering-based segmentation method. We apply the proposed method to various point clouds, and analyze their respective spectral properties. Our experimental results demonstrate the effectiveness and efficiency of the proposed segmentation method.

We address the problem of retrieving a specific moment from an untrimmed video by natural language. It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they do not fully consider the temporal contexts between temporal moments. In this paper, we model the temporal context between video moments by a set of predefined two-dimensional maps under different temporal scales. For each map, one dimension indicates the starting time of a moment and the other indicates the duration. These 2D temporal maps can cover diverse video moments with different lengths, while representing their adjacent contexts at different temporal scales. Based on the 2D temporal maps, we propose a Multi-Scale Temporal Adjacency Network (MS-2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal contexts at each scale, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed MS-2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our MS-2D-TAN outperforms the state of the art.

Few-shot video classification aims to learn new video categories with only a few labeled examples, alleviating the burden of costly annotation in real-world applications. However, it is particularly challenging to learn a class-invariant spatial-temporal representation in such a setting. To address this, we propose a novel matching-based few-shot learning strategy for video sequences in this work. Our main idea is to introduce an implicit temporal alignment for a video pair, capable of estimating the similarity between them in an accurate and robust manner. Moreover, we design an effective context encoding module to incorporate spatial and feature channel context, resulting in better modeling of intra-class variations. To train our model, we develop a multi-task loss for learning video matching, leading to video features with better generalization. Extensive experimental results on two challenging benchmarks, show that our method outperforms the prior arts with a sizable margin on Something-Something-V2 and competitive results on Kinetics.

Real time video traffic has stringent delay requirement and is bandwidth demanding when streaming over the internet. Single path transmission may provide unsatisfactory picture quality to users due to packet loss, jitter, and low bandwidth. Multipath transmission is another option to achieve better service by aggregating bandwidth and exploiting path diversity. As utility maximization theory shows, to achieve better satisfaction is to provide higher throughput and lower frame delivery delay for real time video traffic. The congestion control is responsible to probe maximum available bandwidth and packet scheduling is to take advantages of path diversity, which are two key research points in multipath context and are also the main concern in this paper. Given the BBR congestion control algorithm causes high-packet loss rate and large transmission delay when BBR flows competing for bandwidth resource, a delay response BBR (Delay-BBR) algorithm is proposed for real time video transmission. The main idea is that the sender will actively reduce the rate to let intermediate routers drain the excess occupied buffer once round trip delay signal exceeds a certain threshold. The results show that the delay-BBR can achieve lower transmission delay and lower packet loss rate when compared with QUIC-BBR and WebRTC-BBR. The rate stable feature makes delay-BBR appropriate for real time video transmission. Moreover, a packet scheduling algorithm works in coordination with the rate control algorithm achieves lower frame delivery delay further compared with the benchmark algorithms.

Despite the recent success of deep neural networks, it remains challenging to effectively model the long-tail class distribution in visual recognition tasks. To address this problem, we first investigate the performance bottleneck of the two-stage learning framework via ablative study. Motivated by our discovery, we propose a unified distribution alignment strategy for long-tail visual recognition. Specifically, we develop an adaptive calibration function that enables us to adjust the classification scores for each data point. We then introduce a generalized re-weight method in the two-stage learning to balance the class prior, which provides a flexible and unified solution to diverse scenarios in visual recognition tasks. We validate our method by extensive experiments on four tasks, including image classification, semantic segmentation, object detection, and instance segmentation. Our approach achieves the state-of-the-art results across all four recognition tasks with a simple and unified framework. The code and models will be made publicly available at: https://github.com/Megvii-BaseDetection/DisAlign, Accepted by CVPR 2021

Transferring the sentiment of an image is an unexplored research topic in the area of computer vision. This work proposes a novel framework consisting of a reference image retrieval step and a global sentiment transfer step to transfer sentiments of images according to a given sentiment tag. The proposed image retrieval algorithm is based on the SSIM index. The retrieved reference images by the proposed algorithm are more content-related against the algorithm based on the perceptual loss. Therefore can lead to a better image sentiment transfer result. In addition, we propose a global sentiment transfer step, which employs an optimization algorithm to iteratively transfer sentiment of images based on feature maps produced by the Densenet121 architecture. The proposed sentiment transfer algorithm can transfer the sentiment of images while ensuring the content structure of the input image intact. The qualitative and quantitative experiments demonstrate that the proposed sentiment transfer framework outperforms existing artistic and photorealistic style transfer algorithms in making reliable sentiment transfer results with rich and exact details.

Scene graph generation is an important visual understanding task with a broad range of vision applications. Despite recent tremendous progress, it remains challenging due to the intrinsic long-tailed class distribution and large intra-class variation. To address these issues, we introduce a novel confidence-aware bipartite graph neural network with adaptive message propagation mechanism for unbiased scene graph generation. In addition, we propose an efficient bi-level data resampling strategy to alleviate the imbalanced data distribution problem in training our graph network. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including Visual Genome, Open Images V4/V6, demonstrating its effectiveness and generality., Comment: Accepted by CVPR2021

This article is mainly aimed at defining and verifying whether the project involves environmentally sensitive areas in the process of environmental intelligent survey, calculating the position coordinate information of the target GPS position in the aerial image map under the condition of Gaussian projection, and further calculating the four corners of the image and the position of other target points under the WGS84 coordinate system. First of all, based on the POS data information of aerial equipment, this paper calculates the corresponding Gaussian projection and POS data information, then the pseudo-position information of image center coordinates is further calculated; Second, Based on the coordinates information of gauss projection, this paper calculates the heading open angle and side open angle of the camera, then calculate the width and height of the image for pose correction, and finally get the coordinate position of the corrected image center point. Finally, the GPS positions of other target points can be calculated by pixel coordinates of images, and the calculated results are pasted on Google earth for precision comparison. Experimental results show that the proposed method can accurately calculate the GPS positions of targets in aerial images.

Incremental learning of semantic segmentation has emerged as a promising strategy for visual scene interpretation in the open- world setting. However, it remains challenging to acquire novel classes in an online fashion for the segmentation task, mainly due to its continuously-evolving semantic label space, partial pixelwise ground-truth annotations, and constrained data availability. To ad- dress this, we propose an incremental learning strategy that can fast adapt deep segmentation models without catastrophic forgetting, using a streaming input data with pixel annotations on the novel classes only. To this end, we develop a uni ed learning strategy based on the Expectation-Maximization (EM) framework, which integrates an iterative relabeling strategy that lls in the missing labels and a rehearsal-based incremental learning step that balances the stability-plasticity of the model. Moreover, our EM algorithm adopts an adaptive sampling method to select informative train- ing data and a class-balancing training strategy in the incremental model updates, both improving the e cacy of model learning. We validate our approach on the PASCAL VOC 2012 and ADE20K datasets, and the results demonstrate its superior performance over the existing incremental methods., Comment: Accepted by ACM MM'21

Short-form video social media shifts away from the traditional media paradigm by telling the audience a dynamic story to attract their attention. In particular, different combinations of everyday objects can be employed to represent a unique scene that is both interesting and understandable. Offered by the same company, TikTok and Douyin are popular examples of such new media that has become popular in recent years, while being tailored for different markets (e.g. the United States and China). The hypothesis that they express cultural differences together with media fashion and social idiosyncrasy is the primary target of our research. To that end, we first employ the Faster Regional Convolutional Neural Network (Faster R-CNN) pre-trained with the Microsoft Common Objects in COntext (MS-COCO) dataset to perform object detection. Based on a suite of objects detected from videos, we perform statistical analysis including label statistics, label similarity, and label-person distribution. We further use the Two-Stream Inflated 3D ConvNet (I3D) pre-trained with the Kinetics dataset to categorize and analyze human actions. By comparing the distributional results of TikTok and Douyin, we uncover a wealth of similarity and contrast between the two closely related video social media platforms along the content dimensions of object quantity, object categories, and human action categories., Accepted by IEEE Big Data 2020

Even video telephony services have been pervasively applied, it is still a challenge to provide satisfactory quality of experience to users. Multipath transmission is a promising solution ...

Few-shot semantic segmentation aims to learn to segment new object classes with only a few annotated examples, which has a wide range of real-world applications. Most existing methods either focus on the restrictive setting of one-way few-shot segmentation or suffer from incomplete coverage of object regions. In this paper, we propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes, capable of capturing diverse and fine-grained object features. In addition, we propose to leverage unlabeled data to enrich our part-aware prototypes, resulting in better modeling of intra-class variations of semantic objects. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes based on labeled and unlabeled images. Extensive experimental evaluations on two benchmarks show that our method outperforms the prior art with a sizable margin (Code is available at: https://github.com/Xiangyi1996/PPNet-PyTorch).

Recent skeleton-based action recognition approaches achieve great improvement by using recurrent neural network (RNN) models. Currently, these approaches build an end-to-end network from coordinates of joints to class categories and improve accuracy by extending RNN to spatial domains. First, while such well-designed models and optimization strategies explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, according to the evolution of previous work, we select a set of simple geometric features, and then seperately feed each type of features to a three-layer LSTM framework. Second, we propose a multistream LSTM architecture with a new smoothed score fusion technique to learn classification from different geometric feature streams. Furthermore, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and the fusion results achieve the state-of-the-art performance on four datasets. We also show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art., This paper is accepted by AAAI 2020

Background: Obesity plays crucial roles in the development of cardiovascular diseases. However, the mechanisms that link obesity and cardiovascular diseases remain elusive. Compelling evidence indicates that adipokines play an important role in obesity-related cardiovascular diseases. Here, we found a new adipokine-named family with sequence similarity 19, member A5 (FAM19A5), a protein with unknown function that was predicted to be distantly related to the CC-chemokine family. We aimed to test whether adipose-derived FAM19A5 regulates vascular pathology on injury. Methods: DNA cloning, protein expression, purification, and N-terminal sequencing were applied to characterize FAM19A5. Adenovirus infection and siRNA transfection were performed to regulate FAM19A5 expression. Balloon and wire injury were performed in vivo on the rat carotid arteries and mouse femoral arteries, respectively. Bioinformatics analysis, radioactive ligand-receptor binding assays, receptor internalization, and calcium mobilization assays were used to identify the functional receptor for FAM19A5. Results: We first characterized FAM19A5 as a secreted protein, and the first 43 N-terminal amino acids were the signal peptides. Both FAM19A5 mRNA and protein were abundantly expressed in the adipose tissue but were downregulated in obese mice. Overexpression of FAM19A5 markedly inhibited vascular smooth muscle cell proliferation and migration and neointima formation in the carotid arteries of balloon-injured rats. Accordingly, FAM19A5 silencing in adipocytes significantly promoted vascular smooth muscle cell activation. Adipose-specific FAM19A5 transgenic mice showed greater attenuation of neointima formation compared with wild-type littermates fed with or without Western-style diet. We further revealed that sphingosine-1-phosphate receptor 2 was the functional receptor for FAM19A5, with a dissociation constant ( K d ) of 0.634 nmol/L. Inhibition of sphingosine-1-phosphate receptor 2 or its downstream G12/13-RhoA signaling circumvented the suppressive effects of FAM19A5 on vascular smooth muscle cell proliferation and migration. Conclusions: We revealed that a novel adipokine, FAM19A5, was capable of inhibiting postinjury neointima formation via sphingosine-1-phosphate receptor 2-G12/13-RhoA signaling. Downregulation of FAM19A5 during obesity may trigger cardiometabolic diseases.

Currently RNN-based methods achieve excellent performance on action recognition using skeletons. But the inputs of these approaches are limited to coordinates of joints, and they improve the performance mainly by extending RNN models in different ways and exploring relations of body parts directly from joint coordinates. Our method utilizes a universal spatial model perpendicular to the RNN model enhancement. Specifically, we propose two simple geometric features, inspired by previous work. With experiments on a 3-layer LSTM (Long Short-Term Memory) framework, we find that the geometric relational features based on vectors and normal vectors outperform other methods and achieve state-of-art results on two datasets. Moreover, we show that utilizing our features as input requires less data for training.

Establishing semantic correspondence is a core problem in computer vision and remains challenging due to large intra-class variations and lack of annotated data. In this paper, we aim to incorporate global semantic context in a flexible manner to overcome the limitations of prior work that relies on local semantic representations. To this end, we first propose a context-aware semantic representation that incorporates spatial layout for robust matching against local ambiguities. We then develop a novel dynamic fusion strategy based on attention mechanism to weave the advantages of both local and context features by integrating semantic cues from multiple scales. We instantiate our strategy by designing an end-to-end learnable deep network, named as Dynamic Context Correspondence Network (DCCNet). To train the network, we adopt a multi-auxiliary task loss to improve the efficiency of our weakly-supervised learning procedure. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including PF-Pascal, PF-Willow, and TSS, demonstrating its effectiveness and generality., ICCV 2019

Signal processing over graphs has recently attracted significant attention for dealing with the structured data. Normal graphs, however, only model pairwise relationships between nodes and are not effective in representing and capturing some high-order relationships of data samples, which are common in many applications, such as Internet of Things (IoT). In this article, we propose a new framework of hypergraph signal processing (HGSP) based on the tensor representation to generalize the traditional graph signal processing (GSP) to tackle high-order interactions. We introduce the core concepts of HGSP and define the hypergraph Fourier space. We then study the spectrum properties of hypergraph Fourier transform (HGFT) and explain its connection to mainstream digital signal processing. We derive the novel hypergraph sampling theory and present the fundamentals of hypergraph filter design based on the tensor framework. We present HGSP-based methods for several signal processing and data analysis applications. Our experimental results demonstrate significant performance improvement using our HGSP framework over some traditional signal processing solutions.

We address the problem of video moment localization with natural language, i.e. localizing a video segment described by a natural language sentence. While most prior work focuses on grounding the query as a whole, temporal dependencies and reasoning between events within the text are not fully considered. In this paper, we propose a novel Temporal Compositional Modular Network (TCMN) where a tree attention network first automatically decomposes a sentence into three descriptions with respect to the main event, context event and temporal signal. Two modules are then utilized to measure the visual similarity and location similarity between each segment and the decomposed descriptions. Moreover, since the main event and context event may rely on different modalities (RGB or optical flow), we use late fusion to form an ensemble of four models, where each model is independently trained by one combination of the visual input. Experiments show that our model outperforms the state-of-the-art methods on the TEMPO dataset.

Congestion control provides the endpoint the ability to probe available bandwidth and prevents the network falling into severe congestion. But loss based congestion control algorithms designed for ...

Although the recent success of convolutional neural network (CNN) advances state-of-the-art saliency prediction in static images, few work has addressed the problem of predicting attention in videos. On the other hand, we find that the attention of different subjects consistently focuses on a single face in each frame of videos involving multiple faces. Therefore, we propose in this paper a novel deep learning (DL) based method to predict salient face in multiple-face videos, which is capable of learning features and transition of salient faces across video frames. In particular, we first learn a CNN for each frame to locate salient face. Taking CNN features as input, we develop a multiple-stream long short-term memory (M-LSTM) network to predict the temporal transition of salient faces in video sequences. To evaluate our DL-based method, we build a new eye-tracking database of multiple-face videos. The experimental results show that our method outperforms the prior state-of-the-art methods in predicting visual attention on faces in multiple-face videos.

RNN-based approaches have achieved outstanding performance on action recognition with skeleton inputs. Currently these methods limit their inputs to coordinates of joints and improve the accuracy mainly by extending RNN models to spatial domains in various ways. While such models explore relations between different parts directly from joint coordinates, we provide a simple universal spatial modeling method perpendicular to the RNN model enhancement. Specifically, we select a set of simple geometric features, motivated by the evolution of previous work. With experiments on a 3-layer LSTM framework, we observe that the geometric relational features based on distances between joints and selected lines outperform other features and achieve state-of-art results on four datasets. Further, we show the sparsity of input gate weights in the first LSTM layer trained by geometric features and demonstrate that utilizing joint-line distances as input require less data for training.

Total organic carbon (TOC) is an important parameter for characterizing shale gas and oil reservoirs. Estimation of TOC from well logs has previously been achieved by an empirical model. The radial basis function (RBF) neural network is a new quantitative method that can generate a smooth and continuous function of several input variables to approximate the unknown forward model. We investigated the basic principles of the RBF including network structure, basis function, network training method, and its application in the TOC prediction. The nearest neighbor algorithm was selected for the network training. Then, the Gaussian width was investigated to improve the TOC prediction accuracy through leave-one-out cross-validation. Finally, field cases of organic shale were studied for the TOC prediction, and the prediction results using the RBF method were compared with those of the [Formula: see text] method. Furthermore, according to sensitive attribute ranking, the impacts of different input logs on the predicted results were also investigated through various experiments, and the best network model was finally chosen. The error analysis between the prediction results and lab-measured TOC in some examples indicated that the new approach is more accurate than a single empirical regression method and more flexible than the [Formula: see text] method.

An effective communication protocol for wireless sensor networks should provide a good quality of service (QoS) for network data transfer, but the traditional layering protocol stack can't sufficiently meet this complex requirement. So a new cross-layer communication protocol - directed spanning tree-based power control (PCDST) is proposed. In the algorithm, a directed spanning tree routing algorithm is discussed to resolve unnecessary energy consumption during data transmission. Regarded as a key factor, transmission power of node should be controlled to decrease the intensity of communication contention. A NUM model is constructed to control power. Based on Prim-dual Algorithm, a Lagrange dual function is designed and optimal solution set of transmission power is deduced. Make use of these cross-layer optimal schemes, WSNs achieve energy efficiency and smooth communication mode. The simulation results show that the performance of the proposed protocol is much better in energy efficiency and improvement of communication throughput. Index Terms - Wireless sensor networks; communication protocol; cross-layer optimization; power control; spanning tree.

In order to study the effect of friction state on contact reaction force of rollers and bearing's inner ring and outer ring, contact analysis model was established by the local coordinate system and finite element method for rolling bearings of universal steel mill. Studied the changing rule of force parameters of rolling bearings during steel rolling, the results show that contact reaction is a steady increase with increasing of contact angle, with a step distribution. The rolling reaction force to achieve maximum near the region the body, when the magnitude of contact angle is 90°, within the scope of friction coefficient from 0.01 to 0.07, the fluctuation of rolling reaction force to achieve maximum near the region body when the magnitude of contact angle is 45°. It is important to define limit load of bearings and formulate rolling order.

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction., This paper is accepted by NAACL'21