Search

Instance segmentation in videos, which aims to segment and track multiple objects in video frames, has garnered a flurry of research attention in recent years. In this paper, we present a novel weakly supervised framework with \textbf{S}patio-\textbf{T}emporal \textbf{C}ollaboration for instance \textbf{Seg}mentation in videos, namely \textbf{STC-Seg}. Concretely, STC-Seg demonstrates four contributions. First, we leverage the complementary representations from unsupervised depth estimation and optical flow to produce effective pseudo-labels for training deep networks and predicting high-quality instance masks. Second, to enhance the mask generation, we devise a puzzle loss, which enables end-to-end training using box-level annotations. Third, our tracking module jointly utilizes bounding-box diagonal points with spatio-temporal discrepancy to model movements, which largely improves the robustness to different object appearances. Finally, our framework is flexible and enables image-level instance segmentation methods to operate the video-level task. We conduct an extensive set of experiments on the KITTI MOTS and YT-VIS datasets. Experimental results demonstrate that our method achieves strong performance and even outperforms fully supervised TrackR-CNN and MaskTrack R-CNN. We believe that STC-Seg can be a valuable addition to the community, as it reflects the tip of an iceberg about the innovative opportunities in the weakly supervised paradigm for instance segmentation in videos.

Existing approaches in video captioning concentrate on exploring global frame features in the uncompressed videos, while the free of charge and critical saliency information already encoded in the compressed videos is generally neglected. We propose a video captioning method which operates directly on the stored compressed videos. To learn a discriminative visual representation for video captioning, we design a residuals-assisted encoder (RAE), which spots regions of interest in I-frames under the assistance of the residuals frames. First, we obtain the spatial attention weights by extracting features of residuals as the saliency value of each location in I-frame and design a spatial attention module to refine the attention weights. We further propose a temporal gate module to determine how much the attended features contribute to the caption generation, which enables the model to resist the disturbance of some noisy signals in the compressed videos. Finally, Long Short-Term Memory is utilized to decode the visual representations into descriptions. We evaluate our method on two benchmark datasets and demonstrate the effectiveness of our approach.

Feature pyramid network (FPN) is a critical component in modern object detection frameworks. The performance gain in most of the existing FPN variants is mainly attributed to the increase of computational burden. An attempt to enhance the FPN is enriching the spatial information by expanding the receptive fields, which is promising to largely improve the detection accuracy. In this paper, we first investigate how expanding the receptive fields affect the accuracy and computational costs of FPN. We explore a baseline model called inception FPN in which each lateral connection contains convolution filters with different kernel sizes. Moreover, we point out that not all objects need such a complicated calculation and propose a new dynamic FPN (DyFPN). The output features of DyFPN will be calculated by using the adaptively selected branch according to a dynamic gating operation. Therefore, the proposed method can provide a more efficient dynamic inference for achieving a better trade-off between accuracy and computational cost. Extensive experiments conducted on MS-COCO benchmark demonstrate that the proposed DyFPN significantly improves performance with the optimal allocation of computation resources. For instance, replacing inception FPN with DyFPN reduces about 40% of its FLOPs while maintaining similar high performance.

Vision and voice are two vital keys for agents' interaction and learning. In this paper, we present a novel indoor navigation model called Memory Vision-Voice Indoor Navigation (MVV-IN), which receives voice commands and analyzes multimodal information of visual observation in order to enhance robots' environment understanding. We make use of single RGB images taken by a first-view monocular camera. We also apply a self-attention mechanism to keep the agent focusing on key areas. Memory is important for the agent to avoid repeating certain tasks unnecessarily and in order for it to adapt adequately to new scenes, therefore, we make use of meta-learning. We have experimented with various functional features extracted from visual observation. Comparative experiments prove that our methods outperform state-of-the-art baselines., Comment: 8 pages, 6 figures, 2 tables, accepted at 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2020)

Describing a video automatically with natural language is a challenging task in the area of computer vision. In most cases, the on-site situation of great events is reported in news, but the situation of the off-site spectators in the entrance and exit is neglected which also arouses people's interest. Since the deployment of reporters in the entrance and exit costs lots of manpower, how to automatically describe the behavior of a crowd of off-site spectators is significant and remains a problem. To tackle this problem, we propose a new task called crowd video captioning (CVC) which aims to describe the crowd of spectators. We also provide baseline methods for this task and evaluate them on the dataset WorldExpo'10. Our experimental results show that captioning models have a fairly deep understanding of the crowd in video and perform satisfactorily in the CVC task.

Robotic arm grasping is a fundamental operation in robotic control task goals. Most current methods for robotic grasping focus on RGB-D policy in the table surface scenario or 3D point cloud analysis and inference in the 3D space. Comparing to these methods, we propose a novel real-time multimodal hierarchical encoder-decoder neural network that fuses RGB and depth data to realize robotic humanoid grasping in 3D space with only partial observation. The quantification of raw depth data's uncertainty and depth estimation fusing RGB is considered. We develop a general labeling method to label ground-truth on common RGB-D datasets. We evaluate the effectiveness and performance of our method on a physical robot setup and our method achieves over 90\% success rate in both table surface and 3D space scenarios., Comment: 8 pages, 8 figures, submit to ICRA2020

This paper discusses cooperative stabilization control of rigid formations via an event-based approach. We first design a centralized event-based formation control system, in which a central event controller determines the next triggering time and broadcasts the event signal to all the agents for control input update. We then build on this approach to propose a distributed event control strategy, in which each agent can use its local event trigger and local information to update the control input at its own event time. For both cases, the triggering condition, event function and triggering behavior are discussed in detail, and the exponential convergence of the event-based formation system is guaranteed.

This paper considers a collaborative tracking control problem using a group of fixed-wing unmanned aerial vehicles (UAVs) with constant and non-identical speeds. The dynamics of fixed-wing UAVs are modelled by unicycle-type equations with nonholonomic constraints, assuming that UAVs fly at constant altitudes in the nominal operation mode. The controller is designed such that all fixed-wing UAVs as a group can collaboratively track a desired target's position and velocity. We first present conditions on the relative speeds of tracking UAVs and the target to ensure that the tracking objective can be achieved when UAVs are subject to constant speed constraints. We construct a reference velocity that includes both the target's velocity and position as feedback, which is to be tracked by the group centroid. In this way, all vehicles' headings are controlled such that the group centroid follows a reference trajectory that successfully tracks the target's trajectory. A spacing controller is further devised to ensure that all vehicles stay close to the group centroid trajectory. Trade-offs in the controller design and performance limitations of the target tracking control due to the constant-speed constraint are also discussed in detail. Experimental results with three fixed-wing UAVs tracking a target rotorcraft are provided., Comment: 33 pages (single column). To be published in the AIAA Journal of Guidance, Dynamics, and Control

In this paper, we propose a discontinuous distributed model-independent algorithm for a directed network of Euler-Lagrange agents to track the trajectory of a leader with non-constant velocity. We initially study a fixed network and show that the leader tracking objective is achieved semi-globally exponentially fast if the graph contains a directed spanning tree. By model-independent, we mean that each agent executes its algorithm with no knowledge of the parameter values of any agent's dynamics. Certain bounds on the agent dynamics (including any disturbances) and network topology information are used to design the control gain. This fact, combined with the algorithm's model-independence, results in robustness to disturbances and modelling uncertainties. Next, a continuous approximation of the algorithm is proposed, which achieves practical tracking with an adjustable tracking error. Last, we show that the algorithm is stable for networks that switch with an explicitly computable dwell time. Numerical simulations are given to show the algorithm's effectiveness.

The recently proposed DeGroot-Friedkin model describes the dynamical evolution of individual social power in a social network that holds opinion discussions on a sequence of different issues. This paper revisits that model, and uses nonlinear contraction analysis, among other tools, to establish several novel results. First, we show that for a social network with constant topology, each individual's social power converges to its equilibrium value exponentially fast, whereas previous results only concluded asymptotic convergence. Second, when the network topology is dynamic (i.e., the relative interaction matrix may change between any two successive issues), we show that each individual exponentially forgets its initial social power. Specifically, individual social power is dependent only on the dynamic network topology, and initial (or perceived) social power is forgotten as a result of sequential opinion discussion. Last, we provide an explicit upper bound on an individual's social power as the number of issues discussed tends to infinity; this bound depends only on the network topology. Simulations are provided to illustrate our results., Comment: Extended version of submitted journal paper. Includes additional simulation details

This paper proposes three different distributed event-triggered control algorithms to achieve leader-follower consensus for a network of Euler-Lagrange agents. We firstly propose two model-independent algorithms for a subclass of Euler-Lagrange agents without the vector of gravitational potential forces. By model-independent, we mean that each agent can execute its algorithm with no knowledge of the agent self-dynamics. A variable-gain algorithm is employed when the sensing graph is undirected; algorithm parameters are selected in a fully distributed manner with much greater flexibility compared to all previous work concerning event-triggered consensus problems. When the sensing graph is directed, a constant-gain algorithm is employed. The control gains must be centrally designed to exceed several lower bounding inequalities which require limited knowledge of bounds on the matrices describing the agent dynamics, bounds on network topology information and bounds on the initial conditions. When the Euler-Lagrange agents have dynamics which include the vector of gravitational potential forces, an adaptive algorithm is proposed which requires more information about the agent dynamics but can estimate uncertain agent parameters. For each algorithm, a trigger function is proposed to govern the event update times. At each event, the controller is updated, which ensures that the control input is piecewise constant and saves energy resources. We analyse each controllers and trigger function and exclude Zeno behaviour. Extensive simulations show 1) the advantages of our proposed trigger function as compared to those in existing literature, and 2) the effectiveness of our proposed controllers., Comment: Extended manuscript of journal submission, containing omitted proofs and simulations

This paper analyses the DeGroot-Friedkin model for evolution of the individuals' social powers in a social network when the network topology varies dynamically (described by dynamic relative interaction matrices). The DeGroot-Friedkin model describes how individual social power (self-appraisal, self-weight) evolves as a network of individuals discuss a sequence of issues. We seek to study dynamically changing relative interactions because interactions may change depending on the issue being discussed. In order to explore the problem in detail, two different cases of issue-dependent network topologies are studied. First, if the topology varies between issues in a periodic manner, it is shown that the individuals' self-appraisals admit a periodic solution. Second, if the topology changes arbitrarily, under the assumption that each relative interaction matrix is doubly stochastic and irreducible, the individuals' self-appraisals asymptotically converge to a unique non-trivial equilibrium., Comment: This is the extended version of the paper accepted into 20th IFAC World Congress. It contains proofs for the periodic system, and includes arbitrarily issue-varying relative interaction matrices which are all doubly stochastic

According to the DeGroot-Friedkin model of a social network, an individual's social power evolves as the network discusses individual opinions over a sequence of issues. Under mild assumptions on the connectivity of the network, the social power of every individual converges to a constant strictly positive value as the number of issues discussed increases. If the network has a special topology, termed "star topology", then all social power accumulates with the individual at the centre of the star. This paper studies the strategic introduction of new individuals and/or interpersonal relationships into a social network with star topology to reduce the social power of the centre individual. In fact, several strategies are proposed. For each strategy, we derive necessary and sufficient conditions on the strength of the new interpersonal relationships, based on local information, which ensures that the centre individual no longer has the greatest social power within the social network. Interpretations of these conditions show that the strategies are remarkably intuitive and that certain strategies are favourable compared to others, all of which is sociologically expected., Comment: Extended version of CDC2017 submission, with proofs included

By the distributed averaging problem is meant the problem of computing the average value of a set of numbers possessed by the agents in a distributed network using only communication between neighboring agents. Gossiping is a well-known approach to the problem which seeks to iteratively arrive at a solution by allowing each agent to interchange information with at most one neighbor at each iterative step. Crafting a gossiping protocol which accomplishes this is challenging because gossiping is an inherently collaborative process which can lead to deadlocks unless careful precautions are taken to ensure that it does not. Many gossiping protocols are request-based which means simply that a gossip between two agents will occur whenever one of the two agents accepts a request to gossip placed by the other. In this paper, we present three deterministic request-based protocols. We show by example that the first can deadlock. The second is guaranteed to avoid deadlocks and requires fewer transmissions per iteration than standard broadcast-based distributed averaging protocols by exploiting the idea of local ordering together with the notion of an agent's neighbor queue; the protocol requires the simplest queue updates, which provides an in-depth understanding of how local ordering and queue updates avoid deadlocks. It is shown that a third protocol which uses a slightly more complicated queue update rule can lead to significantly faster convergence; a worst case bound on convergence rate is provided.

Inspired by the concept of network algebraic connectivity, we adopt an extended notion named rigidity preservation index to characterize the rigidity property for a formation framework. A gradient based controller is proposed to ensure the rigidity preservation of multi-robot networks in an unknown environment, while the rigidity metric can be maximized over time during robots' motions. In order to implement the controller in a distributed manner, a distributed inverse power iteration algorithm is developed which allows each robot to estimate the global rigidity index information. Simulation results are provided to demonstrate the effectiveness of the estimation and control scheme., Comment: 8 pages, 3 figures

We study a new variant of consensus problems, termed `local average consensus', in networks of agents. We consider the task of using sensor networks to perform distributed measurement of a parameter which has both spatial (in this paper 1D) and temporal variations. Our idea is to maintain potentially useful local information regarding spatial variation, as contrasted with reaching a single, global consensus, as well as to mitigate the effect of measurement errors. We employ two schemes for computation of local average consensus: exponential weighting and uniform finite window. In both schemes, we design local average consensus algorithms to address first the case where the measured parameter has spatial variation but is constant in time, and then the case where the measured parameter has both spatial and temporal variations. Our designed algorithms are distributed, in that information is exchanged only among neighbors. Moreover, we analyze both spatial and temporal frequency responses and noise propagation associated with the algorithms. The tradeoffs of using local consensus, as compared to standard global consensus, include higher memory requirement and degraded noise performance. Arbitrary updating weights and random spacing between sensors are analyzed in the proposed algorithms.

A fundamental control problem for autonomous vehicle formations is formation shape control, in which the agents must maintain a prescribed formation shape using only information measured or communicated from neighboring agents. While a large and growing literature has recently emerged on distance-based formation shape control, global stability properties remain a significant open problem. Even in four-agent formations, the basic question of whether or not there can exist locally stable incorrect equilibrium shapes remains open. This paper shows how this question can be answered for any size formation in principle using semidefinite programming techniques for semialgebraic problems, involving solutions sets of polynomial equations, inequations, and inequalities., Comment: 6 pages; to appear in the 2013 IEEE Multiconference on Systems and Control

In this paper, we study performance limits of sensor localization from a novel perspective. Specifically, we consider the Cramer-Rao Lower Bound (CRLB) in single-hop sensor localization using measurements from received signal strength (RSS), time of arrival (TOA) and bearing, respectively, but differently from the existing work, we statistically analyze the trace of the associated CRLB matrix (i.e. as a scalar metric for performance limits of sensor localization) by assuming anchor locations are random. By the Central Limit Theorems for $U$-statistics, we show that as the number of the anchors increases, this scalar metric is asymptotically normal in the RSS/bearing case, and converges to a random variable which is an affine transformation of a chi-square random variable of degree 2 in the TOA case. Moreover, we provide formulas quantitatively describing the relationship among the mean and standard deviation of the scalar metric, the number of the anchors, the parameters of communication channels, the noise statistics in measurements and the spatial distribution of the anchors. These formulas, though asymptotic in the number of the anchors, in many cases turn out to be remarkably accurate in predicting performance limits, even if the number is small. Simulations are carried out to confirm our results.

This paper considers a novel problem of how to choose an appropriate geometry for a group of agents with only shape constraints but with a flexible scale. Instead of assigning the formation system with a specific geometry, here the only requirement on the desired geometry is a shape without any location, rotation and, most importantly, scale constraints. Optimal rigid transformation between two different geometries is discussed with especial focus on the scaling operation, and the cooperative performance of the system is evaluated by what we call the geometries degrees of similarity (DOS) with respect to the desired shape during the entire convergence process. The design of the scale when measuring the DOS is discussed from constant value and time-varying function perspectives respectively. Fixed structured nonlinear control laws that are functions on the scale are developed to guarantee the exponential convergence of the system to the assigned shape. Our research is originated from a three-agent formation system and is further extended to multiple (n > 3) agents by defining a triangular complement graph. Simulations demonstrate that formation system with the time-varying scale function outperforms the one with an arbitrary constant scale, and the relationship between underlying topology and the system performance is further discussed based on the simulation observations. Moveover, the control scheme is applied to bearing-only sensor-target localization to show its application potentials., Comment: Submitted

In this paper we study the optimal formation control of multiple agents whose interaction parameters are adjusted upon a cost function consisting of both the control energy and the geometrical performance. By optimizing the interaction parameters and by the linear quadratic regulation(LQR) controllers, the upper bound of the cost function is minimized. For systems with homogeneous agents interconnected over sparse graphs, distributed controllers are proposed that inherit the same underlying graph as the one among agents. For the more general case, a relaxed optimization problem is considered so as to eliminate the nonlinear constraints. Using the subgradient method, interaction parameters among agents are optimized under the constraint of a sparse graph, and the optimum of the cost function is a better result than the one when agents interacted only through the control channel. Numerical examples are provided to validate the effectiveness of the method and to illustrate the geometrical performance of the system., Comment: Submitted

In complex electromagnetic environments it can often be difficult to determine whether signals received by an antenna array emanated from the same source. The failure to appropriately assign signal reception events to the correct emission event makes accurate localisation of the signal source impossible. In this paper we show that as the received signal events must lie on the light-cone of the emission event the Cayley-Menger determinate calculated from using the light-cone geodesic distances between received signals must be zero. This result enables us to construct an algorithm for sorting received signals into groups corresponding to the same far-field emission., Comment: 9 pages, 1 figure. Draft form. Final version published in Applied Physics Letters

This paper treats the problem of the merging of formations, where the underlying model of a formation is graphical. We first analyze the rigidity and persistence of meta-formations, which are formations obtained by connecting several rigid or persistent formations. Persistence is a generalization to directed graphs of the undirected notion of rigidity. In the context of moving autonomous agent formations, persistence characterizes the efficacy of a directed structure of unilateral distance constraints seeking to preserve a formation shape. We derive then, for agents evolving in a two- or three-dimensional space, the conditions under which a set of persistent formations can be merged into a persistent meta-formation, and give the minimal number of interconnections needed for such a merging. We also give conditions for a meta-formation obtained by merging several persistent formations to be persistent., Comment: 1 zip file containing 1 .tex files, and 39 .eps files. The paper (including the appendix) contains 13 Figures

In this paper, we study the construction and transformation of two-dimensional persistent graphs. Persistence is a generalization to directed graphs of the undirected notion of rigidity. In the context of moving autonomous agent formations, persistence characterizes the efficacy of a directed structure of unilateral distances constraints seeking to preserve a formation shape. Analogously to the powerful results about Henneberg sequences in minimal rigidity theory, we propose different types of directed graph operations allowing one to sequentially build any minimally persistent graph (i.e. persistent graph with a minimal number of edges for a given number of vertices), each intermediate graph being also minimally persistent. We also consider the more generic problem of obtaining one minimally persistent graph from another, which corresponds to the on-line reorganization of an autonomous agent formation. We prove that we can obtain any minimally persistent formation from any other one by a sequence of elementary local operations such that minimal persistence is preserved throughout the reorganization process., Comment: 26 pages, 31 .eps files for the figures