Search

Event cameras provide a natural and data efficient representation of visual information, motivating novel computational strategies towards extracting visual information. Inspired by the biological vision system, we propose a behavior driven approach for object-wise distance estimation from event camera data. This behavior-driven method mimics how biological systems, like the human eye, stabilize their view based on object distance: distant objects require minimal compensatory rotation to stay in focus, while nearby objects demand greater adjustments to maintain alignment. This adaptive strategy leverages natural stabilization behaviors to estimate relative distances effectively. Unlike traditional vision algorithms that estimate depth across the entire image, our approach targets local depth estimation within a specific region of interest. By aligning events within a small region, we estimate the angular velocity required to stabilize the image motion. We demonstrate that, under certain assumptions, the compensatory rotational flow is inversely proportional to the object's distance. The proposed approach achieves new state-of-the-art accuracy in distance estimation - a performance gain of 16% on EVIMO2. EVIMO2 event sequences comprise complex camera motion and substantial variance in depth of static real world scenes.

Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of 81.01% on the RAVDESS dataset and 79.34% on the MEAD dataset. We make our code publicly available., Comment: GCPR 2024

Easily accessible sensors, like drones with diverse onboard sensors, have greatly expanded studying animal behavior in natural environments. Yet, analyzing vast, unlabeled video data, often spanning hours, remains a challenge for machine learning, especially in computer vision. Existing approaches often analyze only a few frames. Our focus is on long-term animal behavior analysis. To address this challenge, we utilize classical probabilistic methods for state estimation, such as particle filtering. By incorporating recent advancements in semantic object segmentation, we enable continuous tracking of rapidly evolving object formations, even in scenarios with limited data availability. Particle filters offer a provably optimal algorithmic structure for recursively adding new incoming information. We propose a novel approach for tracking schools of fish in the open ocean from drone videos. Our framework not only performs classical object tracking in 2D, instead it tracks the position and spatial expansion of the fish school in world coordinates by fusing video data and the drone's on board sensor information (GPS and IMU). The presented framework for the first time allows researchers to study collective behavior of fish schools in its natural social and environmental context in a non-invasive and scalable way., Comment: CVPRW: Workshop paper appearing in CV4Animals

Human communication is multi-modal; e.g., face-to-face interaction involves auditory signals (speech) and visual signals (face movements and hand gestures). Hence, it is essential to exploit multiple modalities when designing machine learning-based facial expression recognition systems. In addition, given the ever-growing quantities of video data that capture human facial expressions, such systems should utilize raw unlabeled videos without requiring expensive annotations. Therefore, in this work, we employ a multitask multi-modal self-supervised learning method for facial expression recognition from in-the-wild video data. Our model combines three self-supervised objective functions: First, a multi-modal contrastive loss, that pulls diverse data modalities of the same video together in the representation space. Second, a multi-modal clustering loss that preserves the semantic structure of input data in the representation space. Finally, a multi-modal data reconstruction loss. We conduct a comprehensive study on this multimodal multi-task self-supervised learning method on three facial expression recognition benchmarks. To that end, we examine the performance of learning through different combinations of self-supervised tasks on the facial expression recognition downstream task. Our model ConCluGen outperforms several multi-modal self-supervised and fully supervised baselines on the CMU-MOSEI dataset. Our results generally show that multi-modal self-supervision tasks offer large performance gains for challenging tasks such as facial expression recognition, while also reducing the amount of manual annotations required. We release our pre-trained models as well as source code publicly, Comment: The paper will appear in the CVPR 2024 workshops proceedings

Trajectory prediction is an essential task for successful human robot interaction, such as in autonomous driving. In this work, we address the problem of predicting future pedestrian trajectories in a first person view setting with a moving camera. To that end, we propose a novel action-based contrastive learning loss, that utilizes pedestrian action information to improve the learned trajectory embeddings. The fundamental idea behind this new loss is that trajectories of pedestrians performing the same action should be closer to each other in the feature space than the trajectories of pedestrians with significantly different actions. In other words, we argue that behavioral information about pedestrian action influences their future trajectory. Furthermore, we introduce a novel sampling strategy for trajectories that is able to effectively increase negative and positive contrastive samples. Additional synthetic trajectory samples are generated using a trained Conditional Variational Autoencoder (CVAE), which is at the core of several models developed for trajectory prediction. Results show that our proposed contrastive framework employs contextual information about pedestrian behavior, i.e. action, effectively, and it learns a better trajectory representation. Thus, integrating the proposed contrastive framework within a trajectory prediction model improves its results and outperforms state-of-the-art methods on three trajectory prediction benchmarks [31, 32, 26]., Comment: This paper will appear in the proceedings of The European Conference on Computer Vision (ECCV 2022)

Both a good understanding of geometrical concepts and a broad familiarity with objects lead to our excellent perception of moving objects. The human ability to detect and segment moving objects works in the presence of multiple objects, complex background geometry, motion of the observer and even camouflage. How humans perceive moving objects so reliably is a longstanding research question in computer vision and borrows findings from related areas such as psychology, cognitive science and physics. One approach to the problem is to teach a deep network to model all of these effects. This contrasts with the strategy used by human vision, where cognitive processes and body design are tightly coupled and each is responsible for certain aspects of correctly identifying moving objects. Similarly from the computer vision perspective, there is evidence that classical, geometry-based techniques are better suited to the "motion-based" parts of the problem, while deep networks are more suitable for modeling appearance. In this work, we argue that the coupling of camera rotation and camera translation can create complex motion fields that are difficult for a deep network to untangle directly. We present a novel probabilistic model to estimate the camera's rotation given the motion field. We then rectify the flow field to obtain a rotation-compensated motion field for subsequent segmentation. This strategy of first estimating camera motion, and then allowing a network to learn the remaining parts of the problem, yields improved results on the widely used DAVIS benchmark as well as the recently published motion segmentation data set MoCA (Moving Camouflaged Animals).

Event cameras, inspired by biological vision systems, provide a natural and data efficient representation of visual information. Visual information is acquired in the form of events that are triggered by local brightness changes. Each pixel location of the camera's sensor records events asynchronously and independently with very high temporal resolution. However, because most brightness changes are triggered by relative motion of the camera and the scene, the events recorded at a single sensor location seldom correspond to the same world point. To extract meaningful information from event cameras, it is helpful to register events that were triggered by the same underlying world point. In this work we propose a new model of event data that captures its natural spatio-temporal structure. We start by developing a model for aligned event data. That is, we develop a model for the data as though it has been perfectly registered already. In particular, we model the aligned data as a spatio-temporal Poisson point process. Based on this model, we develop a maximum likelihood approach to registering events that are not yet aligned. That is, we find transformations of the observed events that make them as likely as possible under our model. In particular we extract the camera rotation that leads to the best event alignment. We show new state of the art accuracy for rotational velocity estimation on the DAVIS 240C dataset. In addition, our method is also faster and has lower computational complexity than several competing methods.

Visually exploring the world around us is not a passive process. Instead, we actively explore the world and acquire visual information over time. Here, we present a new model for simulating human eye-movement behavior in dynamic real-world scenes. We model this active scene exploration as a sequential decision making process. We adapt the popular drift-diffusion model (DDM) for perceptual decision making and extend it towards multiple options, defined by objects present in the scene. For each possible choice, the model integrates evidence over time and a decision (saccadic eye movement) is triggered as soon as evidence crosses a decision threshold. Drawing this explicit connection between decision making and object-based scene perception is highly relevant in the context of active viewing, where decisions are made continuously while interacting with an external environment. We validate our model with a carefully designed ablation study and explore influences of our model parameters. A comparison on the VidCom dataset supports the plausibility of the proposed approach., Comment: Accepted for presentation at EPIC@CVPR2021 workshop, 4 pages, 2 figures

Motion segmentation is currently an active area of research in computer Vision. The task of comparing different methods of motion segmentation is complicated by the fact that researchers may use subtly different definitions of the problem. Questions such as "Which objects are moving?", "What is background?", and "How can we use motion of the camera to segment objects, whether they are static or moving?" are clearly related to each other, but lead to different algorithms, and imply different versions of the ground truth. This report has two goals. The first is to offer a precise definition of motion segmentation so that the intent of an algorithm is as well-defined as possible. The second is to report on new versions of three previously existing data sets that are compatible with this definition. We hope that this more detailed definition, and the three data sets that go with it, will allow more meaningful comparisons of certain motion segmentation methods.

The human ability to detect and segment moving objects works in the presence of multiple objects, complex background geometry, motion of the observer, and even camouflage. In addition to all of this, the ability to detect motion is nearly instantaneous. While there has been much recent progress in motion segmentation, it still appears we are far from human capabilities. In this work, we derive from first principles a new likelihood function for assessing the probability of an optical flow vector given the 3D motion direction of an object. This likelihood uses a novel combination of the angle and magnitude of the optical flow to maximize the information about the true motions of objects. Using this new likelihood and several innovations in initialization, we develop a motion segmentation algorithm that beats current state-of-the-art methods by a large margin. We compare to five state-of-the-art methods on two established benchmarks, and a third new data set of camouflaged animals, which we introduce to push motion segmentation to the next level.

Precise measurements of the cosmic microwave background (CMB) are crucial in cosmology, because any proposed model of the universe must account for the features of this radiation. Of all CMB measurements that the scientific community has not yet been able to perform, the CMB B-mode polarization is probably the most challenging from the instrumental point of view. The signature of primordial gravitational waves, which give rise to a B-type polarization, is one of the goals in cosmology today and amongst the first objectives in the field. For this purpose, high-performance low-temperature bolometric cameras, made of thousands of pixels, are currently being developed by many groups, which will improve the sensitivity to B-mode CMB polarization by one or two orders of magnitude compared to the Planck satellite HFI detectors. We present here a new bolometer structure that is able to increase the pixel sensitivities and to simplify the fabrication procedure. This innovative device replaces delicate membrane-based structures and eliminates the mediation of phonons: the incoming energy is directly captured and measured in the electron bath of an appropriate sensor and the thermal decoupling is achieved via the intrinsic electron-phonon decoupling of the sensor at very low temperature. Reported results come from a 204-pixel array of Nb$_{x}$Si$_{1-x}$ transition edge sensors with a meander structure fabricated on a 2-inch silicon wafer using electron-beam co-evaporation and a cleanroom lithography process. To validate the application of this device to CMB measurements, we have performed an optical calibration of our sample in the focal plane of a dilution cryostat test bench. We have demonstrated a light absorption close to 20% and an NEP of about 7$\times10^{-16}$ W/$\sqrt{Hz}$, which is highly encouraging given the scope for improvement in this type of detectors., Comment: 6 pages, 10 figures. arXiv admin note: text overlap with arXiv:1005.0555 by other authors

When submitted to gentle mechanical taps a granular packing slowly compacts until it reaches a stationary state that depends on the tap characteristics. The properties of such stationary states are experimentally investigated. The influence of the initial state, taps properties and tapping protocol are studied. The compactivity of the packings is determinated. Our results strongly support the idea that the stationary states are genuine thermodynamic states., Comment: to be published in EPJE. The original publication will be available at www.europhysj.org

This letter reports experimental and numerical results on particle dynamics in an out-of-equilibrium granular medium. We observed two distinct types of grain motion: the well known cage motion, during which a grain is always surrounded by the same neighbors, and low probability "jumps", during which a grain moves significantly more relative to the others. These observations are similar to the results obtained for other out-of-equilibrium systems (glasses, colloidal systems, etc.). Although such jumps are extremely rare, by inhibiting them in numerical simulations we demonstrate that they play a significant role in the relaxation of out-of-equilibrium systems, Comment: 4 pages, accepted for publication in Physical Review Letters

We report on experiments to measure the temporal and spatial evolution of packing arrangements of anisotropic and weakly confined granular material, using high-resolution $\gamma$-ray adsorption. In these experiments, the particle configurations start from an initially disordered, low-packing-fraction state and under vertical solicitations evolve to a dense state. We find that the packing fraction evolution is slowed by the grain anisotropy but, as for spherically shaped grains, can be well fitted by a stretched exponential. For a given type of grains, the characteristic times of relaxation and of convection are found to be of the same order of magnitude. On the contrary compaction mechanisms in the media strongly depend on the grain anisotropy., Comment: to appear in the european physical journal E (EPJE)

We report the experimental observation of novel vortex patterns in a vertically tapped granular media. Depending on the tapping acceleration two behaviors are observed. For high acceleration a convection vortex appears in the whole media whereas for low acceleration two unstable vortices appear in the upper part of the media and slowly compact the lower part. We explain the formation of the vortices and relate them to granular convection. Our results demonstrate the importance of compression waves propagation on granular compaction.

Several acquisitions of X-ray microtomography have been performed on a beads packing while it compacts under vertical vibrations. An image analysis allows to study the evolution of the packing structure during its progressive densification. In particular, the volume distribution of the pores reveals a large tail, compatible to an exponential law, which slowly reduces as the system gets more compact. This is quite consistent, for large pores, with the free volume theory. These results are also in very good agreement with those obtained by a previous numerical model of granular compaction., Comment: 4 pages, 4 figures. Latex (revtex4). to be published in Phys. Rev. E

We report new experimental results on granular compaction under consecutive vertical taps. The evolution of the mean volume fraction and of the mean potential energy of a granular packing presents a slow densification until a final steady-state, and is reminiscent to usual relaxation in glasses via a stretched exponential law. The intensity of the taps seems to rule the characteristic time of the relaxation according to an Arrhenius's type relation >. Finally, the analysis of the vertical volume fraction profile reveals an almost homogeneous densification in the packing., Comment: 7 pages, 4 figures, to appear in Europhysics Letters

A simple numerical model is used to simulate the effect of vertical taps on a packing of monodisperse hard spheres. Our results are in agreement with an experimantal work done in Chicago and with other previous models, especially concerning the dynamics of the compaction, the influence of the excitation strength on the compaction efficiency, and some ageing effects. The principal asset of the model is that it allows a local analysis of the packings. Vertical and transverse density profiles are used as well as size and volume distributions of the pores. An interesting result concerns the appearance of a vertical gradient in the density profiles during compaction. Furthermore, the volume distribution of the pores suggests that the smallest pores, ranging in size between a tetrahedral and an octahedral site, are not strongly affected by the tapping process, in contrast to the largest pores which are more sensitive to the compaction of the packing., Comment: 8 pages, 15 figures (eps), to be published in Phys. Rev. E. Some corrections have been made, especially in paragraph IV B

We present several numerical results on granular mixtures. In particular, we examine the efficiency of diffusion as a mixing mechanism in these systems. The collisions are inelastic and to compensate the energy loss, we thermalize the grains by adding a random force. Starting with a segregated system, we show that uniform agitation (heating) leads to a uniform mixture of grains of different sizes. We define a characteristic mixing time, $\tau_{mix}$, and study theoretically and numerically its dependence on other parameters like the density. We examine a model for bidisperse systems for which we can calculate some physical quantities. We also examine the effect of a temperature gradient and demonstrate the appearance of an expected segregation., Comment: 15 eps figures, included

We analyze the influence of boundary conditions on numerical simulations of the diffusive properties of a two dimensional granular gas. We show in particular that periodic boundary conditions introduce unphysical correlations in time which cause the coefficient of diffusion to be strongly dependent on the system size. On the other hand, in large enough systems with hard walls at the boundaries, diffusion is found to be independent of the system size. We compare the results obtained in this case with Langevin theory for an elastic gas. Good agreement is found. We then calculate the relaxation time and the influence of the mass for a particle of radius $R_s$ in a sea of particles of radius $R_b$. As granular gases are dissipative, we also study the influence of an external random force on the diffusion process in a forced dissipative system. In particular, we analyze differences in the mean square velocity and displacement between the elastic and inelastic cases., Comment: 15 figures eps figures, included

We report an experimental, numerical and theoretical study of the motion of a ball on a rough inclined surface. The control parameters are $D$, the diameter of the ball, $\theta$, the inclination angle of the rough surface and $E_{ki}$, the initial kinetic energy. When the angle of inclination is larger than some critical value, $\theta>\theta_{T}$, the ball moves at a constant average velocity which is independent of the initial conditions. For an angle $\theta < \theta_{T}$, the balls are trapped after moving a certain distance. The dependence of the travelled distances on $E_{ki}$, $D$ and $\theta$. is analysed. The existence of two kinds of mechanisms of dissipation is thus brought to light. We find that for high initial velocities the friction force is constant. As the velocity decreases below a certain threshold the friction becomes viscous., Comment: 8 pages RevTeX, 12 Postscript figures