Search

We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions -- including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community. Project page: http://ego-exo4d-data.org/, Comment: Expanded manuscript (compared to arxiv v1 from Nov 2023 and CVPR 2024 paper from June 2024) for more comprehensive dataset and benchmark presentation, plus new results on v2 data release

Recently, several new solar radio telescopes have been put into operation and provided spectral-imaging observations with much higher resolutions in decimeter (dm) and centimeter (cm) wavelengths. These telescopes include the Mingantu Spectral Radioheliograph (MUSER, at frequencies of 0.4 - 15 GHz), the Expanded Owens Valley Solar Array (EOVSA, at frequencies of 1 - 18 GHz), and the Siberian Radio Heliograph (SRH, at frequencies of 3 - 24 GHz). These observations offer unprecedented opportunities to study solar physics and space weather, especially to diagnose the coronal magnetic fields, reveal the basic nature of solar eruptions and the related non-thermal energy release, particle accelerations and propagation, and the related emission mechanisms. These results might be the important input to the space weather modeling for predicting the occurrence of disastrous powerful space weather events. In order to provide meaningful reference for other solar physicists and space weather researchers, this paper mainly focus on discussing the potential scientific problems of solar radio spectral-imaging observations in dm-cm wavelengths and its possible applications in the field of space weather. These results will provide a helpful reference for colleagues to make full use of the latest and future observation data obtained from the above solar radio telescopes., Comment: 10 pages, 7 figures, accepted by Advance in Space Research, 2022

In this paper, we explore the challenges associated with biomarker identification for diagnosis purpose in biomedical experiments, and propose a novel approach to handle the above challenging scenario via the generalization of the Dantzig selector. To improve the efficiency of the regularization method, we introduce a transformation from an inherent nonlinear programming due to its nonlinear link function into a linear programming framework. We illustrate the use of of our method on an experiment with binary response, showing superior performance on biomarker identification studies when compared to their conventional analysis. Our proposed method does not merely serve as a variable/biomarker selection tool, its ranking of variable importance provides valuable reference information for practitioners to reach informed decisions regarding the prioritization of factors for further investigations.

Collection of annotated dialogs for training task-oriented dialog systems have been one of the key bottlenecks in improving current models. While dialog response generation has been widely studied on the agent side, it is not evident if similar generative models can be used to generate a large variety of, and often unexpected, user inputs that real dialog systems encounter in practice. Existing data augmentation techniques such as paraphrase generation do not take the dialog context into consideration. In this paper, we develop a novel dialog augmentation model that generates a user turn, conditioning on full dialog context. Additionally, with a new prompt design for language model, and output re-ranking, the dialogs generated from our model can be directly used to train downstream dialog systems. On common benchmark datasets MultiWoZ and SGD, we show that our dialog augmentation model generates high quality dialogs and improves dialog success rate by as much as $8\%$ over baseline., Comment: ECML-PKDD 2023 Workshop on Challenges and Opportunities of Large Language Models in Real-World Machine Learning Applications (COLLM)

We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. Due to the high dimensionality of humanoids and the inherent difficulties in reinforcement learning, prior methods have focused on learning skill embeddings for a narrow range of movement styles (e.g. locomotion, game characters) from specialized motion datasets. This limited scope hampers their applicability in complex tasks. We close this gap by significantly increasing the coverage of our motion representation space. To achieve this, we first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator. This is achieved by using an encoder-decoder structure with a variational information bottleneck. Additionally, we jointly learn a prior conditioned on proprioception (humanoid's own pose and velocities) to improve model expressiveness and sampling efficiency for downstream tasks. By sampling from the prior, we can generate long, stable, and diverse human motions. Using this latent space for hierarchical RL, we show that our policies solve tasks using human-like behavior. We demonstrate the effectiveness of our motion representation by solving generative tasks (e.g. strike, terrain traversal) and motion tracking using VR controllers., Comment: ICLR 2024 Spotlight. Project page: https://zhengyiluo.github.io/PULSE/

Machine learning plays an important and growing role in molecular simulation. The newest version of the OpenMM molecular dynamics toolkit introduces new features to support the use of machine learning potentials. Arbitrary PyTorch models can be added to a simulation and used to compute forces and energy. A higher-level interface allows users to easily model their molecules of interest with general purpose, pretrained potential functions. A collection of optimized CUDA kernels and custom PyTorch operations greatly improves the speed of simulations. We demonstrate these features on simulations of cyclin-dependent kinase 8 (CDK8) and the green fluorescent protein (GFP) chromophore in water. Taken together, these features make it practical to use machine learning to improve the accuracy of simulations at only a modest increase in cost., Comment: 15 pages, 4 figures

Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging. To help address this, we develop two modes of evaluation for natural language explanations that claim individual neurons represent a concept in a text input. In the observational mode, we evaluate claims that a neuron $a$ activates on all and only input strings that refer to a concept picked out by the proposed explanation $E$. In the intervention mode, we construe $E$ as a claim that the neuron $a$ is a causal mediator of the concept denoted by $E$. We apply our framework to the GPT-4-generated explanations of GPT-2 XL neurons of Bills et al. (2023) and show that even the most confident explanations have high error rates and little to no causal efficacy. We close the paper by critically assessing whether natural language is a good choice for explanations and whether neurons are the best level of analysis.

Recently, there has been significant progress in the development of large models. Following the success of ChatGPT, numerous language models have been introduced, demonstrating remarkable performance. Similar advancements have also been observed in image generation models, such as Google's Imagen model, OpenAI's DALL-E 2, and stable diffusion models, which have exhibited impressive capabilities in generating images. However, similar to large language models, these models still encounter unresolved challenges. Fortunately, the availability of open-source stable diffusion models and their underlying mathematical principles has enabled the academic community to extensively analyze the performance of current image generation models and make improvements based on this stable diffusion framework. This survey aims to examine the existing issues and the current solutions pertaining to image generation models.

Cloth manipulation is common in domestic and service tasks, and most studies use fixed-base manipulators to manipulate objects whose sizes are relatively small with respect to the manipulators' workspace, such as towels, shirts, and rags. In contrast, manipulation of large-scale cloth, such as bed making and tablecloth spreading, poses additional challenges of reachability and manipulation control. To address them, this paper presents a novel framework to spread large-scale cloth, with a single-arm mobile manipulator that can solve the reachability issue, for an initial feasibility study. On the manipulation control side, without modeling highly deformable cloth, a vision-based manipulation control scheme is applied and based on an online-update Jacobian matrix mapping from selected feature points to the end-effector motion. To coordinate the control of the manipulator and mobile platform, Behavior Trees (BTs) are used because of their modularity. Finally, experiments are conducted, including validation of the model-free manipulation control for cloth spreading in different conditions and the large-scale cloth spreading framework. The experimental results demonstrate the large-scale cloth spreading task feasibility with a single-arm mobile manipulator and the model-free deformation controller., Comment: 6 pages, 6 figures, submit to CASE2023

Reducing noises and enhancing signal-to-noise ratios (SNRs) have become critical for designing third-generation gravitational-wave (GW) detectors with a GW strain of less than $10^{-23}$/$\rm \sqrt{Hz}$. In this paper, we propose a potential third-generation GW detector based on autocorrelative weak-value amplification (AWVA) for GW detection with a strain of $h_g =$ $4 \times 10^{-25}$/$\rm \sqrt{Hz}$. In our scheme, a GW event induces a phase difference $\Delta \phi$ by passing through an 11-bounce delay line, 10-km arm-length, zero-area Sagnac interferometer illuminated with a 1064-nm laser. Subsequently, $\Delta \phi$ is amplified as the parameter of post-selection by choosing the appropriate pre-selected state and coupling strength in AWVA. In particular, we theoretically investigate the AWVA measurements for GW detection within the frequency band of 200 Hz $\leq$ $f_g$ $\leq$ 800 Hz, considering Gaussian noises with negative-decibel SNRs. The peak response of the AWVA sensitivity $\kappa(f_g)$ occurs at frequency $f_{g, max}$ = 500 Hz, which falls within the frequency band of interest of the current third-generation GW detectors. Our simulation results indicate that AWVA can demonstrate a measurable sensitivity of $\Theta(f_g)$ within the frequency band of interest. Moreover, the robustness of WVA shows promising potential in mitigating the effects of Gaussian noises., Comment: 21 pages, 6 figures