Search

Given a multi-view video, which viewpoint is most informative for a human observer? Existing methods rely on heuristics or expensive "best-view" supervision to answer this question, limiting their applicability. We propose a weakly supervised approach that leverages language accompanying an instructional multi-view video as a means to recover its most informative viewpoint(s). Our key hypothesis is that the more accurately an individual view can predict a view-agnostic text summary, the more informative it is. To put this into action, we propose a framework that uses the relative accuracy of view-dependent caption predictions as a proxy for best view pseudo-labels. Then, those pseudo-labels are used to train a view selector, together with an auxiliary camera pose predictor that enhances view-sensitivity. During inference, our model takes as input only a multi-view video--no language or camera poses--and returns the best viewpoint to watch at each timestep. On two challenging datasets comprised of diverse multi-camera setups and how-to activities, our model consistently outperforms state-of-the-art baselines, both with quantitative metrics and human evaluation.

Many climate processes are characterized using large systems of nonlinear differential equations; this, along with the immense amount of data required to parameterize complex interactions, means that Earth-System Model (ESM) simulations may take weeks to run on large clusters. Uncertainty quantification may require thousands of runs, making ESM simulations impractical for preliminary assessment. Alternatives may include simplifying the processes in the model, but recent efforts have focused on using machine learning to complement these models or even act as full surrogates. \textit{We leverage machine learning, specifically fully-connected neural networks (FCNNs) and graph convolutional neural networks (GCNNs), to enable rapid simulation and uncertainty quantification in order to inform more extensive ESM simulations.} Our surrogate simulated 80 years in approximately 310 seconds on a single A100 GPU, compared to weeks for the ESM model while having mean temperature errors below $0.1^{\circ}C$ and maximum errors below $2^{\circ}C$., Comment: 10 pages, 8 figures