Search

Stereoscopic video conferencing is still challenging due to the need to compress stereo RGB-D video in real-time. Though hardware implementations of standard video codecs such as H.264 / AVC and HEVC are widely available, they are not designed for stereoscopic videos and suffer from reduced quality and performance. Specific multiview or 3D extensions of these codecs are complex and lack efficient implementations. In this paper, we propose a new approach to upgrade a 2D video codec to support stereo RGB-D video compression, by wrapping it with a neural pre- and post-processor pair. The neural networks are end-to-end trained with an image codec proxy, and shown to work with a more sophisticated video codec. We also propose a geometry-aware loss function to improve rendering quality. We train the neural pre- and post-processors on a synthetic 4D people dataset, and evaluate it on both synthetic and real-captured stereo RGB-D videos. Experimental results show that the neural networks generalize well to unseen data and work out-of-box with various video codecs. Our approach saves about 30% bit-rate compared to a conventional video coding scheme and MV-HEVC at the same level of rendering quality from a novel view, without the need of a task-specific hardware upgrade., Comment: Accepted by CVPR 2024 Workshop (AIS: Vision, Graphics and AI for Streaming https://ai4streaming-workshop.github.io )

We propose sandwiching standard image and video codecs between pre- and post-processing neural networks. The networks are jointly trained through a differentiable codec proxy to minimize a given rate-distortion loss. This sandwich architecture not only improves the standard codec's performance on its intended content, it can effectively adapt the codec to other types of image/video content and to other distortion measures. Essentially, the sandwich learns to transmit ``neural code images'' that optimize overall rate-distortion performance even when the overall problem is well outside the scope of the codec's design. Through a variety of examples, we apply the sandwich architecture to sources with different numbers of channels, higher resolution, higher dynamic range, and perceptual distortion measures. The results demonstrate substantial improvements (up to 9 dB gains or up to 30\% bitrate reductions) compared to alternative adaptations. We derive VQ equivalents for the sandwich, establish optimality properties, and design differentiable codec proxies approximating current standard codecs. We further analyze model complexity, visual quality under perceptual metrics, as well as sandwich configurations that offer interesting potentials in image/video compression and streaming.

Pretraining is a popular and powerful paradigm in machine learning to pass information from one model to another. As an example, suppose one has a modest-sized dataset of images of cats and dogs, and plans to fit a deep neural network to classify them from the pixel features. With pretraining, we start with a neural network trained on a large corpus of images, consisting of not just cats and dogs but hundreds of other image types. Then we fix all of the network weights except for the top layer(s) (which makes the final classification) and train (or "fine tune") those weights on our dataset. This often results in dramatically better performance than the network trained solely on our smaller dataset. In this paper, we ask the question "Can pretraining help the lasso?". We develop a framework for the lasso in which a model is fit to a large dataset, and then fine-tuned using a smaller dataset. This latter dataset can be a subset of the original dataset, or it can be a dataset with a different but related outcome. This framework has a wide variety of applications, including stratified models, multinomial responses, multi-response models, conditional average treatment estimation and even gradient boosting. In the stratified model setting, the pretrained lasso pipeline estimates the coefficients common to all groups at the first stage, and then estimates the group-specific coefficients at the second "fine-tuning" stage. We show that under appropriate assumptions, the support recovery rate of the common coefficients is superior to that of the usual lasso trained only on individual groups. This separate identification of common and individual coefficients can also be useful for scientific understanding.

The physical properties of an object, such as mass, significantly affect how we manipulate it with our hands. Surprisingly, this aspect has so far been neglected in prior work on 3D motion synthesis. To improve the naturalness of the synthesized 3D hand object motions, this work proposes MACS the first MAss Conditioned 3D hand and object motion Synthesis approach. Our approach is based on cascaded diffusion models and generates interactions that plausibly adjust based on the object mass and interaction type. MACS also accepts a manually drawn 3D object trajectory as input and synthesizes the natural 3D hand motions conditioned by the object mass. This flexibility enables MACS to be used for various downstream applications, such as generating synthetic training data for ML tasks, fast animation of hands for graphics workflows, and generating character interactions for computer games. We show experimentally that a small-scale dataset is sufficient for MACS to reasonably generalize across interpolated and extrapolated object masses unseen during the training. Furthermore, MACS shows moderate generalization to unseen objects, thanks to the mass-conditioned contact labels generated by our surface contact synthesis model ConNet. Our comprehensive user study confirms that the synthesized 3D hand-object interactions are highly plausible and realistic.

Many modern datasets, such as those in ecology and geology, are composed of samples with spatial structure and dependence. With such data violating the usual independent and identically distributed (IID) assumption in machine learning and classical statistics, it is unclear a priori how one should measure the performance and generalization of models. Several authors have empirically investigated cross-validation (CV) methods in this setting, reaching mixed conclusions. We provide a class of unbiased estimation methods for general quadratic errors, correlated Gaussian response, and arbitrary prediction function $g$, for a noise-elevated version of the error. Our approach generalizes the coupled bootstrap (CB) from the normal means problem to general normal data, allowing correlation both within and between the training and test sets. CB relies on creating bootstrap samples that are intelligently decoupled, in the sense of being statistically independent. Specifically, the key to CB lies in generating two independent "views" of our data and using them as stand-ins for the usual independent training and test samples. Beginning with Mallows' $C_p$, we generalize the estimator to develop our generalized $C_p$ estimators (GC). We show at under only a moment condition on $g$, this noise-elevated error estimate converges smoothly to the noiseless error estimate. We show that when Stein's unbiased risk estimator (SURE) applies, GC converges to SURE as in the normal means problem. Further, we use these same tools to analyze CV and provide some theoretical analysis to help understand when CV will provide good estimates of error. Simulations align with our theoretical results, demonstrating the effectiveness of GC and illustrating the behavior of CV methods. Lastly, we apply our estimator to a model selection task on geothermal data in Nevada., Comment: 28 pages, 13 figures

We define a class of morphisms between \'etale groupoids and show that there is a functor from the category with these morphisms to the category of $C^*$-algebras. We show that all homomorphisms between Cartan pairs of $C^*$-algebras that preserve the Cartan structure arise from such morphisms between the underlying Weyl groupoids and twists, and attain an equivalence of categories between Cartan pairs with structure preserving homomorphisms and a their associated twists. We define analogous morphisms for Fell bundles over $C^*$-algebras and show these functorially induce ${}^*$-homomorphisms between the Fell bundle $C^*$-algebras. We also construct colimit groupoids and Fell bundles for inductive systems of such morphisms, and show that the functor to $C^*$-algebras preserves these colimits., Comment: 44 pages, comments welcome, version 2 adds references and remarks regarding the relationship of the paper to the third reference and changes some technical assumptions to fix problems associated with Lemma 4.14 in the previous version (and its consequences)

Inference for prediction errors is critical in time series forecasting pipelines. However, providing statistically meaningful uncertainty intervals for prediction errors remains relatively under-explored. Practitioners often resort to forward cross-validation (FCV) for obtaining point estimators and constructing confidence intervals based on the Central Limit Theorem (CLT). The naive version assumes independence, a condition that is usually invalid due to time correlation. These approaches lack statistical interpretations and theoretical justifications even under stationarity. This paper systematically investigates uncertainty intervals for prediction errors in time series forecasting. We first distinguish two key inferential targets: the stochastic test error over near future data points, and the expected test error as the expectation of the former. The stochastic test error is often more relevant in applications needing to quantify uncertainty over individual time series instances. To construct prediction intervals for the stochastic test error, we propose the quantile-based forward cross-validation (QFCV) method. Under an ergodicity assumption, QFCV intervals have asymptotically valid coverage and are shorter than marginal empirical quantiles. In addition, we also illustrate why naive CLT-based FCV intervals fail to provide valid uncertainty intervals, even with certain corrections. For non-stationary time series, we further provide rolling intervals by combining QFCV with adaptive conformal prediction to give time-average coverage guarantees. Overall, we advocate the use of QFCV procedures and demonstrate their coverage and efficiency through simulations and real data examples., Comment: 35 pages, 17 figures

We have conducted an extensive towed-magnetic-sled survey during the period 14-28 June, 2023, over the seafloor centered around the calculated path of the bolide CNEOS 2014-01-08 (IM1) about 85 km north of Manus Island, Papua New Guinea. We found about 700 spherules of diameter 0.05-1.3 millimeters in our samples, of which 57 were analyzed so far. The spherules were significantly concentrated along the expected meteor path. Mass spectrometry of 47 spherules near the high-yield regions along IM1's path reveals a distinct extra-solar abundance pattern for 5 of them, while background spherules have abundances consistent with a solar system origin. The unique spherules show an excess of Be, La and U, by up to three orders of magnitude relative to the solar system standard of CI chondrites. These "BeLaU"-type spherules, never seen before, also have very low refractory siderophile elements such as Re. Volatile elements, such as Mn, Zn, Pb, are depleted as expected from evaporation losses during a meteor's airburst. In addition, the mass-dependent variations in $^{57}$Fe/$^{54}$Fe and $^{56}$Fe/$^{54}$Fe are also consistent with evaporative loss of the light isotopes during the spherules' travel in the atmosphere. The "BeLaU" abundance pattern is not found in control regions outside of IM1's path and does not match commonly manufactured alloys or natural meteorites in the solar system. This evidence points towards an association of "BeLaU"-type spherules with IM1, supporting its interstellar origin independently of the high velocity and unusual material strength implied from the CNEOS data. We suggest that the "BeLaU" abundance pattern could have originated from a highly differentiated magma ocean of a planet with an iron core outside the solar system or from more exotic sources., Comment: Submitted for publication in a peer-reviewed journal

We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images. Unlike existing hand pose estimation methods, where one typically trains a deep network to regress hand model parameters from single RGB image, we consider a more challenging problem setting where we directly regress the absolute root poses of two-hands with extended forearm at high resolution from egocentric view. As existing datasets are either infeasible for egocentric viewpoints or lack background variations, we create a large-scale synthetic dataset with diverse scenarios and collect a real dataset from multi-calibrated camera setup to verify our proposed multi-view image feature fusion strategy. To make the reconstruction physically plausible, we propose two strategies: (i) a coarse-to-fine spectral graph convolution decoder to smoothen the meshes during upsampling and (ii) an optimisation-based refinement stage at inference to prevent self-penetrations. Through extensive quantitative and qualitative evaluations, we show that our framework is able to produce realistic two-hand reconstructions and demonstrate the generalisation of synthetic-trained models to real data, as well as real-time AR/VR applications., Comment: Accepted to ICCV 2023

Optical tweezers enable non-contact trapping of micro-scale objects using light. Despite their widespread use, it is currently not known how tightly it is possible to three-dimensionally trap micro-particles with a given photon budget. Reaching this elusive limit would enable maximally-stiff particle trapping for precision measurements on the nanoscale, and photon-efficient tweezing of light-sensitive objects. Here we solve this problem by customising a trapping light field to suit a specific particle, with the aim of simultaneously optimising trap stiffness in all three dimensions. Initially taking a theoretical approach, we develop an efficient multi-parameter optimisation routine to design bespoke optical traps for a wide range of micro-particles. We show that the confinement volume of micro-spheres held in these sculpted traps can be reduced by one-to-two orders-of-magnitude in comparison to a conventional optical tweezer of the same power. We go on to conduct proof-of-principle experiments, and use a wavefront shaping inspired strategy to suppress the Brownian fluctuations of optically trapped micro-spheres in every direction concurrently, thus demonstrating order-of-magnitude reductions in their confinement volumes. Our work paves the way towards the fundamental limits of optical control over the mesoscopic realm.

We propose sandwiched video compression -- a video compression system that wraps neural networks around a standard video codec. The sandwich framework consists of a neural pre- and post-processor with a standard video codec between them. The networks are trained jointly to optimize a rate-distortion loss function with the goal of significantly improving over the standard codec in various compression scenarios. End-to-end training in this setting requires a differentiable proxy for the standard video codec, which incorporates temporal processing with motion compensation, inter/intra mode decisions, and in-loop filtering. We propose differentiable approximations to key video codec components and demonstrate that, in addition to providing meaningful compression improvements over the standard codec, the neural codes of the sandwich lead to significantly better rate-distortion performance in two important scenarios.When transporting high-resolution video via low-resolution HEVC, the sandwich system obtains 6.5 dB improvements over standard HEVC. More importantly, using the well-known perceptual similarity metric, LPIPS, we observe 30% improvements in rate at the same quality over HEVC. Last but not least, we show that pre- and post-processors formed by very modestly-parameterized, light-weight networks can closely approximate these results., Comment: Published at the International Conference on Image Processing (ICIP), 2023

We define the local multiplier module of a Hilbert module in analogy to the local multiplier algebra for $C^*$-algebras. We use properties of the local multiplier module to lift non-closed actions on $C^*$-algebras by Hilbert bimodules to closed actions on local multiplier algebras, and descend known results on such closed actions down to their unclosed counterparts. We define aperiodic dynamical inclusions and characterise them as crossed products by inverse semigroup actions. We describe the slice structure for such inclusions and show that all slices arise as linear combinations of slices already present in the inverse semigroup action., Comment: 24 pages, results in this article also exist in the author's PhD thesis

We introduce a pivot for exact selective inference with randomization. Not only does our pivot lead to exact inference in Gaussian regression models, but it is also available in closed form. We reduce the problem of exact selective inference to a bivariate truncated Gaussian distribution. By doing so, we give up some power that is achieved with approximate maximum likelihood estimation in Panigrahi and Taylor (2022). Yet our pivot always produces narrower confidence intervals than a closely related data splitting procedure. We investigate the trade-off between power and exact selective inference on simulated datasets and an HIV drug resistance dataset., Comment: 48 pages, 8 Figures, 2 Tables

Through a series of federal initiatives and orders, the U.S. Government has been making a concerted effort to ensure American leadership in AI. These broad strategy documents have influenced organizations such as the United States Department of the Air Force (DAF). The DAF-MIT AI Accelerator is an initiative between the DAF and MIT to bridge the gap between AI researchers and DAF mission requirements. Several projects supported by the DAF-MIT AI Accelerator are developing public challenge problems that address numerous Federal AI research priorities. These challenges target priorities by making large, AI-ready datasets publicly available, incentivizing open-source solutions, and creating a demand signal for dual use technologies that can stimulate further research. In this article, we describe these public challenges being developed and how their application contributes to scientific advances.

We define essential commutative Cartan pairs of $C^*$-algebras generalising the definition of Renault and show that such pairs are given by essential twisted groupoid $C^*$-algebras as defined by Kwa\'sniewski and Meyer. We show that the underlying twisted groupoid is effective, and is unique up to isomorphism among twists over effective groupoids giving rise to the essential commutative Cartan pair. We also show that for twists over effective groupoids giving rise to such pairs, the automorphism group of the twist is isomorphic to the automorphism group of the induced essential Cartan pair via explicit constructions., Comment: 25 pages, v3 incorporates recommendations from an anonymous referee report. Also added Proposition 5.8, which shows that (twists over) effective groupoids give rise to masa inclusions

We prove that an inductive limit of aperiodic noncommutative Cartan inclusions is a noncommutative Cartan inclusion whenever the connecting maps are injective, preserve normalisers and entwine conditional expectations. We show that under the additional assumption that the inductive limit Cartan subalgebra is either essentially separable, essentially simple or essentially of Type I we get an aperiodic inclusion in the limit. Consequently, we subsume the case where the building block Cartan subalgebras are commutative and provide a proof of a theorem of Xin Li without passing to twisted \'etale groupoids., Comment: Final version, some minor changes. To appear in Proc. Amer. Math. Soc

Conditional selective inference requires an exact characterization of the selection event, which is often unavailable except for a few examples like the lasso. This work addresses this challenge by introducing a generic approach to estimate the selection event, facilitating feasible inference conditioned on the selection event. The method proceeds by repeatedly generating bootstrap data and running the selection algorithm on the new datasets. Using the outputs of the selection algorithm, we can estimate the selection probability as a function of certain summary statistics. This leads to an estimate of the distribution of the data conditioned on the selection event, which forms the basis for conditional selective inference. We provide a theoretical guarantee assuming both asymptotic normality of relevant statistics and accurate estimation of the selection probability. The applicability of the proposed method is demonstrated through a variety of problems that lack exact characterizations of selection, where conditional selective inference was previously infeasible.

We propose VoLux-GAN, a generative framework to synthesize 3D-aware faces with convincing relighting. Our main contribution is a volumetric HDRI relighting method that can efficiently accumulate albedo, diffuse and specular lighting contributions along each 3D ray for any desired HDR environmental map. Additionally, we show the importance of supervising the image decomposition process using multiple discriminators. In particular, we propose a data augmentation technique that leverages recent advances in single image portrait relighting to enforce consistent geometry, albedo, diffuse and specular components. Multiple experiments and comparisons with other generative frameworks show how our model is a step forward towards photorealistic relightable 3D generative models.

The quest to understand the nature of the electronic state in SmB6 has been challenging, perplexing and surprising researchers for over half a century. In the theoretically predicted topological Kondo insulator SmB6, the nature of the bulk electronic structure is not characterised unambiguously by quantum oscillations due to contrary interpretations. One simple definition of an electrical insulator is a material that lacks a Fermi surface and here we report the results of our investigation into its existence in SmB6 by Compton scattering. Compton scattering measures occupied electron momentum states, is bulk sensitive due to the high energy of the incoming photons and is also an ultra-fast probe of the correlated many-body electron wavefunction. Remarkably, direct evidence for a three-dimensional remnant Fermi surface is observed. However, a further dichotomy is raised in that the full occupancy expected of a conventional metal is not reproduced. Our observation of a remnant Fermi surface using a momentum-resolved probe provides significant new evidence that the paradigm of a bulk insulator is not robust for SmB6., Comment: 14 pages, 10 figures (Supplementary information included)

We define the localisation of a Hilbert module in analogy to the local multiplier algebra. We use properties of this localisation to enrich non-closed actions on $C^*$-algebras to closed actions on local multiplier algebras, and descend known results on such closed actions down to their unclosed counterparts. We define weak Cartan inclusions and characterise them as crossed products by inverse semigroup actions. We show that in the commutative case we show that weak Cartan subalgebras are maximal abelian, thereby generalising the case studied by Renault., Comment: Counterexamples to some of the main results have been found

The tube method or the volume-of-tube method approximates the tail probability of the maximum of a smooth Gaussian random field with zero mean and unit variance. This method evaluates the volume of a spherical tube about the index set, and then transforms it to the tail probability. In this study, we generalize the tube method to a case in which the variance is not constant. We provide the volume formula for a spherical tube with a non-constant radius in terms of curvature tensors, and the tail probability formula of the maximum of a Gaussian random field with inhomogeneous variance, as well as its Laplace approximation. In particular, the critical radius of the tube is generalized for evaluation of the asymptotic approximation error. As an example, we discuss the approximation of the largest eigenvalue distribution of the Wishart matrix with a non-identity matrix parameter. The Bonferroni method is the tube method when the index set is a finite set. We provide the formula for the asymptotic approximation error for the Bonferroni method when the variance is not constant., Comment: 30 pages, 3 figures