Search

We realize all irreducible unitary representations of the group $\mathrm{SO}_0(n+1,1)$ on explicit Hilbert spaces of vector-valued $L^2$-functions on $\mathbb{R}^n\setminus\{0\}$. The key ingredient in our construction is an explicit expression for the standard Knapp-Stein intertwining operators between arbitrary principal series representations in terms of the Euclidean Fourier transform on a maximal unipotent subgroup isomorphic to $\mathbb{R}^n$. As an application, we describe the space of Whittaker vectors on all irreducible Casselman-Wallach representations. Moreover, the new realizations of the irreducible unitary representations immediately reveal their decomposition into irreducible representations of a parabolic subgroup, thus providing a simple proof of a recent result of Liu-Oshima-Yu., Comment: 27 pages

The pair of real reductive groups $(G,H)=(\operatorname{GL}(n+1,\mathbb{R}),\operatorname{GL}(n,\mathbb{R}))$ is a strong Gelfand pair, i.e. the multiplicities $\dim\operatorname{Hom}_H(\pi|_H,\tau)$ are either $0$ or $1$ for all irreducible Casselman-Wallach representations $\pi$ of $G$ and $\tau$ of $H$. This paper is concerned with the construction of explicit intertwining operators in $\operatorname{Hom}_H(\pi|_H,\tau)$, so-called symmetry breaking operators, in the case where both $\pi$ and $\tau$ are principal series representations. Such operators come in families that depend meromorphically on the induction parameters, and we show how to normalize them in order to make the parameter dependence holomorphic. This is done by establishing explicit Bernstein-Sato identities for their distribution kernels as well as explicit functional identities for the composition of symmetry breaking operators with standard Knapp-Stein intertwining operators for $G$ and $H$. We also show that the obtained normalization is optimal and identify a subset of parameters for which the family of operators vanishes. Finally, we relate the operators to the local archimedean Rankin-Selberg integrals and use this relation to evaluate them on the spherical vectors., Comment: 33 pages

This paper builds on our previous work in which we showed that, for all connected semisimple linear Lie groups $G$ acting on a non-compactly causal symmetric space $M = G/H$, every irreducible unitary representation of $G$ can be realized by boundary value maps of holomorphic extensions in distributional sections of a vector bundle over $M$. In the present paper we discuss this procedure for the connected Lorentz group $G = SO_{1,d}(R)_e$ acting on de Sitter space $M = dS^d$. We show in particular that the previously constructed nets of real subspaces satisfy the locality condition. Following ideas of Bros and Moschella from the 1990's, we show that the matrix-valued spherical function that corresponds to our extension process extends analytically to a large domain $G_C^{cut}$ in the complexified group $G_C = \SO_{1,d}(C)$, which for $d = 1$ specializes to the complex cut plane $C \setminus (-\infinity, 0]$. A number of special situations is discussed specifically: (a) The case $d = 1$, which closely corresponds to standard subspaces in Hilbert spaces, (b) the case of scalar-valued functions, which for $d > 2$ is the case of spherical representations, for which we also describe the jump singularities of the holomorphic extensions on the cut in de Sitter space, (c) the case $d = 3$, where we obtain rather explicit formulas for the matrix-valued spherical functions., Comment: 53pp

For every simple Hermitian Lie group $G$, we consider a certain maximal parabolic subgroup whose unipotent radical $N$ is either abelian (if $G$ is of tube type) or two-step nilpotent (if $G$ is of non-tube type). By the generalized Whittaker Plancherel formula we mean the Plancherel decomposition of $L^2(G/N,\omega)$, the space of square-integrable sections of the homogeneous vector bundle over $G/N$ associated with an irreducible unitary representation $\omega$ of $N$. Assuming that the central character of $\omega$ is contained in a certain cone, we construct embeddings of all holomorphic discrete series representations of $G$ into $L^2(G/N,\omega)$ and show that the multiplicities are equal to the dimensions of the lowest $K$-types. The construction is in terms of a kernel function which can be explicitly defined using certain projections inside a complexification of $G$. This kernel function carries all information about the holomorphic discrete series embedding, the lowest $K$-type as functions on $G/N$, as well as the associated Whittaker vectors., Comment: 18 pages

On a finite regular graph, (co)resonant states are eigendistributions of the transfer operator associated to the shift on one-sided infinite non-backtracking paths. We introduce two pairings of resonant and coresonant states, the vertex pairing which involves only the dependence on the initial/terminal vertex of the path, and the geodesic pairing which is given by integrating over all geodesics the evaluation of the coresonant state on the first half of the geodesic times the resonant state on the second half. The main result is that these two pairings coincide up to a constant which depends on the resonance, i.e. the corresponding eigenvalue of the transfer operator., Comment: 41 pages

For a finite graph, we establish natural isomorphisms between eigenspaces of a Laplace operator acting on functions on the edges and eigenspaces of a transfer operator acting on functions on one-sided infinite non-backtracking paths. Interpreting the transfer operator as a classical dynamical system and the Laplace operator as its quantization, this result can be viewed as a quantum-classical correspondence. In contrast to previously established quantum-classical correspondences for the vertex Laplacian which exclude certain exceptional spectral parameters, our correspondence is valid for all parameters. This allows us to relate certain spectral quantities to topological properties of the graph such as the cyclomatic number and the 2-colorability. The quantum-classical correspondence for the edge Laplacian is induced by an edge Poisson transform on the universal covering of the graph which is a tree of bounded degree. In the special case of regular trees, we relate both the vertex and the edge Poisson transform to the representation theory of the automorphism group of the tree and study associated operator valued Hecke algebras., Comment: 46 pages

For a quadratic extension $\mathbb{E}/\mathbb{F}$ of non-archimedean local fields we construct explicit holomorphic families of intertwining operators between principal series representations of $\operatorname{PGL}(2,\mathbb{E})$ and $\operatorname{PGL}(2,\mathbb{F})$, also referred to as symmetry breaking operators. These families are given in terms of their distribution kernels which can be viewed as distributions on $\mathbb{E}$ depending holomorphically on the principal series parameters. For all such parameters we determine the support of these distributions, and we study their mapping properties. This leads to a classification of all intertwining operators between principal series representations, not necessarily irreducible. As an application, we show that every Steinberg representation of $\operatorname{PGL}(2,\mathbb{E})$ contains a Steinberg representation of $\operatorname{PGL}(2,\mathbb{F})$ as a direct summand of Hilbert spaces., Comment: 42 pages

We propose a fast and generalizable solution to Multi-view Photometric Stereo (MVPS), called MVPSNet. The key to our approach is a feature extraction network that effectively combines images from the same view captured under multiple lighting conditions to extract geometric features from shading cues for stereo matching. We demonstrate these features, termed `Light Aggregated Feature Maps' (LAFM), are effective for feature matching even in textureless regions, where traditional multi-view stereo methods fail. Our method produces similar reconstruction results to PS-NeRF, a state-of-the-art MVPS method that optimizes a neural network per-scene, while being 411$\times$ faster (105 seconds vs. 12 hours) in inference. Additionally, we introduce a new synthetic dataset for MVPS, sMVPS, which is shown to be effective to train a generalizable MVPS method.

Every simple Hermitian Lie group has a unique family of spherical representations induced from a maximal parabolic subgroup whose unipotent radical is a Heisenberg group. For most Hermitian groups, this family contains a complementary series, and at its endpoint sits a proper unitarizable subrepresentation. We show that this subrepresentation is next-to-minimal in the sense that its associated variety is a next-to-minimal nilpotent coadjoint orbit. Moreover, for the Hermitian groups $\operatorname{SO}_0(2,n)$ and $E_{6(-14)}$ we study some branching problems of these next-to-minimal representations., Comment: 21 pages

The goal of this article is to study the branching problem for a holomorphic discrete series representation of the conformal group of a simple Euclidean Jordan algebra $V$ restricted to the subgroup $\operatorname{PSL}_2(\mathbb{R})\times\operatorname{Aut}(V)$ where $\operatorname{Aut}(V)$ denotes the compact group of automorphisms of $V$. We use a realization of the holomorphic discrete series on a space of vector-values $L^2$-functions as well as the stratified model developed by the second author to relate the branching problem to the decomposition of certain representations of the compact group $\operatorname{Aut}(V)$ and to vector-valued orthogonal polynomials., Comment: 16 pages

The success of data mixing augmentations in image classification tasks has been well-received. However, these techniques cannot be readily applied to object detection due to challenges such as spatial misalignment, foreground/background distinction, and plurality of instances. To tackle these issues, we first introduce a novel conceptual framework called Supervision Interpolation (SI), which offers a fresh perspective on interpolation-based augmentations by relaxing and generalizing Mixup. Based on SI, we propose LossMix, a simple yet versatile and effective regularization that enhances the performance and robustness of object detectors and more. Our key insight is that we can effectively regularize the training on mixed data by interpolating their loss errors instead of ground truth labels. Empirical results on the PASCAL VOC and MS COCO datasets demonstrate that LossMix can consistently outperform state-of-the-art methods widely adopted for detection. Furthermore, by jointly leveraging LossMix with unsupervised domain adaptation, we successfully improve existing approaches and set a new state of the art for cross-domain object detection., Comment: AAAI-24 Camera Ready Version, with supplementary material, 15 pages

Let G be a connected simple linear Lie group and H in G a symmetric subgroup such that the corresponding symmetric space G/H is non-compactly causal. We show that any irreducible unitary representation of G leads naturally to a net of standard subspaces on G/H that is isotone, covariant and has the Reeh--Schlieder and the Bisognano--Wichmann property. We also show that this result extends to the universal covering group of SL(2,R) which has some interesting application to intersections of standard subspaces associated to representations of such groups. For this a detailed study of hyperfunction and distribution vectors is needed. In particular we show that every H-finite hyperfunction vector is in fact a distribution vector., Comment: Final version, 75 pages

Reconstructing a 3D surface from colonoscopy video is challenging due to illumination and reflectivity variation in the video frame that can cause defective shape predictions. Aiming to overcome this challenge, we utilize the characteristics of surface normal vectors and develop a two-step neural framework that significantly improves the colonoscopy reconstruction quality. The normal-based depth initialization network trained with self-supervised normal consistency loss provides depth map initialization to the normal-depth refinement module, which utilizes the relationship between illumination and surface normals to refine the frame-wise normal and depth predictions recursively. Our framework's depth accuracy performance on phantom colonoscopy data demonstrates the value of exploiting the surface normals in colonoscopy reconstruction, especially on en face views. Due to its low depth error, the prediction result from our framework will require limited post-processing to be clinically applicable for real-time colonoscopy reconstruction., Comment: Accepted at IPMI 2023; first two authors contributed equally

For any pseudo-Riemannian hyperbolic space $X$ over $\mathbb{R},\mathbb{C},\mathbb{H}$ or $\mathbb{O}$, we show that the resolvent $R(z)=(\Box-z\operatorname{Id})^{-1}$ of the Laplace-Beltrami operator $-\Box$ on $X$ can be extended meromorphically across the spectrum of $\Box$ as a family of operators $C_c^\infty(X)\to \mathcal{D}'(X)$. Its poles are called resonances and we determine them explicitly in all cases. For each resonance, the image of the corresponding residue operator in $\mathcal{D}'(X)$ forms a representation of the isometry group of $X$, which we identify with a subrepresentation of a degenerate principal series. Our study includes in particular the case of even functions on de Sitter and Anti-de Sitter spaces. For Riemannian symmetric spaces analogous results were obtained by Miatello-Will and Hilgert-Pasquale. The main qualitative differences between the Riemannian and the non-Riemannian setting are that for non-Riemannian spaces the resolvent can have poles of order two, it can have a pole at the branching point of the covering to which $R(z)$ extends, and the residue representations can be infinite-dimensional., Comment: 23 pages, final published version

We present the design of a productionized end-to-end stereo depth sensing system that does pre-processing, online stereo rectification, and stereo depth estimation with a fallback to monocular depth estimation when rectification is unreliable. The output of our depth sensing system is then used in a novel view generation pipeline to create 3D computational photography effects using point-of-view images captured by smart glasses. All these steps are executed on-device on the stringent compute budget of a mobile phone, and because we expect the users can use a wide range of smartphones, our design needs to be general and cannot be dependent on a particular hardware or ML accelerator such as a smartphone GPU. Although each of these steps is well studied, a description of a practical system is still lacking. For such a system, all these steps need to work in tandem with one another and fallback gracefully on failures within the system or less than ideal input data. We show how we handle unforeseen changes to calibration, e.g., due to heat, robustly support depth estimation in the wild, and still abide by the memory and latency constraints required for a smooth user experience. We show that our trained models are fast, and run in less than 1s on a six-year-old Samsung Galaxy S8 phone's CPU. Our models generalize well to unseen data and achieve good results on Middlebury and in-the-wild images captured from the smart glasses., Comment: Accepted at CVPR2023

In this work, we propose a novel and scalable solution to address the challenges of developing efficient dense predictions on edge platforms. Our first key insight is that MultiTask Learning (MTL) and hardware-aware Neural Architecture Search (NAS) can work in synergy to greatly benefit on-device Dense Predictions (DP). Empirical results reveal that the joint learning of the two paradigms is surprisingly effective at improving DP accuracy, achieving superior performance over both the transfer learning of single-task NAS and prior state-of-the-art approaches in MTL, all with just 1/10th of the computation. To the best of our knowledge, our framework, named EDNAS, is the first to successfully leverage the synergistic relationship of NAS and MTL for DP. Our second key insight is that the standard depth training for multi-task DP can cause significant instability and noise to MTL evaluation. Instead, we propose JAReD, an improved, easy-to-adopt Joint Absolute-Relative Depth loss, that reduces up to 88% of the undesired noise while simultaneously boosting accuracy. We conduct extensive evaluations on standard datasets, benchmark against strong baselines and state-of-the-art approaches, as well as provide an analysis of the discovered optimal architectures., Comment: WACV 2023. 14 pages, 5 figures

For a Hermitian Lie group $G$, we study the family of representations induced from a character of the maximal parabolic subgroup $P=MAN$ whose unipotent radical $N$ is a Heisenberg group. Realizing these representations in the non-compact picture on a space $I(\nu)$ of functions on the opposite unipotent radical $\bar{N}$, we apply the Heisenberg group Fourier transform mapping functions on $\bar N$ to operators on Fock spaces. The main result is an explicit expression for the Knapp-Stein intertwining operators $I(\nu)\to I(-\nu)$ on the Fourier transformed side. This gives a new construction of the complementary series and of certain unitarizable subrepresentations at points of reducibility. Further auxiliary results are a Bernstein-Sato identity for the Knapp-Stein kernel on $\bar{N}$ and the decomposition of the metaplectic representation under the non-compact group $M$., Comment: 44 pages, v2: final published version

We study degenerate Whittaker vectors in scalar type holomorphic discrete series representations of tube type Hermitian Lie groups and their analytic continuation. In four different realizations, the bounded domain picture, the tube domain picture, the $L^2$-model and the Fock model, we find their explicit $K$-type expansions. The coefficients are expressed in terms of the generalized Laguerre functions on the corresponding symmetric cone, and we relate the $K$-type expansions to the formula for the generating function of the Laguerre polynomials and to their recurrence relations., Comment: 19 pages, v2: final published version

Keystroke inference attacks are a form of side-channel attacks in which an attacker leverages various techniques to recover a user's keystrokes as she inputs information into some display (e.g., while sending a text message or entering her pin). Typically, these attacks leverage machine learning approaches, but assessing the realism of the threat space has lagged behind the pace of machine learning advancements, due in-part, to the challenges in curating large real-life datasets. We aim to overcome the challenge of having limited number of real data by introducing a video domain adaptation technique that is able to leverage synthetic data through supervised disentangled learning. Specifically, for a given domain, we decompose the observed data into two factors of variation: Style and Content. Doing so provides four learned representations: real-life style, synthetic style, real-life content and synthetic content. Then, we combine them into feature representations from all combinations of style-content pairings across domains, and train a model on these combined representations to classify the content (i.e., labels) of a given datapoint in the style of another domain. We evaluate our method on real-life data using a variety of metrics to quantify the amount of information an attacker is able to recover. We show that our method prevents our model from overfitting to a small real-life training set, indicating that our method is an effective form of data augmentation, thereby making keystroke inference attacks more practical.

For a Hermitian Lie group $G$ of tube type we find the contribution of the holomorphic discrete series to the Plancherel decomposition of the Whittaker space $L^2(G/N,\psi)$, where $N$ is the unipotent radical of the Siegel parabolic subgroup and $\psi$ is a certain non-degenerate unitary character on $N$. The holomorphic discrete series embeddings are constructed in terms of generalized Whittaker vectors for which we find explicit formulas in the bounded domain realization, the tube domain realization and the $L^2$-model of the holomorphic discrete series. Although $L^2(G/N,\psi)$ does not have finite multiplicities in general, the holomorphic discrete series contribution does. Moreover, we obtain an explicit formula for the formal dimensions of the holomorphic discrete series embeddings, and we interpret the holomorphic discrete series contribution to $L^2(G/N,\psi)$ as boundary values of holomorphic functions on a domain $\Xi$ in a complexification $G_{\mathbb{C}}$ of $G$ forming a Hardy type space $\mathcal{H}_2(\Xi,\psi)$., Comment: 36 pages, final published version

We introduce a unified single and multi-view neural implicit 3D reconstruction framework VPFusion. VPFusion attains high-quality reconstruction using both - 3D feature volume to capture 3D-structure-aware context, and pixel-aligned image features to capture fine local detail. Existing approaches use RNN, feature pooling, or attention computed independently in each view for multi-view fusion. RNNs suffer from long-term memory loss and permutation variance, while feature pooling or independently computed attention leads to representation in each view being unaware of other views before the final pooling step. In contrast, we show improved multi-view feature fusion by establishing transformer-based pairwise view association. In particular, we propose a novel interleaved 3D reasoning and pairwise view association architecture for feature volume fusion across different views. Using this structure-aware and multi-view-aware feature volume, we show improved 3D reconstruction performance compared to existing methods. VPFusion improves the reconstruction quality further by also incorporating pixel-aligned local image features to capture fine detail. We verify the effectiveness of VPFusion on the ShapeNet and ModelNet datasets, where we outperform or perform on-par the state-of-the-art single and multi-view 3D shape reconstruction methods.

One of the key elements of reconstructing a 3D mesh from a monocular video is generating every frame's depth map. However, in the application of colonoscopy video reconstruction, producing good-quality depth estimation is challenging. Neural networks can be easily fooled by photometric distractions or fail to capture the complex shape of the colon surface, predicting defective shapes that result in broken meshes. Aiming to fundamentally improve the depth estimation quality for colonoscopy 3D reconstruction, in this work we have designed a set of training losses to deal with the special challenges of colonoscopy data. For better training, a set of geometric consistency objectives was developed, using both depth and surface normal information. Also, the classic photometric loss was extended with feature matching to compensate for illumination noise. With the training losses powerful enough, our self-supervised framework named ColDE is able to produce better depth maps of colonoscopy data as compared to the previous work utilizing prior depth knowledge. Used in reconstruction, our network is able to reconstruct good-quality colon meshes in real-time without any post-processing, making it the first to be clinically applicable., Comment: 13 pages, 5 figures

Let $X$ be a compact hyperbolic surface with finite order singularities, $X_1$ its unit tangent bundle. We consider the Ruelle zeta function $R(s;\rho)$ associated to a representation $\rho\colon\pi_1(X_1)\to\operatorname{GL}(V_\rho)$. If $\rho$ does not factor through $\pi_1(X)$, we show that the value at $0$ of the Ruelle zeta function equals the sign-refined Reidemeister-Turaev torsion of $(X_1, \rho)$ with respect to the Euler structure induced by the geodesic flow and to the natural homology orientation of $X_1$. It generalizes Fried's conjecture to non-unitary representations, and solves the phase and sign ambiguity in the unitary case. We also compute the vanishing order and the leading coefficient of the Ruelle zeta function at $s=0$ when $\rho$ factors through $\pi_1(X)$., Comment: 48 pages, final published version

Let $X$ be a compact, hyperbolic surface of genus $g\geq 2$. In this paper, we prove that the twisted Selberg and Ruelle zeta functions, associated with an arbitrary, finite-dimensional, complex representation $\chi$ of $\pi_1(X)$ admit a meromorphic continuation to $\mathbb{C}$. Moreover, we study the behaviour of the twisted Ruelle zeta function at $s=0$ and prove that at this point, it has a zero of order $\dim(\chi)(2g-2)$., Comment: v2: The functional equation for the twisted Ruelle zeta function is made more explicit

The restriction of an irreducible unitary representation $\pi$ of a real reductive group $G$ to a reductive subgroup $H$ decomposes into a direct integral of irreducible unitary representations $\tau$ of $H$ with multiplicities $m(\pi,\tau)\in\mathbb{N}\cup\{\infty\}$. We show that on the smooth vectors of $\pi$, the direct integral is pointwise defined. This implies that $m(\pi,\tau)$ is bounded above by the dimension of the space $\operatorname{Hom}_H(\pi^\infty|_H,\tau^\infty)$ of intertwining operators between the smooth vectors, also called symmetry breaking operators, and provides a precise relation between these two concepts of multiplicity., Comment: 5 pages

For a simple real Lie group $G$ with Heisenberg parabolic subgroup $P$, we study the corresponding degenerate principal series representations. For a certain induction parameter the kernel of the conformally invariant system of second order differential operators constructed by Barchini, Kable and Zierau is a subrepresentation which turns out to be the minimal representation. To study this subrepresentation, we take the Heisenberg group Fourier transform in the non-compact picture and show that it yields a new realization of the minimal representation on a space of $L^2$-functions. The Lie algebra action is given by differential operators of order $\leq3$ and we find explicit formulas for the functions constituting the lowest $K$-type. These $L^2$-models were previously known for the groups $\operatorname{SO}(n,n)$, $E_{6(6)}$, $E_{7(7)}$ and $E_{8(8)}$ by Kazhdan and Savin, for the group $G_{2(2)}$ by Gelfand, and for the group $\widetilde{\operatorname{SL}}(3,\mathbb{R})$ by Torasso, using different methods. Our new approach provides a uniform and systematic treatment of these cases and also constructs new $L^2$-models for $E_{6(2)}$, $E_{7(-5)}$ and $E_{8(-24)}$ for which the minimal representation is a continuation of the quaternionic discrete series, and for the groups $\widetilde{\operatorname{SO}}(p,q)$ with either $p\geq q=3$ or $p,q\geq4$ and $p+q$ even. As a byproduct of our construction, we find an explicit formula for the group action of a non-trivial Weyl group element that, together with the simple action of a parabolic subgroup, generates $G$., Comment: 140 pages, final published version

Despite remarkable improvements in speed and accuracy, convolutional neural networks (CNNs) still typically operate as monolithic entities at inference time. This poses a challenge for resource-constrained practical applications, where both computational budgets and performance needs can vary with the situation. To address these constraints, we propose the Any-Width Network (AWN), an adjustable-width CNN architecture and associated training routine that allow for fine-grained control over speed and accuracy during inference. Our key innovation is the use of lower-triangular weight matrices which explicitly address width-varying batch statistics while being naturally suited for multi-width operations. We also show that this design facilitates an efficient training routine based on random width sampling. We empirically demonstrate that our proposed AWNs compare favorably to existing methods while providing maximally granular control during inference., Comment: 8 pages. Published at CVPR 2020 Workshop on Efficient Deep Learning in Computer Vision. Code at https://github.com/thanhmvu/awn

A vision-based keystroke inference attack is a side-channel attack in which an attacker uses an optical device to record users on their mobile devices and infer their keystrokes. The threat space for these attacks has been studied in the past, but we argue that the defining characteristics for this threat space, namely the strength of the attacker, are outdated. Previous works do not study adversaries with vision systems that have been trained with deep neural networks because these models require large amounts of training data and curating such a dataset is expensive. To address this, we create a large-scale synthetic dataset to simulate the attack scenario for a keystroke inference attack. We show that first pre-training on synthetic data, followed by adopting transfer learning techniques on real-life data, increases the performance of our deep learning models. This indicates that these models are able to learn rich, meaningful representations from our synthetic data and that training on the synthetic data can help overcome the issue of having small, real-life datasets for vision-based key stroke inference attacks. For this work, we focus on single keypress classification where the input is a frame of a keypress and the output is a predicted key. We are able to get an accuracy of 95.6% after pre-training a CNN on our synthetic data and training on a small set of real-life data in an adversarial domain adaptation framework. Source Code for Simulator: https://github.com/jlim13/keystroke-inference-attack-synthetic-dataset-generator-

3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo. We present an end-to-end system for creating and viewing 3D photos, and the algorithmic and design choices therein. Our 3D photos are captured in a single shot and processed directly on a mobile device. The method starts by estimating depth from the 2D input image using a new monocular depth estimation network that is optimized for mobile devices. It performs competitively to the state-of-the-art, but has lower latency and peak memory consumption and uses an order of magnitude fewer parameters. The resulting depth is lifted to a layered depth image, and new geometry is synthesized in parallax regions. We synthesize color texture and structures in the parallax regions as well, using an inpainting network, also optimized for mobile devices, on the LDI directly. Finally, we convert the result into a mesh-based representation that can be efficiently transmitted and rendered even on low-end devices and over poor network connections. Altogether, the processing takes just a few seconds on a mobile device, and the result can be instantly viewed and shared. We perform extensive quantitative evaluation to validate our system and compare its new components against the current state-of-the-art., Comment: Project page: https://facebookresearch.github.io/one_shot_3d_photography/ Code: https://github.com/facebookresearch/one_shot_3d_photography

Low-frequency long-range errors (drift) are an endemic problem in 3D structure from motion, and can often hamper reasonable reconstructions of the scene. In this paper, we present a method to dramatically reduce scale and positional drift by using extended structural features such as planes and vanishing points. Unlike traditional feature matches, our extended features are able to span non-overlapping input images, and hence provide long-range constraints on the scale and shape of the reconstruction. We add these features as additional constraints to a state-of-the-art global structure from motion algorithm and demonstrate that the added constraints enable the reconstruction of particularly drift-prone sequences such as long, low field-of-view videos without inertial measurements. Additionally, we provide an analysis of the drift-reducing capabilities of these constraints by evaluating on a synthetic dataset. Our structural features are able to significantly reduce drift for scenes that contain long-spanning man-made structures, such as aligned rows of windows or planar building facades., Comment: 3DV 2020

In this work, we propose "tangent images," a spherical image representation that facilitates transferable and scalable $360^\circ$ computer vision. Inspired by techniques in cartography and computer graphics, we render a spherical image to a set of distortion-mitigated, locally-planar image grids tangent to a subdivided icosahedron. By varying the resolution of these grids independently of the subdivision level, we can effectively represent high resolution spherical images while still benefiting from the low-distortion icosahedral spherical approximation. We show that training standard convolutional neural networks on tangent images compares favorably to the many specialized spherical convolutional kernels that have been developed, while also scaling efficiently to handle significantly higher spherical resolutions. Furthermore, because our approach does not require specialized kernels, we show that we can transfer networks trained on perspective images to spherical data without fine-tuning and with limited performance drop-off. Finally, we demonstrate that tangent images can be used to improve the quality of sparse feature detection on spherical images, illustrating its usefulness for traditional computer vision tasks like structure-from-motion and SLAM., Comment: Updated version of CVPR 2020 publication (9 pages, 13 pages supplementary). Code: https://github.com/meder411/Tangent-Images

The rapid development of inexpensive commodity depth sensors has made keypoint detection and matching in the depth image modality an important problem in computer vision. Despite great improvements in recent RGB local feature learning methods, adapting them directly in the depth modality leads to unsatisfactory performance. Most of these methods do not explicitly reason beyond the visible pixels in the images. To address the limitations of these methods, we propose a framework ViewSynth, to jointly learn: (1) viewpoint invariant keypoint-descriptor from depth images using a proposed Contrastive Matching Loss, and (2) view synthesis of depth images from different viewpoints using the proposed View Synthesis Module and View Synthesis Loss. By learning view synthesis, we explicitly encourage the feature extractor to encode information about not only the visible, but also the occluded parts of the scene. We demonstrate that in the depth modality, ViewSynth outperforms the state-of-the-art depth and RGB local feature extraction techniques in the 3D keypoint matching and camera localization tasks on the RGB-D datasets 7-Scenes, TUM RGBD and CoRBS in most scenarios. We also show the generalizability of ViewSynth in 3D keypoint matching across different datasets., Comment: Accepted to BMVC 2020

We present a versatile formulation of the convolution operation that we term a "mapped convolution." The standard convolution operation implicitly samples the pixel grid and computes a weighted sum. Our mapped convolution decouples these two components, freeing the operation from the confines of the image grid and allowing the kernel to process any type of structured data. As a test case, we demonstrate its use by applying it to dense inference on spherical data. We perform an in-depth study of existing spherical image convolution methods and propose an improved sampling method for equirectangular images. Then, we discuss the impact of data discretization when deriving a sampling function, highlighting drawbacks of the cube map representation for spherical data. Finally, we illustrate how mapped convolutions enable us to convolve directly on a mesh by projecting the spherical image onto a geodesic grid and training on the textured mesh. This method exceeds the state of the art for spherical depth estimation by nearly 17%. Our findings suggest that mapped convolutions can be instrumental in expanding the application scope of convolutional neural networks.

Applying convolutional neural networks to spherical images requires particular considerations. We look to the millennia of work on cartographic map projections to provide the tools to define an optimal representation of spherical images for the convolution operation. We propose a representation for deep spherical image inference based on the icosahedral Snyder equal-area (ISEA) projection, a projection onto a geodesic grid, and show that it vastly exceeds the state-of-the-art for convolution on spherical images, improving semantic segmentation results by 12.6%., Comment: Oral presentation at the 2019 SUMO Workshop on 360{\deg} Indoor Scene Understanding and Modeling at CVPR2019, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2019

Deep learning-based, single-view depth estimation methods have recently shown highly promising results. However, such methods ignore one of the most important features for determining depth in the human vision system, which is motion. We propose a learning-based, multi-view dense depth map and odometry estimation method that uses Recurrent Neural Networks (RNN) and trains utilizing multi-view image reprojection and forward-backward flow-consistency losses. Our model can be trained in a supervised or even unsupervised mode. It is designed for depth and visual odometry estimation from video where the input frames are temporally correlated. However, it also generalizes to single-view depth estimation. Our method produces superior results to the state-of-the-art approaches for single-view and multi-view learning-based depth estimation on the KITTI driving dataset.

The Branson-Gover operators are conformally invariant differential operators of even degree acting on differential forms. They can be interpolated by a holomorphic family of conformally invariant integral operators called fractional Branson-Gover operators. For Euclidean spaces we show that the fractional Branson-Gover operators can be obtained as Dirichlet-to-Neumann operators of certain conformally invariant boundary value problems, generalizing the work of Caffarelli-Silvestre for the fractional Laplacians to differential forms. The relevant boundary value problems are studied in detail and we find appropriate Sobolev type spaces in which there exist unique solutions and obtain the explicit integral kernels of the solution operators as well as some of its properties., Comment: 25 pages

For a pair of real reductive groups $G'\subset G$ we consider the space ${\rm Hom}_{G'}(\pi|_{G'},\tau)$ of intertwining operators between spherical principal series representations $\pi$ of $G$ and $\tau$ of $G'$, also called \emph{symmetry breaking operators}. Restricting to those pairs $(G,G')$ where ${\rm dim\,Hom}_{G'}(\pi|_{G'},\tau)<\infty$ and $G$ and $G'$ are of real rank one, we classify all symmetry breaking operators explicitly in terms of their distribution kernels. This generalizes previous work by Kobayashi--Speh for $(G,G')=({\rm O}(1,n+1),{\rm O}(1,n))$ to the reductive pairs $$ (G,G') = ({\rm U}(1,n+1;\mathbb{F}),{\rm U}(1,m+1;\mathbb{F})\times F) \qquad \mbox{with $\mathbb{F}=\mathbb{C},\mathbb{H},\mathbb{O}$ and $F<{\rm U}(n-m;\mathbb{F})$.} $$ In most cases, all symmetry breaking operators can be constructed using one meromorphic family of distributions whose poles and residues we describe in detail. In addition to this family, there may occur some sporadic symmetry breaking operators which we determine explicitly., Comment: 56 pages

Learning-based, single-view depth estimation often generalizes poorly to unseen datasets. While learning-based, two-frame depth estimation solves this problem to some extent by learning to match features across frames, it performs poorly at large depth where the uncertainty is high. There exists few learning-based, multi-view depth estimation methods. In this paper, we present a learning-based, multi-view dense depth map and ego-motion estimation method that uses Recurrent Neural Networks (RNN). Our model is designed for 3D reconstruction from video where the input frames are temporally correlated. It is generalizable to single- or two-view dense depth estimation. Compared to recent single- or two-view CNN-based depth estimation methods, our model leverages more views and achieves more accurate results, especially at large distances. Our method produces superior results to the state-of-the-art learning-based, single- or two-view depth estimation methods on both indoor and outdoor benchmark datasets. We also demonstrate that our method can even work on extremely difficult sequences, such as endoscopic video, where none of the assumptions (static scene, constant lighting, Lambertian reflection, etc.) from traditional 3D reconstruction methods hold.

We present a framework for edge-aware optimization that is an order of magnitude faster than the state of the art while having comparable performance. Our key insight is that the optimization can be formulated by leveraging properties of the domain transform, a method for edge-aware filtering that defines a distance-preserving 1D mapping of the input space. This enables our method to improve performance for a variety of problems including stereo, depth super-resolution, and render from defocus, while keeping the computational complexity linear in the number of pixels. Our method is highly parallelizable and adaptable, and it has demonstrable scalability with respect to image resolution.

We give a dynamical description, in terms of a Weil-type zeta function, to the holomorphic torsion with coefficients for certain compact Hermitian locally symmetric manifolds, whose connected group G of isometries of the universal cover has only one conjugacy class of cuspidal maximal parabolic subgroup and satisfies a technical Ansatz relative to the given coefficients. A distinguishing feature of our zeta function is that its construction involves in an essential way the geometry of a standard compactification of the universal cover. The two senior authors are indebted to their junior colleague, Jan Frahm, for his laborious work shedding light on the scope of the validity of the Ansatz, and for writing up the attached Appendix. The results therein show that for real rank one groups G the Ansatz is satisfied with respect to any coefficients, for some rank two groups G it is satisfied with respect to certain coefficients, and also that there are groups G which do not obey the Ansatz., Comment: Appendix by Jan Frahm. Revisions were made in Sections 2 and 3; clarifying material was added to Sections 4--9; Section 10 is new

The space of smooth sections of an equivariant line bundle over the real projective space $\mathbb{R}{\rm P}^n$ forms a natural representation of the group ${\rm GL}(n+1,\mathbb{R})$. We explicitly construct and classify all intertwining operators between such representations of ${\rm GL}(n+1,\mathbb{R})$ and its subgroup ${\rm GL}(n,\mathbb{R})$, intertwining for the subgroup. Intertwining operators of this form are called symmetry breaking operators, and they describe the occurrence of a representation of ${\rm GL}(n,\mathbb{R})$ inside the restriction of a representation of ${\rm GL}(n+1,\mathbb{R})$. In this way, our results contribute to the study of branching problems for the real reductive pair $({\rm GL}(n+1,\mathbb{R}),{\rm GL}(n,\mathbb{R}))$. The analogous classification is carried out for intertwining operators between algebraic sections of line bundles, where the Lie group action of ${\rm GL}(n,\mathbb{R})$ is replaced by the action of its Lie algebra $\mathfrak{gl}(n,\mathbb{R})$, and it turns out that all intertwining operators arise as restrictions of operators between smooth sections., Comment: 44 pages

We construct a minimal representation of the orthosymplectic Lie supergroup $OSp(p,q|2n)$, generalising the Schr\"odinger model of the minimal representation of $O(p,q)$ to the super case. The underlying Lie algebra representation is realized on functions on the minimal orbit inside the Jordan superalgebra associated with $\mathfrak{osp}(p,q|2n)$, so that our construction is in line with the orbit philosophy. Its annihilator is given by a Joseph-like ideal for $\mathfrak{osp}(p,q|2n)$, and therefore the representation is a natural generalization of a minimal representations to the context of Lie superalgebras. We also calculate its Gelfand--Kirillov dimension and construct a non-degenerate sesquilinear form for which the representation is skew-symmetric and which is the analogue of an $L^2$-inner product in the supercase., Comment: 45 pages