Search

Generative adversarial networks (GANs) are pow- erful generative models based on providing feed- back to a generative network via a discriminator network. However, the discriminator usually as- sesses individual samples. This prevents the dis- criminator from accessing global distributional statistics of generated samples, and often leads to mode dropping: the generator models only part of the target distribution. We propose to feed the discriminator with mixed batches of true and fake samples, and train it to predict the ratio of true samples in the batch. The latter score does not depend on the order of samples in a batch. Rather than learning this invariance, we introduce a generic permutation-invariant discriminator ar- chitecture. This architecture is provably a uni- versal approximator of all symmetric functions. Experimentally, our approach reduces mode col- lapse in GANs on two synthetic datasets, and obtains good results on the CIFAR10 and CelebA datasets, both qualitatively and quantitatively., Comment: Accepted at ICML 2018 (long oral)

Contextual bandit algorithms are essential for solving many real-world interactive machine learning problems. Despite multiple recent successes on statistically and computationally efficient methods, the practical behavior of these algorithms is still poorly understood. We leverage the availability of large numbers of supervised learning datasets to empirically evaluate contextual bandit algorithms, focusing on practical methods that learn by relying on optimization oracles from supervised learning. We find that a recent method (Foster et al., 2018) using optimism under uncertainty works the best overall. A surprisingly close second is a simple greedy baseline that only explores implicitly through the diversity of contexts, followed by a variant of Online Cover (Agarwal et al., 2014) which tends to be more conservative but robust to problem specification by design. Along the way, we also evaluate various components of contextual bandit algorithm design such as loss estimators. Overall, this is a thorough study and review of contextual bandit methodology., JMLR

International audience; Hyperspectral images (HSIs) are of crucial importance in order to better understand features from a large number of spectral channels. Restricted by its inner imaging mechanism, the spatial resolution is often limited for HSIs. To alleviate this issue, in this work, we propose a simple and efficient architecture of deep convolutional neural networks to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to a new architecture based on: 1) the use of the LR-HSI at the HR-MSI’s scale to get an output with satisfied spectral preservation and 2) the application of the attention and pixelShuffle modules to extract information, aiming to output high-quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves the best performance (both qualitatively and quantitatively) compared with recent state-of-the-art HSI super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as a better network generalization ability, a limited computational burden, and the robustness with respect to the number of training samples. Please find the source code and pretrained models from https://liangjiandeng.github.io/Projects_Res/HSRnet_2021tnnls.html .

Exoplanet detection by direct imaging is a difficult task: the faint signals from the objects of interest are buried under a spatially structured nuisance component induced by the host star. The exoplanet signals can only be identified when combining several observations with dedicated detection algorithms. In contrast to most of existing methods, we propose to learn a model of the spatial, temporal and spectral characteristics of the nuisance, directly from the observations. In a pre-processing step, a statistical model of their correlations is built locally, and the data are centered and whitened to improve both their stationarity and signal-to-noise ratio (SNR). A convolutional neural network (CNN) is then trained in a supervised fashion to detect the residual signature of synthetic sources in the pre-processed images. Our method leads to a better trade-off between precision and recall than standard approaches in the field. It also outperforms a state-of-the-art algorithm based solely on a statistical framework. Besides, the exploitation of the spectral diversity improves the performance compared to a similar model built solely from spatio-temporal data., accepted to EUSIPCO 2023

Self-supervised learning models have been shown to learn rich visual representations without requiring human annotations. However, in many real-world scenarios, labels are partially available, motivating a recent line of work on semi-supervised methods inspired by self-supervised principles. In this paper, we propose a conceptually simple yet empirically powerful approach to turn clustering-based self-supervised methods such as SwAV or DINO into semi-supervised learners. More precisely, we introduce a multi-task framework merging a supervised objective using ground-truth labels and a self-supervised objective relying on clustering assignments with a single cross-entropy loss. This approach may be interpreted as imposing the cluster centroids to be class prototypes. Despite its simplicity, we provide empirical evidence that our approach is highly effective and achieves state-of-the-art performance on CIFAR100 and ImageNet., Comment: CVPR 2023 - Code available at https://github.com/pietroastolfi/suave-daino

Data augmentation is known to improve the generalization capabilities of neural networks, provided that the set of transformations is chosen with care, a selection often performed manually. Automatic data augmentation aims at automating this process. However, most recent approaches still rely on some prior information; they start from a small pool of manually-selected default transformations that are either used to pretrain the network or forced to be part of the policy learned by the automatic data augmentation algorithm. In this paper, we propose to directly learn the augmentation policy without leveraging such prior knowledge. The resulting bilevel optimization problem becomes more challenging due to the larger search space and the inherent instability of bilevel optimization algorithms. To mitigate these issues (i) we follow a successive cold-start strategy with a Kullback-Leibler regularization, and (ii) we parameterize magnitudes as continuous distributions. Our approach leads to competitive results on standard benchmarks despite a more challenging setting, and generalizes beyond natural images., Accepted to CVPR 2023

Observing a human demonstrator manipulate objects provides a rich, scalable and inexpensive source of data for learning robotic policies. However, transferring skills from human videos to a robotic manipulator poses several challenges, not least a difference in action and observation spaces. In this work, we use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. Thanks to the diversity of this training data, the learned reward function sufficiently generalizes to image observations from a previously unseen robot embodiment and environment to provide a meaningful prior for directed exploration in reinforcement learning. We propose two methods for scoring states relative to a goal image: through direct temporal regression, and through distances in an embedding space obtained with time-contrastive learning. By conditioning the function on a goal image, we are able to reuse one model across a variety of tasks. Unlike prior work on leveraging human videos to teach robots, our method, Human Offline Learned Distances (HOLD) requires neither a priori data from the robot environment, nor a set of task-specific human demonstrations, nor a predefined notion of correspondence across morphologies, yet it is able to accelerate training of several manipulation tasks on a simulated robot arm compared to using only a sparse reward obtained from task completion.

International audience; Gradient clipping is a popular modification to standard (stochastic) gradient descent, at every iteration limiting the gradient norm to a certain value $c >0$. It is widely used for example for stabilizing the training of deep learning models (Goodfellow et al., 2016), or for enforcing differential privacy (Abadi et al., 2016). Despite popularity and simplicity of the clipping mechanism, its convergence guarantees often require specific values of $c$ and strong noise assumptions. In this paper, we give convergence guarantees that show precise dependence on arbitrary clipping thresholds $c$ and show that our guarantees are tight with both deterministic and stochastic gradients. In particular, we show that (i) for deterministic gradient descent, the clipping threshold only affects the higher-order terms of convergence, (ii) in the stochastic setting convergence to the true optimum cannot be guaranteed under the standard noise assumption, even under arbitrary small step-sizes. We give matching upper and lower bounds for convergence of the gradient norm when running clipped SGD, and illustrate these results with experiments.

Face frontalization consists of synthesizing a frontally-viewed face from an arbitrarily-viewed one. The main contribution of this paper is a frontalization methodology that preserves non-rigid facial deformations in order to boost the performance of visually assisted speech communication. The method alternates between the estimation of (i)~the rigid transformation (scale, rotation, and translation) and (ii)~the non-rigid deformation between an arbitrarily-viewed face and a face model. The method has two important merits: it can deal with non-Gaussian errors in the data and it incorporates a dynamical face deformation model. For that purpose, we use the generalized Student t-distribution in combination with a linear dynamic system in order to account for both rigid head motions and time-varying facial deformations caused by speech production. We propose to use the zero-mean normalized cross-correlation (ZNCC) score to evaluate the ability of the method to preserve facial expressions. The method is thoroughly evaluated and compared with several state of the art methods, either based on traditional geometric models or on deep learning. Moreover, we show that the method, when incorporated into deep learning pipelines, namely lip reading and speech enhancement, improves word recognition and speech intelligibilty scores by a considerable margin. Supplemental material is accessible at https://team.inria.fr/robotlearn/research/facefrontalization/, arXiv admin note: text overlap with arXiv:2202.00538

International audience; We consider a decentralized optimization problem, in which n nodes collaborate to optimize a global objective function using local communications only. While many decentralized algorithms focus on gossip communications (pairwise averaging), we consider a different scheme, in which a "token" that contains the current estimate of the model performs a random walk over the network, and updates its model using the local model of the node it is at. Indeed, token algorithms generally benefit from improved communication efficiency and privacy guarantees. We frame the token algorithm as a randomized gossip algorithm on a conceptual graph, which allows us to prove a series of convergence results for variance-reduced and accelerated token algorithms for the complete graph. We also extend these results to the case of multiple tokens by extending the conceptual graph, and to general graphs by tweaking the communication procedure. The reduction from token to well-studied gossip algorithms leads to tight rates for many token algorithms, and we illustrate their performance empirically.

We introduce a novel self-attention mechanism, which we call CSA (Chromatic Self-Attention), which extends the notion of attention scores to attention _filters_, independently modulating the feature channels. We showcase CSA in a fully-attentional graph Transformer CGT (Chromatic Graph Transformer) which integrates both graph structural information and edge features, completely bypassing the need for local message-passing components. Our method flexibly encodes graph structure through node-node interactions, by enriching the original edge features with a relative positional encoding scheme. We propose a new scheme based on random walks that encodes both structural and positional information, and show how to incorporate higher-order topological information, such as rings in molecular graphs. Our approach achieves state-of-the-art results on the ZINC benchmark dataset, while providing a flexible framework for encoding graph structure and incorporating higher-order topology.

International audience; Model-based approaches to pansharpening still constitute a class of widely employed methods, thanks to their straightforward applicability to many problems, dispensing the user from time-consuming training phases. The injection scheme based on an accurate estimation (exploiting regression) of the relationship between the details contained in the panchromatic (PAN) image and those required for the enhancement of the multispectral (MS) image represents the most updated approach to this problem, being characterized by both theoretical and practical optimality. We elaborated on this scheme by designing a procedure for estimating the key parameters required for the optimal setting of such a regression-based approach. We tested this approach on several datasets acquired by the WorldView satellites comparing the proposed approach with a benchmark consisting of some state-of-the-art pansharpening methods.

International audience; Deep learning works normally in PolSAR image classification because the complex terrain scattering characteristic results in large intraclass differences and high interclass similarity. Deep metric learning (DML) aims to make the features keep a closer intraclass and a farther interclass distance. Therefore, we introduce DML and then propose an N-cluster generative adversarial net (N-cluster GAN) framework for PolSAR image classification. However, existing DML losses mainly focus on the relationship between individual samples in feature space. Hence, we propose N-cluster loss that pays more attention to the overall structure of all samples. Meanwhile, traditional hard negative sample mining methods occupy lots of computational resources. In addition, the hard level of the negative samples will affect the model’s performance. Therefore, we explore a new method based on a GAN framework to replace the sample mining. Positive N-cluster loss is added to the discriminator ( D ), and a negative one is added to the generator ( G ). In this way, D will possess better classification ability, and G can produce hard negative samples for D . Then, the hard level of the generated negative samples will change with the discrimination of D , which is appropriate for the proposed model. N-cluster loss can be directly calculated through the extracted features rather than redundant data preparation. The proposed model is verified on four PolSAR datasets from two aspects of the loss function and negative samples mining. Then, it achieves competitive performance compared with state-of-the-art algorithms.

International audience; Model-based polarimetric target decomposition (TD) generally solves scattering components and parameters under a preset decomposition base; then, decomposition features are also obtained. However, a preset base could not be adjusted according to different scenes. Furthermore, solving the polarimetric parameters needs exploring additional information or considering limiting conditions to build equations, which is hard and easily brings negative effects into decomposition features. To this end, we regard the TD as a process of learning decomposition base and features by deep learning. Then, the polarimetric decomposition feature learning (PDFL) model is proposed in this article. Strictly, this model is not an incoherent TD method but a learning-based method. It does not need to construct the parameter solution equations or fixed base. Then, the decomposition base and feature can be adaptively learned according to the scattering characteristics of the current dataset. Due to the characteristics of unsupervised reconstruction, the deep autoencoder (DAE) is used as the model foundation. Then, some adjustments and constraints are utilized to make the DAE fit closely with TD. The encoder extracts the latent vector from polarimetric synthetic aperture radar (PolSAR) data, and then, the decoder reconstructs pseudodata on this latent vector. The reconstruction can be regarded as the inverse process of TD, so the base matrix of the decoder and the latent vector indicate the learned decomposition base and features when the model converges. The effectiveness of PDFL is verified on simulated and real PolSAR datasets. Compared with representative algorithms, the proposed model gains more discriminative features and achieves competitive performance on terrain classification and segmentation tasks.

International audience; Target decomposition features are the cornerstone of subsequent analyses for PolSAR images. Generally, adopting single or several decomposition algorithms limits the representation ability for original terrain characteristics. Using all the existing decomposition features, however, will definitely increase computational complexity. Besides, some features even have a negative effect on the following tasks. To address these problems, a sparse variational autoencoder feature selection framework (SVAE-FS) is proposed in this article. In detail, the encoder transforms the original feature set into latent space and then decoder reconstructs the corresponding pseudo set on this latent space. Similarly, a pseudo subset is subsequently obtained by the SVAE. The discrepancy, namely reconstruction error, between the pseudo set and the pseudo subset is taken as an evaluation criterion which reflects the feature representation ability of pseudo subset. Sparse constraint in the encoder makes the representative features stand out. Meanwhile, the linear feature transformation layer of the encoder enables the SVAE to evaluate different scale subsets without repeated training. Finally, a greedy selection approach with search scale K is proposed to find the suboptimal subset. This procedure not only reduces time consumption, but also ensures the performance of the subset. The selected features are analyzed on four real PolSAR datasets according to the terrain scattering characteristics. Furthermore, these features have achieved competitive performance on three PolSAR image tasks.

International audience; In recent years, enormous research has been made to improve the classification performance of single-modal remote sensing (RS) data. However, with the ever-growing availability of RS data acquired from satellite or airborne platforms, simultaneous processing and analysis of multimodal RS data pose a new challenge to researchers in the RS community. To this end, we propose a deep-learning-based new framework for multimodal RS data classification, where convolutional neural networks (CNNs) are taken as a backbone with an advanced cross-channel reconstruction module, called CCR-Net. As the name suggests, CCR-Net learns more compact fusion representations of different RS data sources by the means of the reconstruction strategy across modalities that can mutually exchange information in a more effective way. Extensive experiments conducted on two multimodal RS datasets, including hyperspectral (HS) and light detection and ranging (LiDAR) data, i.e., the Houston2013 dataset, and HS and synthetic aperture radar (SAR) data, i.e., the Berlin dataset, demonstrate the effectiveness and superiority of the proposed CCR-Net in comparison with several state-of-the-art multimodal RS data classification methods. The codes will be openly and freely available at https://github.com/danfenghong/IEEE_TGRS_CCR-Net for the sake of reproducibility.

International audience; In recent years, deep learning (DL) has attracted increasing attention in hyperspectral unmixing (HU) applications due to its powerful learning and data fitting ability. The autoencoder (AE) framework, as an unmixing baseline network, achieves good performance in HU by automatically learning low-dimensional embeddings and reconstructing data. Nevertheless, the conventional AE-based architecture, which focuses more on the pixel-level reconstruction loss, tends to lose some significant detailed information of certain materials (e.g., material-related properties) in the reconstruction process. Therefore, inspired by the perception mechanism, we propose a cycle-consistency unmixing network, called CyCU-Net, by learning two cascaded AEs in an end-to-end fashion, to enhance the unmixing performance more effectively. CyCU-Net is capable of reducing the detailed and material-related information loss in the process of reconstruction by relaxing the original pixel-level reconstruction assumption to cycle consistency dominated by the cascaded AEs. More specifically, cycle consistency can be achieved by a newly proposed self-perception loss, which consists of two spectral reconstruction terms and one abundance reconstruction term. By taking advantage of the self-perception loss in the network, the high-level semantic information can be well preserved in the unmixing process. Moreover, we investigate the performance gain of CyCU-Net with extensive ablation studies. Experimental results on one synthetic and three real hyperspectral data sets demonstrate the effectiveness and competitiveness of the proposed CyCU-Net in comparison with several state-of-the-art unmixing algorithms.

International audience; A hyperspectral image sequence can be obtained at different time in the same region from a hyperspectral sensor. The environmental change usually leads to variation in endmember reflectance, which has an important influence on unmixing process. In this article, a Bayesian unmixing model considering spectral variability for hyperspectral sequence is proposed, in which composite prior distributions of abundance and endmember variability are developed. The abundance priors consider the continuity of abundance in the temporal and spatial domains, simultaneously. Specifically, in the spatial domain, a data-adaptive variance of the abundance prior distribution is put forward based on local spatial difference. Moreover, the priors of endmember variability in temporal continuity and spectral smoothness are also exploited. Finally, a joint posterior distribution is obtained by the likelihood function and the parameter prior distributions, which can be calculated by the Markov chain Monte Carlo (MCMC) algorithm. Experiments on synthetic and real data sets demonstrate the effectiveness of the proposed approach in terms of abundance, endmember, and its variability estimation accuracy.

International audience; Recently, deep learning-based methodologies have attained unprecedented performance in hyperspectral (HS) pansharpening, which aims to improve the spatial quality of HS images (HSIs) by making use of details extracted from the high-resolution panchromatic (HR-PAN) image. However, it remains challenging to incorporate the details into the pansharpened image effectively, while alleviating the spectral distortion simultaneously. To tackle this problem, in this article, we propose an adaptive feature modulation-based detail injection network (AFM-DIN) for HS pansharpening, which mainly consists of four phases: high-frequency details generation of the HR-PAN image, multiscale feature extraction of the upsampled HSI, AFM-based detail injection, and reconstruction of the HR-HSI. First, a novel octave convolution unit is employed to decompose the HR-PAN image into high and low frequencies, and then merge the high-frequency features together to generate the comprehensive PAN-details. Second, the spatial and spectral separable 3-D convolution units with multiple kernel sizes are designed to extract multiscale features of the upsampled HSI in a computationally efficient manner. Subsequently, by taking the critical PAN-details as prior, the proposed AFM module is able to not only incorporate the detail information effectively, but also adjust the injected details adaptively to ensure the spectral fidelity. Finally, the anticipated HR-HSI is obtained through adding the upsampled HSI to the predicted HSI-details reconstructed from informative modulated features. Extensive comparison experiments with several state-of-the-arts conducted on simulated and real HS data sets demonstrate that our proposed AFM-DIN can achieve superior pansharpening accuracy in both spatial and spectral aspects.

International audience; Traditional hyperspectral (HS) pansharpening aims at fusing a HS image with its panchromatic (PAN) counterpart, to bring the spatial resolution of the HS image to that of the PAN image. However, in many practical applications, arbitrary resolution HS (ARHS) pansharpening is required, where the HS and PAN images need to be integrated to generate a pansharpened HS image with arbitrary resolution (usually higher than that of the PAN image). Such an innovative task brings forth new challenges for the pansharpening technique, mainly including how to reconstruct HS images beyond the training scale and how to guarantee spectral fidelity at any spatial resolutions. To tackle the challenges, we present a novel convolutional neural network (CNN)-based method for ARHS pansharpening called ARHS-CNN. It is based on a two-step relay optimization process, which is associated with a multilevel enhancement subnetwork and a rescaling subnetwork. With a careful design following the thread, our ARHS-CNN is able to pansharpen HS images to any spatial resolutions using just a single CNN model trained on a limited number of scales while meantime to keep spectral fidelity at those resolutions, which wins an obvious advantage over traditional pansharpening methods. Experimental results obtained on several datasets verify the excellent performance of our ARHS-CNN method.

International audience; Although the performance of pansharpening has been significantly improved by advanced deep-learning (DL) technologies in recent years, most DL-based methods fail to process multispectral (MS) images with arbitrary numbers of bands by a single model. Consequently, it is inevitable to train separate models for MS images with different numbers of bands, which is time- and storage-consuming as well as inefficient in practice. To tackle the above problem, we propose a bidirectional recurrent pansharpening network (named ArbRPN) for MS images with arbitrary numbers of bands. Our ArbRPN can dynamically reconstruct high-resolution (HR) MS images with different numbers of bands by adaptively changing the number of recurrence to the number of bands of the low-resolution (LR) MS images. Leveraging on the ability of the ArbRPN to process MS images with any number of bands, one can even customize the bands to be pansharpened. Moreover, to achieve superior performance, spectral discrepancy and dependence are considered in the ArbRPN. Details from the panchromatic (PAN) image are adaptively injected into the fused product according to the captured spectral dependence. Furthermore, training strategies of existing DL-based pansharpening methods can only group MS images with a constant number of bands into mini-batches. Therefore, we present a mask-based training method (called mask-training) to solve this problem. Benefiting from the mask-training, our ArbRPN can achieve superior performance and robustness during pansharpening. Extensive experiments show the superior performance of our ArbRPN with respect to the state-of-the-art (SOTA) methods applied to MS images with different numbers of bands. The code of our ArbRPN is available on https://github.com/Lihui-Chen/ArbRPN.git.

Direct imaging is an active research topic in astronomy for the detection and the characterization of young sub-stellar objects. The very high contrast between the host star and its companions makes the observations particularly challenging. In this context, post-processing methods combining several images recorded with the pupil tracking mode of telescope are needed. In previous works, we have presented a data-driven algorithm, PACO, capturing locally the spatial correlations of the data with a multi-variate Gaussian model. PACO delivers better detection sensitivity and confidence than the standard post-processing methods of the field. However, there is room for improvement due to the approximate fidelity of the PACO statistical model to the time evolving observations. In this paper, we propose to combine the statistical model of PACO with supervised deep learning. The data are first pre-processed with the PACO framework to improve the stationarity and the contrast. A convolutional neural network (CNN) is then trained in a supervised fashion to detect the residual signature of synthetic sources. Finally, the trained network delivers a detection map. The photometry of detected sources is estimated by a second CNN. We apply the proposed approach to several datasets from the VLT/SPHERE instrument. Our results show that its detection stage performs significantly better than baseline methods (cADI, PCA), and leads to a contrast improvement up to half a magnitude compared to PACO. The characterization stage of the proposed method performs on average on par with or better than the comparative algorithms (PCA, PACO) for angular separation above 0.5''., Submitted to Monthly Notices of the Royal Astronomical Society

Developing agents that can execute multiple skills by learning from pre-collected datasets is an important problem in robotics, where online interaction with the environment is extremely time-consuming. Moreover, manually designing reward functions for every single desired skill is prohibitive. Prior works targeted these challenges by learning goal-conditioned policies from offline datasets without manually specified rewards, through hindsight relabelling. These methods suffer from the issue of sparsity of rewards, and fail at long-horizon tasks. In this work, we propose a novel self-supervised learning phase on the pre-collected dataset to understand the structure and the dynamics of the model, and shape a dense reward function for learning policies offline. We evaluate our method on three continuous control tasks, and show that our model significantly outperforms existing approaches, especially on tasks that involve long-term planning., Comment: Code: https://github.com/facebookresearch/go-fresh

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. In the decentralized setting, in which workers communicate over a sparse graph, current theory fails to capture important aspects of real-world behavior. First, the `spectral gap' of the communication graph is not predictive of its empirical performance in (deep) learning. Second, current theory does not explain that collaboration enables larger learning rates than training alone. In fact, it prescribes smaller learning rates, which further decrease as graphs become larger, failing to explain convergence dynamics in infinite graphs. This paper aims to paint an accurate picture of sparsely-connected distributed optimization. We quantify how the graph topology influences convergence in a quadratic toy problem and provide theoretical results for general smooth and (strongly) convex objectives. Our theory matches empirical observations in deep learning, and accurately describes the relative merits of different graph topologies. This paper is an extension of the conference paper by Vogels et. al. (2022). Code: https://github.com/epfml/topology-in-decentralized-learning., Comment: Extended version of the other paper (with the same name), that includes (among other things) theory for the heterogeneous case. arXiv admin note: substantial text overlap with arXiv:2206.03093

Vision Transformers (ViTs) have become a dominant paradigm for visual representation learning with self-attention operators. Although these operators provide flexibility to the model with their adjustable attention kernels, they suffer from inherent limitations: (1) the attention kernel is not discriminative enough, resulting in high redundancy of the ViT layers, and (2) the complexity in computation and memory is quadratic in the sequence length. In this paper, we propose a novel attention operator, called lightweight structure-aware attention (LiSA), which has a better representation power with log-linear complexity. Our operator learns structural patterns by using a set of relative position embeddings (RPEs). To achieve log-linear complexity, the RPEs are approximated with fast Fourier transforms. Our experiments and ablation studies demonstrate that ViTs based on the proposed operator outperform self-attention and other existing operators, achieving state-of-the-art results on ImageNet, and competitive results on other visual understanding benchmarks such as COCO and Something-Something-V2. The source code of our approach will be released online., Comment: 8 pages, 5 figures

Recent image generation models such as Stable Diffusion have exhibited an impressive ability to generate fairly realistic images starting from a simple text prompt. Could such models render real images obsolete for training image prediction models? In this paper, we answer part of this provocative question by investigating the need for real images when training models for ImageNet classification. Provided only with the class names that have been used to build the dataset, we explore the ability of Stable Diffusion to generate synthetic clones of ImageNet and measure how useful these are for training classification models from scratch. We show that with minimal and class-agnostic prompt engineering, ImageNet clones are able to close a large part of the gap between models produced by synthetic images and models trained with real images, for the several standard classification benchmarks that we consider in this study. More importantly, we show that models trained on synthetic images exhibit strong generalization properties and perform on par with models trained on real data for transfer. Project page: https://europe.naverlabs.com/imagenet-sd/, Accepted to CVPR 2023

International audience; Recent works in autonomous driving have widely adopted the bird'seye-view (BEV) semantic map as an intermediate representation of the world. Online prediction of these BEV maps involves non-trivial operations such as multi-camera data extraction as well as fusion and projection into a common topview grid. This is usually done with error-prone geometric operations (e.g., homography or back-projection from monocular depth estimation) or expensive direct dense mapping between image pixels and pixels in BEV (e.g., with MLP or attention). In this work, we present 'LaRa', an efficient encoder-decoder, transformer-based model for vehicle semantic segmentation from multiple cameras. Our approach uses a system of cross-attention to aggregate information over multiple sensors into a compact, yet rich, collection of latent representations. These latent representations, after being processed by a series of selfattention blocks, are then reprojected with a second cross-attention in the BEV space. We demonstrate that our model outperforms the best previous works using transformers on nuScenes. The code and trained models are available at https://github.com/valeoai/LaRa.

Despite significant advances in machine learning in recent years, robotic control learned from data has yet to show large-scale impact in the real world. One of the main limitations is access to data -- especially when coupled with the complexity of high-dimensional and underactuated control problems. Unlike in domains such as image classification or machine translation, explicit training examples cannot be easily sourced and annotated on the internet but data collection is bounded by real-time robot operation.This thesis presents several ways to leverage external data sources, from task demonstrations to full-length tutorial videos, to address the challenge of slow data collection and thus accelerate learning of robotic manipulation tasks. First, we propose a method to efficiently leverage a small number of demonstrations as a starting point, and autonomously improve this initial policy through residual reinforcement learning. No reward shaping, controller engineering or state estimation is needed as the policy uses image and proprioceptive inputs as well as sparse task completion rewards only.In our second contribution, we show that robotic agents can acquire inductive biases for manipulation by watching videos of humans using their hands and arms before ever interacting with the world themselves. We demonstrate that our reward functions, though trained exclusively on human data, are able to generalize their predictions of task progress to robot arms and accelerate training of several unseen manipulation tasks.Finally, we propose that narrated instruction videos can not only help agents gain subtask execution skills, but also teach them which subtasks are needed to accomplish long-horizon goals and in which order, as well as how they map to natural language instructions. Specifically, we present a discriminative clustering based method leveraging the temporal alignment of the narration and visual streams for automatic subtask discovery and segmentation. Both short and long-form instructional videos are especially promising data sources as they are widely available on the internet.In this thesis, we investigate the performance of purely learning-based algorithms for robotic manipulation, while acknowledging that optimal control as well as hybrid approaches can provide complementary solutions to some of the open challenges. Our key argument is that advances in the related fields of computer vision, signal processing, natural language processing, imitation and deep reinforcement learning can help lead the way towards more adaptive robotic agents. In manipulation domains, in particular, the variety of materials, shapes and tasks present in the real world beyond tightly controlled operating conditions poses great difficulty for fixed control strategies and the precise physical modelling required by classical model-predictive control approaches. Our overarching goal is therefore to enable more capable and versatile robotic manipulation through data-driven methods. Reducing the amount of domain expertise required to train robots by emphasizing example-based learning and autonomous improvement will ultimately support more widespread adoption of adaptive robotic solutions.; Malgré des progrès considérables réalisés ces dernières années dans l'apprentissage automatique, son utilisation dans le cadre de la commande de robots n'a pas encore eu d'impact à grande échelle. L'une des principales limitations est l'accès aux données, surtout si l'on tient compte de la complexité des problèmes de commande en haute dimension et pour des systèmes sous-actionnés. Contrairement au cas des domaines tels que la classification d'images ou la traduction automatique, il est difficile de trouver des exemples d'entraînement annotés sur Internet, et la collecte de données dans des environnements physiques est limitée par le fonctionnement du robot.Cette thèse présente plusieurs façons d'exploiter des sources de données externes, de démonstrations de tâches aux tutoriels vidéo, pour relever le défi de la lenteur de la collecte de données et ainsi accélérer l'apprentissage des tâches de manipulation robotique. Nous proposons d'abord une méthode pour exploiter efficacement un petit nombre de démonstrations comme point de départ, et améliorer de manière automatique la politique initiale par un apprentissage par renforcement résiduel. Des étapes d'affinement de la fonction de récompense, des contrôleurs ou encore d'estimation d'état ne sont pas nécessaires car la politique utilise uniquement des entrées d'image et proprioceptives ainsi que des récompenses binaires obtenues en fonction de la réussite de la tâche désirée.Dans notre deuxième contribution, nous montrons que les agents robotiques peuvent acquérir des biais inductifs pour la manipulation avant d'interagir eux-mêmes avec le monde physique, en regardant des vidéos de personnes utilisant leurs mains et leurs bras. Nous démontrons que nos fonctions de récompense, bien qu'entraînées exclusivement sur des données humaines, sont capables de généraliser leurs prédictions de progression de tâche aux bras robotiques et d'accélérer l'entraînement de plusieurs tâches de manipulation.Enfin, nous proposons d'utiliser des tutoriels vidéo pour enseigner aux agents des compétences en matière d'exécution de sous-tâches, quelles sont les sous-tâches nécessaires pour accomplir une tâche à long terme, leur ordre d'exécution, et comment elles correspondent aux instructions en langage naturel. Plus précisément, nous présentons une méthode basée sur le clustering discriminant qui exploite l'alignement temporel de la narration et du flux visuel pour la découverte et la segmentation des sous-tâches. Les tutoriels vidéo, qu'ils soient courts ou longs, sont des sources de données particulièrement prometteuses car ils sont abondants sur Internet.Dans cette thèse, nous étudions la performance des algorithmes purement basés sur l'apprentissage pour la manipulation robotique, tout en reconnaissant que la commande optimale ainsi que les approches hybrides peuvent fournir des solutions complémentaires à certains des défis ouverts. Notre argument central est que les avancées dans les domaines connexes de la vision par ordinateur, du traitement du signal, du traitement du langage naturel, des l'apprentissage par imitation et par renforcement profond peuvent aider à ouvrir la voie à des agents robotiques plus adaptatifs. C'est particulièrement le cas pour le domaine de la manipulation dans le monde réel, en dehors de conditions d'exploitation étroitement contrôlées. En effet, la variété des matériaux, des formes et des tâches pose de grandes difficultés pour les stratégies de contrôle fixes et les approches classiques de commande prédictive qui nécessitent une modélisation physique précise. Notre objectif principal est donc de permettre une manipulation robotique plus performante et polyvalente grâce à des méthodes appris à partir des données. Réduire l'expertise liée au domaine nécessaire pour former les robots, en mettant l'accent sur l'apprentissage à partir d'exemples et l'amélioration autonome, favorisera en fin de compte l'adoption plus large de solutions robotiques adaptatives.

Despite significant advances, the performance of state-of-the-art continual learning approaches hinges on the unrealistic scenario of fully labeled data. In this paper, we tackle this challenge and propose an approach for continual semi-supervised learning--a setting where not all the data samples are labeled. A primary issue in this scenario is the model forgetting representations of unlabeled data and overfitting the labeled samples. We leverage the power of nearest-neighbor classifiers to nonlinearly partition the feature space and flexibly model the underlying data distribution thanks to its non-parametric nature. This enables the model to learn a strong representation for the current task, and distill relevant information from previous tasks. We perform a thorough experimental evaluation and show that our method outperforms all the existing approaches by large margins, setting a solid state of the art on the continual semi-supervised learning paradigm. For example, on CIFAR-100 we surpass several others even when using at least 30 times less supervision (0.8% vs. 25% of annotations). Finally, our method works well on both low and high resolution images and scales seamlessly to more complex datasets such as ImageNet-100. The code is publicly available on https://github.com/kangzhiq/NNCSL, 13 pages

We propose Continual Repeated Annealed Flow Transport Monte Carlo (CRAFT), a method that combines a sequential Monte Carlo (SMC) sampler (itself a generalization of Annealed Importance Sampling) with variational inference using normalizing flows. The normalizing flows are directly trained to transport between annealing temperatures using a KL divergence for each transition. This optimization objective is itself estimated using the normalizing flow/SMC approximation. We show conceptually and using multiple empirical examples that CRAFT improves on Annealed Flow Transport Monte Carlo (Arbel et al., 2021), on which it builds and also on Markov chain Monte Carlo (MCMC) based Stochastic Normalizing Flows (Wu et al., 2020). By incorporating CRAFT within particle MCMC, we show that such learnt samplers can achieve impressively accurate results on a challenging lattice field theory example., Comment: 21 pages, 6 figures Published at International Conference on Machine Learning (ICML) 2022

International audience; Despite the progress seen in classification methods, current approaches for handling videos with distribution shifts in source and target domains remain source-dependent as they require access to the source data during the adaptation stage. In this paper, we present a self-training based source-free video domain adaptation approach (without bells and whistles) to address this challenge by bridging the gap between the source and the target domains. We use the source pre-trained model to generate pseudo-labels for the target domain samples, which are inevitably noisy. We treat the problem of source-free video domain adaptation as learning from noisy labels and argue that the samples with correct pseudo-labels can help in the adaptation stage. To this end, we leverage the cross-entropy loss as an indicator of the correctness of pseudo-labels, and use the resulting small-loss samples from the target domain for fine-tuning the model. Extensive experimental evaluations show that our method termed as CleanAdapt achieves ∼ 7% gain over the source-only model and outperforms the state-of-the-art approaches on various open datasets.

Year after year advances in deep learning allow to solve a rapidly increasing range of challenging tasks, as well as to set new, even more ambitious goals. Such a success, however, comes at a price of increasing requirements for all aspects of learning: large-scale models, which tend to perform best, require large quantities of data, memory, computational resources and time to be properly trained. This cannot always be achieved in practice, especially on smaller datasets, which promotes exploration of the ways to transfer knowledge, i.e. re-purposing existing state-of-the-art models to solving new tasks.This problem of transferring knowledge between tasks comes with its own challenges caused by different factors, such as the type of knowledge which needs to be transferred, or availability of data. In this thesis we focus on two setups from this category: few-shot learning and multi-domain learning. Both problems share the motivation to learn a model that would be able to generalise to solving the same type of task, e.g. image classification, on a number of different domains.Our first contribution explores probabilistic modeling for few-shot classification, where the model aims to solve a wide range of classification tasks, each accompanied with a handful of labeled examples. Limited supervision leads to high uncertainty about the predictions, which can be naturally tackled by probabilistic framework. We treat the task-specific classifier as a latent variable, and propose a novel amortised variational inference scheme which uses a single network to predict parameters of the distribution both for the prior and for the approximated posterior of the latent variable in the considered graphical model. The prior is conditioned on the support set of the task, while the approximated posterior is conditioned on the union of the support and query sets. Minimisation of the distance between these two distributions provides additional guidance from the support set during training, allowing us to exploit the disparity between the two sets of data. We evaluate our model on several few-shot classification benchmarks, and show that it can achieve competitive results on all of them. We also demonstrate the benefits of modeling uncertainty by showing that a sampled ensemble of classifiers slightly improves the performance compared to the inferred classifier mean. This result that cannot be achieved by models relying on Monte Carlo approximations, which, according to our experiments, tend to underestimate the true variance.Our second contribution proposes a novel type of adaptation modules for multi-domain classification, which considers a fixed set of classification tasks on a limited number of domains. We adopt the common approach of using a pre-trained feature extractor as a base network, and adjusting it to novel domains using domain-specific adapters applied to each convolutional layer in the base. For each output channel, our modulation adapter separately scales each kernel in the convolutional weight tensor with its own output-specific scalar. This results in a set of independent re-weightings of the input feature maps, which makes the resulting feature adaptation more flexible compared to previous approaches. To reduce the parameter budget, we factorise our modulation adapter as a product of the two smaller matrices. We evaluate our model one the two common multi-domain classification benchmarks, and show that both the full and the factorised versions achieve state-of-the-art results. In addition to that, we provide results of our model on a span of parameter budgets, which is one of the advantages of our approach. For each parameter budget, modulation adapters outperform the competitors which generally offer only a single budget setting.; Année après année, le progrès de l’apprentissage profond permet de résoudre un nombre croissant de tâches difficiles, ainsi que de se fixer de nouveaux objectifs encore plus ambitieux. Un tel succès, cependant, se fait au prix d’exigences croissantes pour tous les aspects de l’apprentissage : les modèles à grande échelle, qui ont tendance à être les plus performants, nécessitent de grandes quantités de données, de mémoire, de ressources de calcul et de temps d’entraînement. Ça peut poser problème en pratique, en particulier sur des petits ensembles de données. Cela motive l’exploration des moyens de transférer les connaissances, c’est-à-dire la valorisation des modèles existants pour résoudre de nouvelles tâches.Ce problème de transfert de connaissances a ses propres défis, tels que le type de connaissances à transférer ou la disponibilité des données. Dans cette thèse, nous nous concentrons sur deux configurations de cette catégorie : l’apprentissage avec très peu de données et l’apprentissage multi-domaines. Les deux problèmes partagent la motivation d’apprendre un modèle qui pourrait se généraliser à la résolution du même type de tâche, par ex. classification d’images, sur un certain nombre de domaines différents.Notre première contribution explore la modélisation probabiliste pour l’apprentissage avec très peu de données, où le modèle vise à résoudre un large éventail de tâches de classification, chacune accompagnée d’une poignée d’exemples étiquetés. Une supervision limitée conduit à une forte incertitude sur les prédictions, qui peut naturellement être résolue par un cadre probabiliste. Nous traitons le classificateur spécifique à une tâche comme une variable latente et proposons un nouveau schéma d’inférence variationnelle amortie qui utilise un seul réseau pour prédire les paramètres de la distribution à la fois pour le prior et pour le postérieur approximé de la variable latente dans le modèle graphique considéré. Nous évaluons notre modèle sur plusieurs benchmarks de classification, et montrons qu’il peut obtenir des résultats compétitifs sur chacun d’eux. Nous démontrons également les avantages de la modélisation de l’incertitude en montrant qu’un ensemble échantillonné de classificateurs améliore légèrement les performances par rapport à la moyenne déduite du classificateur. Ce résultat ne peut être atteint par des modèles reposant sur des approximations de Monte Carlo, qui, selon nos expérimentations, ont tendance à sous-estimer la vraie variance.Notre deuxième contribution propose un nouveau type de modules d’adaptation pour la classification multi-domaine, qui considère un ensemble fixe de tâches de classification sur un nombre limité de domaines. Nous adoptons l’approche commune consistant à utiliser un extracteur de caractéristiques pré-formé comme réseau de base et à l’adapter à de nouveaux domaines à l’aide d’adaptateurs spécifiques au domaine appliqués à chaque couche convolutive de la base. Pour chaque canal de sortie, notre adaptateur de modulation multiplie chaque noyau dans le tenseur de poids convolutif avec son propre scalaire spécifique à la sortie. Cela se traduit par un ensemble de pondérations indépendantes des cartes de caractéristiques d’entrée, ce qui rend l’adaptation des caractéristiques résultante plus flexible par rapport aux approches précédentes. Pour réduire le budget des paramètres, nous factorisons notre adaptateur de modulation comme un produit des deux plus petites matrices. Nous évaluons notre modèle sur les deux benchmarks et montrons que les versions complètes et factorisées obtiennent des résultats de pointe. En plus de cela, nous fournissons les résultats de notre modèle sur une gamme de budgets de paramètres, ce qui est l’un des avantages de notre approche. Pour chaque budget de paramètre, les adaptateurs de modulation surpassent les concurrents qui n’offrent généralement qu’un seul réglage de budget de paramètre.

International audience; Bilevel optimization problems involve two nested objectives, where an upper-level objective depends on a solution to a lower-level problem. When the latter is nonconvex, multiple critical points may be present, leading to an ambiguous definition of the problem. In this paper, we introduce a key ingredient for resolving this ambiguity through the concept of a selection map which allows one to choose a particular solution to the lower-level problem. Using such maps, we define a class of hierarchical games between two agents that resolve the ambiguity in bilevel problems. This new class of games requires introducing new analytical tools in Morse theory to extend implicit differentiation, a technique used in bilevel optimization resulting from the implicit function theorem. In particular, we establish the validity of such a method even when the latter theorem is inapplicable due to degenerate critical points. Finally, we show that algorithms for solving bilevel problems based on unrolled optimization solve these games up to approximation errors due to finite computational power. A simple correction to these algorithms is then proposed for removing these errors.

We introduce two synthetic likelihood methods for Simulation-Based Inference (SBI), to conduct either amortized or targeted inference from experimental observations when a high-fidelity simulator is available. Both methods learn a conditional energy-based model (EBM) of the likelihood using synthetic data generated by the simulator, conditioned on parameters drawn from a proposal distribution. The learned likelihood can then be combined with any prior to obtain a posterior estimate, from which samples can be drawn using MCMC. Our methods uniquely combine a flexible Energy-Based Model and the minimization of a KL loss: this is in contrast to other synthetic likelihood methods, which either rely on normalizing flows, or minimize score-based objectives; choices that come with known pitfalls. Our first method, Amortized Unnormalized Neural Likelihood Estimation (AUNLE), introduces a tilting trick during training that allows to significantly lower the computational cost of inference by enabling the use of efficient MCMC techniques. Our second method, Sequential UNLE (SUNLE), employs a robust doubly intractable approach in order to re-use simulation data and improve posterior accuracy on a specific dataset. We demonstrate the properties of both methods on a range of synthetic datasets, and apply them to a neuroscience model of the pyloric network in the crab Cancer Borealis, matching the performance of other synthetic likelihood methods at a fraction of the simulation budget.

In this work, we address the problem of image-goal navigation in the context of visually-realistic 3D environments. This task involves navigating to a location indicated by a target image in a previously unseen environment. Earlier attempts, including RL-based and SLAM-based approaches, have either shown poor generalization performance, or are heavily-reliant on pose/depth sensors. We present a novel method that leverages a cross-episode memory to learn to navigate. We first train a state-embedding network in a self-supervised fashion, and then use it to embed previously-visited states into a memory. In order to avoid overfitting, we propose to use data augmentation on the RGB input during training. We validate our approach through extensive evaluations, showing that our data-augmented memory-based model establishes a new state of the art on the image-goal navigation task in the challenging Gibson dataset. We obtain this competitive performance from RGB input only, without access to additional sensors such as position or depth.

With the ever-increasing consumption of audio-visual media on the internet, video understanding has become an important problem in order to provide users with the right content. Compared to more traditional media, the video signal is inherently multi-modal, its information is scattered across several modalities such as speech, audio or vision. This thesis aims at designing and training deep learning models that exploit those different modalities to automatically understand video content. While the community has developed high performance deep-learning models for uni-modal problems, processing multi-modal inputs such as video has received less attention. In the first part of this thesis, we introduce a transformer-based architecture that leverages pre-extracted uni-modal features obtained at different moments in the video and fuses them into a single video representation. Taking advantage of the attention mechanism, we show that our model processes the video signal across both modalities and time. We couple our video encoder with a caption encoder into a cross-modal architecture and obtain state-of-the art results for the text-to-video retrieval task on three datasets.Training such a model unfortunately requires a large amount of manually captioned videos. In order to leverage the billions of unlabelled videos on the internet, the common approach has been to use the accompanying speech as supervision to pretrain a video encoder on the other modalities. While the resulting encoder is capable of processing visual information and sound, it has not been trained to attend the spoken language in videos. In the second part of this thesis, we propose a method to pretrain a multi-modal encoder on all video modalities, including speech. At each training batch, we completely mask out a different modality from the video encoder input and use it as supervision. We finetune our model on the task of text-to-video retrieval and show that our approach is particularly suitable for datasets where user queries relate to the spoken content in the video.Extracting spoken language from the audio signal can be challenging, particularly when the audio is noisy. In the case of audio-visual media, the visual modality can provide precious cues to better extract speech from videos. In the third part of this thesis we introduce an encoder-decoder architecture as well as a pretraining strategy to not only train a speech recognition model on the audio signal but also on the visual signal. We evaluate the contribution of our approach on a challenging test set which demonstrates the contribution of the visual modality under challenging audio conditions.; Avec la consommation toujours croissante de contenu audiovisuel sur Internet, la compréhension automatique de vidéos est devenue un problème important afin de proposer aux utilisateurs le contenu qui correspond à leurs attentes. Comparé aux médias plus traditionnels, le signal vidéo est intrinsèquement multimodal, ses informations sont réparties entre plusieurs modalités telles que la parole, l'audio ou la vision. Cette thèse vise à concevoir et entraîner des modèles d'apprentissage profond capables d'exploiter ces différentes modalités pour comprendre automatiquement le contenu vidéo.Bien que des modèles d'apprentissage particulièrement performants aient été développés pour les problèmes unimodaux, le traitement des données multimodales telles que la vidéo a reçu comparativement moins d'attention. Dans la première partie de ce manuscrit, nous introduisons une architecture basée sur un transformer qui exploite des descripteurs unimodaux pré-extraits à différents moments de la vidéo et les fusionne en une unique représentation. Bénéficiant du mécanisme d'attention, nous montrons que notre modèle est capable de traiter le signal vidéo à travers ses différentes modalités ainsi que temporellement. Nous couplons notre encodeur vidéo avec un encodeur de texte dans une architecture intermodale et établissons un nouvel état de l'art pour la tâche de recherche texte-vidéo sur trois jeux de données.L'entraînement d'un tel modèle nécessite cependant une grande quantité de vidéos manuellement annotées. Afin de tirer parti des milliards de vidéos non annotées disponibles sur Internet, l'approche courante a été d'utiliser les mots prononcés dans la séquence comme supervision pour pré-entraîner un encodeur vidéo sur les autres modalités. Bien que l'encodeur résultant soit capable de traiter les informations visuelles et sonores, il n'a pas été entraîné à exploiter le discours oral des vidéos. Dans la deuxième partie de ce manuscrit, nous proposons une méthode pour pré-entraîner un encodeur multimodal sur toutes les modalités vidéo, y compris la parole. A chaque étape d'entraînement, nous masquons entièrement une modalité différente de l'entrée de l'encodeur et l'utilisons comme supervision. Nous affinons notre modèle sur la tâche de recherche de vidéos et montrons que notre approche est particulièrement adaptée aux jeux de données où les requêtes des utilisateurs concernent le discours prononcé dans la vidéo.Extraire la transcription du signal audio peut être difficile, en particulier lorsqu'il est bruité. Dans le cas d'un contenu audiovisuel, la modalité visuelle peut fournir des indices précieux pour mieux extraire la parole des vidéos. Dans la troisième partie de ce manuscrit, nous introduisons une architecture encodeur-décodeur ainsi qu'une stratégie de pré-apprentissage pour entraîner un modèle de reconnaissance vocale non seulement sur le signal audio, mais également sur le signal visuel. Nous évaluons la contribution de notre approche sur un jeu de données de test qui démontre la contribution de la modalité visuelle pour la reconnaissance de la parole dans des conditions audio difficiles.

Semantic segmentation is an essential part of deep learning. In recent years, with the development of remote sensing big data, semantic segmentation has been increasingly used in remote sensing. Deep convolutional neural networks (DCNNs) face the challenge of feature fusion: very-high-resolution remote sensing image multisource data fusion can increase the network's learnable information, which is conducive to correctly classifying target objects by DCNNs; simultaneously, the fusion of high-level abstract features and low-level spatial features can improve the classification accuracy at the border between target objects. In this paper, we propose a multipath encoder structure to extract features of multipath inputs, a multipath attention-fused block module to fuse multipath features, and a refinement attention-fused block module to fuse high-level abstract features and low-level spatial features. Furthermore, we propose a novel convolutional neural network architecture, named attention-fused network (AFNet). Based on our AFNet, we achieve state-of-the-art performance with an overall accuracy of 91.7% and a mean F1 score of 90.96% on the ISPRS Vaihingen 2D dataset and an overall accuracy of 92.1% and a mean F1 score of 93.44% on the ISPRS Potsdam 2D dataset., Comment: 35 pages. Published by ISPRS Journal of Photogrammetry and Remote Sensing

In the problem of online portfolio selection as formulated by Cover (1991), the trader repeatedly distributes her capital over $ d $ assets in each of $ T > 1 $ rounds, with the goal of maximizing the total return. Cover proposed an algorithm, termed Universal Portfolios, that performs nearly as well as the best (in hindsight) static assignment of a portfolio, with an $ O(d\log(T)) $ regret in terms of the logarithmic return. Without imposing any restrictions on the market this guarantee is known to be worst-case optimal, and no other algorithm attaining it has been discovered so far. Unfortunately, Cover's algorithm crucially relies on computing certain $ d $-dimensional integral which must be approximated in any implementation; this results in a prohibitive $ \tilde O(d^4(T+d)^{14}) $ per-round runtime for the fastest known implementation due to Kalai and Vempala (2002). We propose an algorithm for online portfolio selection that admits essentially the same regret guarantee as Universal Portfolios -- up to a constant factor and replacement of $ \log(T) $ with $ \log(T+d) $ -- yet has a drastically reduced runtime of $ \tilde O(d^2(T+d)) $ per round. The selected portfolio minimizes the current logarithmic loss regularized by the log-determinant of its Hessian -- equivalently, the hybrid logarithmic-volumetric barrier of the polytope specified by the asset return vectors. As such, our work reveals surprising connections of online portfolio selection with two classical topics in optimization theory: cutting-plane and interior-point algorithms.

In this paper, we introduce a new algorithm based on archetypal analysis for blind hyperspectral unmixing, assuming linear mixing of endmembers. Archetypal analysis is a natural formulation for this task. This method does not require the presence of pure pixels (i.e., pixels containing a single material) but instead represents endmembers as convex combinations of a few pixels present in the original hyperspectral image. Our approach leverages an entropic gradient descent strategy, which (i) provides better solutions for hyperspectral unmixing than traditional archetypal analysis algorithms, and (ii) leads to efficient GPU implementations. Since running a single instance of our algorithm is fast, we also propose an ensembling mechanism along with an appropriate model selection procedure that make our method robust to hyper-parameter choices while keeping the computational complexity reasonable. By using six standard real datasets, we show that our approach outperforms state-of-the-art matrix factorization and recent deep learning methods. We also provide an open-source PyTorch implementation: https://github.com/inria-thoth/EDAA.

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.; In this paper, we aim to improve the mathematical interpretability of convolutional neural networks for image classification. When trained on natural image datasets, such networks tend to learn parameters in the first layer that closely resemble oriented Gabor filters. By leveraging the properties of discrete Gabor-like convolutions, we prove that, under specific conditions, feature maps computed by the subsequent max pooling operator tend to approximate the modulus of complex Gabor-like coefficients, and as such, are stable with respect to certain input shifts. We then compute a probabilistic measure of shift invariance for these layers. More precisely, we show that some filters, depending on their frequency and orientation, are more likely than others to produce stable image representations. We experimentally validate our theory by considering a deterministic feature extractor based on the dual-tree wavelet packet transform, a particular case of discrete Gabor-like decomposition. We demonstrate a strong correlation between shift invariance on the one hand and similarity with complex modulus on the other hand.

International audience; Direct imaging is an active research topic in astronomy for the detection and the characterization of young sub- stellar objects. The very high contrast between the host star and its companions makes detection particularly challenging. In addition to the use of an extreme adaptive optics system and a coronagraph to strongly attenuate the starlight contamination, dedicated post processing methods combining several images recorded with the pupil tracking mode of the telescope are needed. In previous works, we have presented the PACO algorithm capturing the spatial correlations of the data with a multi-variate Gaussian model whose parameters are estimated in a data-driven fashion at the scale of a patch of a few tens of pixels. PACO is parameter free and delivers reliable detection confidences with an improved sensitivity compared to the standard methods of the field. However, there is a room for improvement in the detection sensitivity due to the approximate fidelity of the PACO statistical model with respect to the observations. We propose to combine the statistics-based model of PACO with a deep learning approach in a three- step algorithm. First, the data are centered and whitened locally using the PACO framework to improve the stationarity and the contrast in a preprocessing step. Second, a convolutional neural network is trained in a supervised fashion to detect the signature of synthetic sources in the preprocessed science data. The network is trained from scratch with a custom data augmentation strategy allowing to generate a large training set from a single spatio-temporal dataset. Finally, the trained network is applied to the preprocessed observations and delivers a detection map. We apply our method on eleven datasets from the VLT/SPHERE-IRDIS instrument and compare our method with PACO and other baselines of the field (cADI, PCA). Our results show that the proposed method performs on-par with or better than these algorithms, with a contrast improvement up to half a magnitude with respect to PACO.

International audience; Recently, the majority of successful matching approaches are based on convolutional neural networks, which focus on learning the invariant and discriminative features for individual image patches based on image content. However, the image patch matching task is essentially to predict the matching relationship of patch pairs, that is, matching (similar) or non-matching (dissimilar). Therefore, we consider that the feature relation (FR) learning is more important than individual feature learning for image patch matching problem. Motivated by this, we propose an element-wise FR learning network for image patch matching, which transforms the image patch matching task into an image relationship-based pattern classification problem and dramatically improves generalization performances on image matching. Meanwhile, the proposed element-wise learning methods encourage full interaction between feature information and can naturally learn FR. Moreover, we propose to aggregate FR from multilevels, which integrates the multiscale FR for more precise matching. Experimental results demonstrate that our proposal achieves superior performances on cross-spectral image patch matching and single spectral image patch matching, and good generalization on image patch retrieval.

International audience; We study the problem of $K$-armed dueling bandit for both stochastic and adversarial environments, where the goal of the learner is to aggregate information through relative preferences of pair of decisions points queried in an online sequential manner. We first propose a novel reduction from any (general) dueling bandits to multi-armed bandits and despite the simplicity, it allows us to improve many existing results in dueling bandits. In particular, \emph{we give the first best-of-both world result for the dueling bandits regret minimization problem} -- a unified framework that is guaranteed to perform optimally for both stochastic and adversarial preferences simultaneously. Moreover, our algorithm is also the first to achieve an optimal $O(\sum_{i = 1}^K \frac{\log T}{\Delta_i})$ regret bound against the Condorcet-winner benchmark, which scales optimally both in terms of the arm-size $K$ and the instance-specific suboptimality gaps $\{\Delta_i\}_{i = 1}^K$. This resolves the long-standing problem of designing an instancewise gap-dependent order optimal regret algorithm for dueling bandits (with matching lower bounds up to small constant factors). We further justify the robustness of our proposed algorithm by proving its optimal regret rate under adversarially corrupted preferences -- this outperforms the existing state-of-the-art corrupted dueling results by a large margin. In summary, we believe our reduction idea will find a broader scope in solving a diverse class of dueling bandits setting, which are otherwise studied separately from multi-armed bandits with often more complex solutions and worse guarantees. The efficacy of our proposed algorithms is empirically corroborated against the existing dueling bandit methods.

International audience; Standard hyperspectral (HS) pansharpening relies on fusion to enhance low-resolution HS (LRHS) images to the resolution of their matching panchromatic (PAN) images, whose practical implementation is normally under a stipulation of scale invariance of the model across the training phase and the pansharpening phase. By contrast, arbitrary resolution HS (ARHS) pansharpening seeks to pansharpen LRHS images to any user-customized resolutions. For such a new HS pansharpening task, it is not feasible to train and store convolution neural network (CNN) models for all possible candidate scales, which implies that the single model acquired from the training phase should be capable of being generalized to yield HS images with any resolutions in the pansharpening phase. To address the challenge, a novel variable subpixel convolution (VSPC)-based CNN (VSPC-CNN) method following our arbitrary upsampling CNN (AU-CNN) framework is developed for ARHS pansharpening. The VSPC-CNN method comprises a two-stage elevating thread. The first stage is to improve the spatial resolution of the input HS image to that of the PAN image through a prepansharpening module, and then, a VSPC-encapsulated arbitrary scale attention upsampling (ASAU) module is cascaded for arbitrary resolution adjustment. After training with given scales, it can be generalized to pansharpen HS image to arbitrary scales under the spatial patterns invariance across the training and pansharpening phases. Experimental results from several specific VSPC-CNNs on both simulated and real HS datasets show the superiority of the proposed method.

Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our knowledge) to the reconstruction of high-resolution, high-dynamic range color images from raw photographic bursts captured by a handheld camera with exposure bracketing. This method uses a physically-accurate model of image formation to combine an iterative optimization algorithm for solving the corresponding inverse problem with a learned image representation for robust alignment and a learned natural image prior. The proposed algorithm is fast, with low memory requirements compared to state-of-the-art learning-based approaches to image restoration, and features that are learned end to end from synthetic yet realistic data. Extensive experiments demonstrate its excellent performance with super-resolution factors of up to $\times 4$ on real photographs taken in the wild with hand-held cameras, and high robustness to low-light conditions, noise, camera shake, and moderate object motion., Accepted to Siggraph 2022 Technical Papers program

International audience; As remote sensing (RS) data obtained from different sensors become available largely and openly, multimodal data processing and analysis techniques have been garnering increasing interest in the RS and geoscience community. However, due to the gap between different modalities in terms of imaging sensors, resolutions, and contents, embedding their complementary information into a consistent, compact, accurate, and discriminative representation, to a great extent, remains challenging. To this end, we propose a shared and specific feature learning (S2FL) model. S2FL is capable of decomposing multimodal RS data into modality-shared and modality-specific components, enabling the information blending of multi-modalities more effectively, particularly for heterogeneous data sources. Moreover, to better assess multimodal baselines and the newly-proposed S2FL model, three multimodal RS benchmark datasets, i.e., Houston2013 – hyperspectral and multispectral data, Berlin – hyperspectral and synthetic aperture radar (SAR) data, Augsburg – hyperspectral, SAR, and digital surface model (DSM) data, are released and used for land cover classification. Extensive experiments conducted on the three datasets demonstrate the superiority and advancement of our S2FL model in the task of land cover classification in comparison with previously-proposed state-of-the-art baselines. Furthermore, the baseline codes and datasets used in this paper will be made available freely at https://github.com/danfenghong/ISPRS_S2FL.

International audience; This letter proposes a novel method for combining multiview features in polarimetric synthetic aperture radar (PolSAR) for land cover classification. It is well-known that feature extraction and classifier design are two significant steps in machine learning methods for PolSAR data interpretation. Each PolSAR pixel can be represented in different feature spaces, such as polarimetric data scattering, or the polarimetric target decomposition spaces. In this letter, a tensor-based multiview embedding algorithm is proposed to fuse those features from different spaces in order to obtain a distinctive set of features for the subsequent classification. Based on the pixel-based classification tasks, a modified tensor distance (MTD) is designed to accurately calculate the distance between tensors. It emphasizes the importance of the central pixel, and decreases the influence of the neighbors in the feature patch when calculating tensor distance. Furthermore, the complementary properties of different views are exploited by an MTD measured tensor multiview spectral embedding method, so as to obtain relevant low-dimensional features. Compared with state-of-the-art methods, the validation and effectiveness of the proposed method is demonstrated on two real PolSAR data sets.

International audience; Hyperspectral anomaly detection (AD) aims to detect objects significantly different from their surrounding background. Recently, many detectors based on autoencoder (AE) exhibited promising performances in hyperspectral AD tasks. However, the fundamental hypothesis of the AE-based detector that anomaly is more challenging to be reconstructed than background may not always be true in practice. We demonstrate that an AE could well reconstruct anomalies even without anomalies for training, because AE models mainly focus on the quality of sample reconstruction and do not care if the encoded features solely represent the background rather than anomalies. If more information is preserved than needed to reconstruct the background, the anomalies will be well reconstructed. This article proposes a cluster-memory augmented deep autoencoder via optimal transportation for hyperspectral anomaly detection (OTCMA) clustering for hyperspectral AD to solve this problem. The deep clustering method based on optimal transportation (OT) is proposed to enhance the features consistency of samples within the same categories and features discrimination of samples in different categories. The memory module stores the background’s consistent features, which are the cluster centers for each category background. We retrieve more consistent features from the memory module instead of reconstructing a sample utilizing its own encoded features. The network focuses more on consistent feature reconstruction by training AE with a memory module. This effectively restricts the reconstruction ability of AE and prevents reconstructing anomalies. Extensive experiments on the benchmark datasets demonstrate that our proposed OTCMA achieves state-of-the-art results. Besides, this article presents further discussions about the effectiveness of our proposed memory module and different criteria for better AD.

We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model's generalization and maintaining its performance on the original task. Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We extensively analyze supervised training using multi-scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us to control the trade-off between performance on the training task and transferability. We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks. Code and pretrained models: https://europe.naverlabs.com/t-rex, Accepted to ICLR 2023 (spotlight)

International audience; Learning a diverse set of skills by interacting with an environment without any external supervision is an important challenge. In particular, obtaining a goal-conditioned agent that can reach any given state is useful in many applications. We propose a novel method for training such a goal-conditioned agent without any external rewards or any domain knowledge. We use random walk to train a reachability network that predicts the similarity between two states. This reachability network is then used in building goal memory containing past observations that are diverse and well-balanced. Finally, we train a goal-conditioned policy network with goals sampled from the goal memory and reward it by the reachability network and the goal memory. All the components are kept updated throughout training as the agent discovers and learns new goals. We apply our method to a continuous control navigation and robotic manipulation tasks.