Search

We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing a version of our model, the dataset and an interactive demo., Comment: Website: https://ai.meta.com/sam2

Helical magnetohydrodynamic turbulence with Hall effects is ubiquitous in heliophysics and plasma physics, such as star formation and solar activities, and its intrinsic mechanisms are still not clearly explained. Direct numerical simulations reveal that when the forcing scale is comparable to the ion inertial scale, Hall effects induce remarkable cross helicity. It then suppresses the inverse cascade efficiency, leading to the accumulation of large-scale magnetic energy and helicity. The process is accompanied by the breaking of current sheets via filaments along magnetic fields. Using the Ulysses data, the numerical findings are separately confirmed. These results suggest a novel mechanism wherein small-scale Hall effects could strongly affect large-scale magnetic fields through cross helicity., Comment: 5 figures, 6 pages

In this letter, we propose a reduced-order model to bridge the particle transport mechanics and the macroscopic fluid dynamics in the highly scattered regime. A rigorous mathematical derivation and a concise physical interpretation are presented for an anisotropic-scattering transport process with arbitrary order of scattering kernel. The prediction of the theoretical model perfectly agrees with the numerical experiments. A clear picture of the diffusion physics is revealed for the neutral particle transport in the asymptotic optically thick regime.

Recently, the development of pre-trained vision language foundation models (VLFMs) has led to remarkable performance in many tasks. However, these models tend to have strong single-image understanding capability but lack the ability to understand multiple images. Therefore, they cannot be directly applied to cope with image change understanding (ICU), which requires models to capture actual changes between multiple images and describe them in language. In this paper, we discover that existing VLFMs perform poorly when applied directly to ICU because of the following problems: (1) VLFMs generally learn the global representation of a single image, while ICU requires capturing nuances between multiple images. (2) The ICU performance of VLFMs is significantly affected by viewpoint variations, which is caused by the altered relationships between objects when viewpoint changes. To address these problems, we propose a Viewpoint Integration and Registration method. Concretely, we introduce a fused adapter image encoder that fine-tunes pre-trained encoders by inserting designed trainable adapters and fused adapters, to effectively capture nuances between image pairs. Additionally, a viewpoint registration flow and a semantic emphasizing module are designed to reduce the performance degradation caused by viewpoint variations in the visual and semantic space, respectively. Experimental results on CLEVR-Change and Spot-the-Diff demonstrate that our method achieves state-of-the-art performance in all metrics.

The emergence of large-scale large language models, with GPT-4 as a prominent example, has significantly propelled the rapid advancement of artificial general intelligence and sparked the revolution of Artificial Intelligence 2.0. In the realm of remote sensing (RS), there is a growing interest in developing large vision language models (VLMs) specifically tailored for data analysis in this domain. However, current research predominantly revolves around visual recognition tasks, lacking comprehensive, large-scale image-text datasets that are aligned and suitable for training large VLMs, which poses significant challenges to effectively training such models for RS applications. In computer vision, recent research has demonstrated that fine-tuning large vision language models on small-scale, high-quality datasets can yield impressive performance in visual and language understanding. These results are comparable to state-of-the-art VLMs trained from scratch on massive amounts of data, such as GPT-4. Inspired by this captivating idea, in this work, we build a high-quality Remote Sensing Image Captioning dataset (RSICap) that facilitates the development of large VLMs in the RS field. Unlike previous RS datasets that either employ model-generated captions or short descriptions, RSICap comprises 2,585 human-annotated captions with rich and high-quality information. This dataset offers detailed descriptions for each image, encompassing scene descriptions (e.g., residential area, airport, or farmland) as well as object information (e.g., color, shape, quantity, absolute position, etc). To facilitate the evaluation of VLMs in the field of RS, we also provide a benchmark evaluation dataset called RSIEval. This dataset consists of human-annotated captions and visual question-answer pairs, allowing for a comprehensive assessment of VLMs in the context of RS.

This study delves into the temporal dynamics within the equity market through the lens of bond traders. Recognizing that the riskless interest rate fluctuates over time, we leverage the Black-Derman-Toy model to trace its temporal evolution. To gain insights from a bond trader's perspective, we focus on a specific type of bond: the zero-coupon bond. This paper introduces a pricing algorithm for this bond and presents a formula that can be used to ascertain its real value. By crafting an equation that juxtaposes the theoretical value of a zero-coupon bond with its actual value, we can deduce the risk-neutral probability. It is noteworthy that the risk-neutral probability correlates with variables like the instantaneous mean return, instantaneous volatility, and inherent upturn probability in the equity market. Examining these relationships enables us to discern the temporal shifts in these parameters. Our findings suggest that the mean starts at a negative value, eventually plateauing at a consistent level. The volatility, on the other hand, initially has a minimal positive value, peaks swiftly, and then stabilizes. Lastly, the upturn probability is initially significantly high, plunges rapidly, and ultimately reaches equilibrium., Comment: 15 pages, 4 figures

Data-driven medium-range weather forecasting has attracted much attention in recent years. However, the forecasting accuracy at high resolution is unsatisfactory currently. Pursuing high-resolution and high-quality weather forecasting, we develop a data-driven model SwinRDM which integrates an improved version of SwinRNN with a diffusion model. SwinRDM performs predictions at 0.25-degree resolution and achieves superior forecasting accuracy to IFS (Integrated Forecast System), the state-of-the-art operational NWP model, on representative atmospheric variables including 500 hPa geopotential (Z500), 850 hPa temperature (T850), 2-m temperature (T2M), and total precipitation (TP), at lead times of up to 5 days. We propose to leverage a two-step strategy to achieve high-resolution predictions at 0.25-degree considering the trade-off between computation memory and forecasting accuracy. Recurrent predictions for future atmospheric fields are firstly performed at 1.40625-degree resolution, and then a diffusion-based super-resolution model is leveraged to recover the high spatial resolution and finer-scale atmospheric details. SwinRDM pushes forward the performance and potential of data-driven models for a large margin towards operational applications.

Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera., Comment: ICML 2023 Oral version. Code+Models: https://github.com/facebookresearch/hiera

Quantum Hall systems host chiral edge states extending along the one-dimensional boundary of any two-dimensional sample. In solid state materials, the edge states serve as perfectly robust transport channels that produce a quantised Hall conductance; due to their chirality, and the topological protection by the Chern number of the bulk bandstructure, they cannot be spatially localized by defects or disorder. Here, we show experimentally that the chiral edge states of a lossy quantum Hall system can be localized. In a gyromagnetic photonic crystal exhibiting the quantum Hall topological phase, an appropriately structured loss configuration imparts the edge states' complex energy spectrum with a feature known as point-gap winding. This intrinsically non-Hermitian topological invariant is distinct from the Chern number invariant of the bulk (which remains intact) and induces mode localization via the "non-Hermitian skin effect". The interplay of the two topological phenomena - the Chern number and point-gap winding - gives rise to a non-Hermitian generalisation of the paradigmatic Chern-type bulk-boundary correspondence principle. Compared to previous realisations of the non-Hermitian skin effect, the skin modes in this system have superior robustness against local defects and disorders.

The remarkable achievements of ChatGPT and GPT-4 have sparked a wave of interest and research in the field of large language models for Artificial General Intelligence (AGI). These models provide intelligent solutions close to human thinking, enabling us to use general artificial intelligence to solve problems in various applications. However, in remote sensing (RS), the scientific literature on the implementation of AGI remains relatively scant. Existing AI-related research in remote sensing primarily focuses on visual understanding tasks while neglecting the semantic understanding of the objects and their relationships. This is where vision-language models excel, as they enable reasoning about images and their associated textual descriptions, allowing for a deeper understanding of the underlying semantics. Vision-language models can go beyond visual recognition of RS images, model semantic relationships, and generate natural language descriptions of the image. This makes them better suited for tasks requiring visual and textual understanding, such as image captioning, and visual question answering. This paper provides a comprehensive review of the research on vision-language models in remote sensing, summarizing the latest progress, highlighting challenges, and identifying potential research opportunities., Comment: Accepted by IEEE Geoscience and Remote Sensing Magazine

We introduce a discrete binary tree for pricing contingent claims with the underlying security prices exhibiting history dependence characteristic of that induced by market microstructure phenomena. Example dependencies considered include moving average or autoregressive behavior. Our model is market-complete, arbitrage-free, and preserves all of the parameters governing the historical (natural world) price dynamics when passing to an equivalent martingale (risk-neutral) measure. Specifically, this includes the instantaneous mean and variance of the asset return and the instantaneous probabilities for the direction of asset price movement. We believe this is the first paper to demonstrate the ability to include market microstructure effects in dynamic asset/option pricing in a market-complete, no-arbitrage, format.

Motivated by the Corns-Satchell, continuous time, option pricing model, we develop a binary tree pricing model with underlying asset price dynamics following It\^o-Mckean skew Brownian motion. While the Corns-Satchell market model is incomplete, our discrete time market model is defined in the natural world; extended to the risk neutral world under the no-arbitrage condition where derivatives are priced under uniquely determined risk-neutral probabilities; and is complete. The skewness introduced in the natural world is preserved in the risk neutral world. Furthermore, we show that the model preserves skewness under the continuous-time limit. We provide numerical applications of our model to the valuation of European put and call options on exchange-traded funds tracking the S&P Global 1200 index., Comment: 49 pages, 7 figures

In this paper, we extend the unified kinetic particle (UGKP) method to the frequency-dependent radiative transfer equation with both absorption-emission and scattering processes. The extended UGKP method could not only capture the diffusion and free transport limit, but also provide a smooth transition in the physical and frequency space in the regime between the above two limits. The proposed scheme has the properties of asymptotic-preserving, regime-adaptive, and entropy-preserving, which make it an accurate and efficient scheme in the simulation of multiscale photon transport problems. The methodology of scheme construction is a coupled evolution of macroscopic energy equation and the microscopic radiant intensity equation, where the numerical flux in macroscopic energy equation and the closure in microscopic radiant intensity equation are constructed based on the integral solution. Both numerical dissipation and computational complexity are well controlled especially in the optical thick regime. A 2D multi-thread code on a general unstructured mesh has been developed. Several numerical tests have been simulated to verify the numerical scheme and code, covering a wide range of flow regimes. The numerical scheme and code that we developed are highly demanded and widely applicable in the high energy density engineering applications.

The field of topological photonics was initiated with the realization of a Chern insulator phase in a gyromagnetic photonic crystal (PhC) with broken time-reversal symmetry (T), hosting chiral edge states that are topologically protected propagating modes. Recent advances in higher-order band topology have discovered another type of topological state, as manifested by those modes localized at the corners of a sample, which are known as corner states. Here we report the realization of a quadrupole higher-order topological insulator phase in a gyromagnetic PhC, induced by a topological phase transition from the previously demonstrated Chern insulator phase. The evolution of the boundary modes from propagating chiral edge states to localized corner states has been characterized by microwave measurements. We also demonstrate topological bound states in the continuum, when the gyromagnetic PhC is magnetically tuned. These results extend the quadrupole topological insulator phase into T-broken systems, and integrate topologically protected propagating and localized modes in the same platform., Comment: 7 pages,4 figures

Armchair graphene nanoribbons (AGNRs) with sub-nanometer width are potential materials for fabrication of novel nanodevices thanks to their moderate direct band gaps. AGNRs are usually synthesized by polymerizing precursor molecules on substrate surface. However, it is time-consuming and not suitable for large-scale production. AGNRs can also be grown by transforming precursor molecules inside single-walled carbon nanotubes via furnace annealing, but the obtained AGNRs are normally twisted. In this work, microwave heating is applied for transforming precursor molecules into AGNRs. The fast heating process allows synthesizing the AGNRs in seconds. Several different molecules were successfully transformed into AGNRs, suggesting that it is a universal method. More importantly, as demonstrated by Raman spectroscopy, aberration-corrected high-resolution transmission electron microscopy and theoretical calculations, less twisted AGNRs are synthesized by the microwave heating than the furnace annealing. Our results reveal a route for rapid production of AGNRs in large scale, which would benefit future applications in novel AGNRs-based semiconductor devices., Comment: 7 pages, 7 figures

We present PolyBuilding, a fully end-to-end polygon Transformer for building extraction. PolyBuilding direct predicts vector representation of buildings from remote sensing images. It builds upon an encoder-decoder transformer architecture and simultaneously outputs building bounding boxes and polygons. Given a set of polygon queries, the model learns the relations among them and encodes context information from the image to predict the final set of building polygons with fixed vertex numbers. Corner classification is performed to distinguish the building corners from the sampled points, which can be used to remove redundant vertices along the building walls during inference. A 1-d non-maximum suppression (NMS) is further applied to reduce vertex redundancy near the building corners. With the refinement operations, polygons with regular shapes and low complexity can be effectively obtained. Comprehensive experiments are conducted on the CrowdAI dataset. Quantitative and qualitative results show that our approach outperforms prior polygonal building extraction methods by a large margin. It also achieves a new state-of-the-art in terms of pixel-level coverage, instance-level precision and recall, and geometry-level properties (including contour regularity and polygon complexity).

We consider option pricing using replicating binomial trees, with a two fold purpose. The first is to introduce ESG valuation into option pricing. We explore this in a number of scenarios, including enhancement of yield due to trader information and the impact of the past history of a market driver. The second is to emphasize the use of discrete dynamic pricing, rather than continuum models, as the natural model that governs actual market practice. We further emphasize that discrete option pricing models must use discrete compounding (such as risk-free rate compounding of $1+r_f \Delta t$) rather than continuous compounding (such as $e^{r_f \Delta t})$.

Single-view RGB-D human reconstruction with implicit functions is often formulated as per-point classification. Specifically, a set of 3D locations within the view-frustum of the camera are first projected independently onto the image and a corresponding feature is subsequently extracted for each 3D location. The feature of each 3D location is then used to classify independently whether the corresponding 3D point is inside or outside the observed object. This procedure leads to sub-optimal results because correlations between predictions for neighboring locations are only taken into account implicitly via the extracted features. For more accurate results we propose the occupancy planes (OPlanes) representation, which enables to formulate single-view RGB-D human reconstruction as occupancy prediction on planes which slice through the camera's view frustum. Such a representation provides more flexibility than voxel grids and enables to better leverage correlations than per-point classification. On the challenging S3D data we observe a simple classifier based on the OPlanes representation to yield compelling results, especially in difficult situations with partial occlusions due to other objects and partial visibility, which haven't been addressed by prior work., Comment: AAAI2023; Code: https://github.com/Xiaoming-Zhao/oplanes

Data-driven approaches for medium-range weather forecasting are recently shown extraordinarily promising for ensemble forecasting for their fast inference speed compared to traditional numerical weather prediction (NWP) models, but their forecast accuracy can hardly match the state-of-the-art operational ECMWF Integrated Forecasting System (IFS) model. Previous data-driven attempts achieve ensemble forecast using some simple perturbation methods, like initial condition perturbation and Monte Carlo dropout. However, they mostly suffer unsatisfactory ensemble performance, which is arguably attributed to the sub-optimal ways of applying perturbation. We propose a Swin Transformer-based Variational Recurrent Neural Network (SwinVRNN), which is a stochastic weather forecasting model combining a SwinRNN predictor with a perturbation module. SwinRNN is designed as a Swin Transformer-based recurrent neural network, which predicts future states deterministically. Furthermore, to model the stochasticity in prediction, we design a perturbation module following the Variational Auto-Encoder paradigm to learn multivariate Gaussian distributions of a time-variant stochastic latent variable from data. Ensemble forecasting can be easily achieved by perturbing the model features leveraging noise sampled from the learned distribution. We also compare four categories of perturbation methods for ensemble forecasting, i.e. fixed distribution perturbation, learned distribution perturbation, MC dropout, and multi model ensemble. Comparisons on WeatherBench dataset show the learned distribution perturbation method using our SwinVRNN model achieves superior forecast accuracy and reasonable ensemble spread due to joint optimization of the two targets. More notably, SwinVRNN surpasses operational IFS on surface variables of 2-m temperature and 6-hourly total precipitation at all lead times up to five days.

Optimization within a layer of a deep-net has emerged as a new direction for deep-net layer design. However, there are two main challenges when applying these layers to computer vision tasks: (a) which optimization problem within a layer is useful?; (b) how to ensure that computation within a layer remains efficient? To study question (a), in this work, we propose total variation (TV) minimization as a layer for computer vision. Motivated by the success of total variation in image processing, we hypothesize that TV as a layer provides useful inductive bias for deep-nets too. We study this hypothesis on five computer vision tasks: image classification, weakly supervised object localization, edge-preserving smoothing, edge detection, and image denoising, improving over existing baselines. To achieve these results we had to address question (b): we developed a GPU-based projected-Newton method which is $37\times$ faster than existing solutions., Comment: CVPR 2022

Designing equivariance as an inductive bias into deep-nets has been a prominent approach to build effective models, e.g., a convolutional neural network incorporates translation equivariance. However, incorporating these inductive biases requires knowledge about the equivariance properties of the data, which may not be available, e.g., when encountering a new domain. To address this, we study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. We propose to use the partition distance to empirically quantify the accuracy of the recovered equivariance. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme. Empirically, we show that the approach recovers known equivariances, such as permutations and shifts, on sum of numbers and spatially-invariant data., Comment: AISTATS 2022

The glass-forming ability (GFA) of alloys, colloidal dispersions, and other particulate materials, as measured by the critical cooling rate $R_c$, can span more than ten orders of magnitude. Even after numerous previous studies, the physical features that control the GFA are still not well understood. For example, it is well-known that mixtures are better glass-formers than monodisperse systems and that particle size and cohesive energy differences among constituents improve the GFA, but it is not currently known how particle size differences couple to cohesive energy differences to determine the GFA. We perform molecular dynamics simulations to determine the GFA of equimolar, binary Lennard-Jones (LJ) mixtures versus the normalized cohesive energy difference $\epsilon_\_$ and mixing energy $\bar \epsilon_{AB}$ between particles A and B. We find several important results. First, the $\log_{10} R_c$ contours in the $\bar \epsilon_{AB}$-$\epsilon_\_$ plane are ellipsoidal for all diameter ratios, and thus $R_c$ is determined by the Mahalanobis distance $d_M$ from a given point in the $\bar \epsilon_{AB}$-$\epsilon_\_$ plane to the center of the ellipsoidal contours. Second, LJ systems for which the larger particles have larger cohesive energy are generally better glass formers than those for which the larger particles have smaller cohesive energy. Third, $d_M(\epsilon_\_,\bar \epsilon_{AB})$ is determined by the relative Voronoi volume difference between particles and local chemical order $S_{AB}$, which gives the average fraction of nearest-neighbor B particles surrounding an A particle and vice-versa. In particular, the shifted Mahalanobis distance $d_M - d^0_M$ versus the shifted chemical order $S_{AB}-S_{AB}^0$ collapses onto a hyperbolic master curve for all diameter ratios.