Search

The Segment Anything Model (SAM) was originally designed for label-agnostic mask generation. Does this model also possess inherent semantic understanding, of value to broader visual tasks? In this work we follow a multi-staged approach towards exploring this question. We firstly quantify SAM's semantic capabilities by comparing base image encoder efficacy under classification tasks, in comparison with established models (CLIP and DINOv2). Our findings reveal a significant lack of semantic discriminability in SAM feature representations, limiting potential for tasks that require class differentiation. This initial result motivates our exploratory study that attempts to enable semantic information via in-context learning with lightweight fine-tuning where we observe that generalisability to unseen classes remains limited. Our observations culminate in the proposal of a training-free approach that leverages DINOv2 features, towards better endowing SAM with semantic understanding and achieving instance-level class differentiation through feature-based similarity. Our study suggests that incorporation of external semantic sources provides a promising direction for the enhancement of SAM's utility with respect to complex visual tasks that require semantic understanding., Comment: Preprint. Work in progress

Intels efforts to build a practical quantum computer are focused on developing a scalable spin-qubit platform leveraging industrial high-volume semiconductor manufacturing expertise and 300 mm fabrication infrastructure. Here, we provide an overview of the design, fabrication, and demonstration of a new customized quantum test chip, which contains 12-quantum-dot spin-qubit linear arrays, code named Tunnel Falls. These devices are fabricated using immersion and extreme ultraviolet lithography (EUV), along with other standard high-volume manufacturing (HVM) processes, as well as production-level process control. We present key device features and fabrication details, as well as qubit characterization results confirming device functionality. These results corroborate our fabrication methods and are a crucial step towards scaling of extensible 2D qubit array schemes.

CMOS-based microelectronics are limited to ~150{\deg}C and therefore not suitable for the extreme high temperatures in aerospace, energy, and space applications. While wide bandgap semiconductors can provide high-temperature logic, nonvolatile memory devices at high temperatures have been challenging. In this work, we develop a nonvolatile electrochemical memory cell that stores and retains analog and digital information at temperatures as high as 600 {\deg}C. Through correlative electron microscopy, we show that this high-temperature information retention is a result of composition phase separation between the oxidized and reduced forms of amorphous tantalum oxide. This result demonstrates a memory concept that is resilient at extreme temperatures and reveals phase separation as the principal mechanism that enables nonvolatile information storage in these electrochemical memory cells., Comment: 19 pages, 4 figures

Mathematically modelling diffusive and advective transport of particles in heterogeneous layered media is important to many applications in computational, biological and medical physics. While deterministic continuum models of such transport processes are well established, they fail to account for randomness inherent in many problems and are valid only for a large number of particles. To address this, this paper derives a suite of equivalent stochastic (discrete-time discrete-space random walk) models for several standard continuum (partial differential equation) models of diffusion and advection-diffusion across a fully- or semi-permeable interface. Our approach involves discretising the continuum model in space and time to yield a Markov chain, which governs the transition probabilities between spatial lattice sites during each time step. Discretisation in space is carried out using a standard finite volume method while two options are considered for discretisation in time. A simple forward Euler discretisation yields a stochastic model taking the form of a local (nearest-neighbour) random walk with simple analytical expressions for the transition probabilities while an exact exponential discretisation yields a non-local random walk with transition probabilities defined numerically via a matrix exponential. Constraints on the size of the spatial and/or temporal steps are provided for each option to ensure the transition probabilities are non-negative. MATLAB code comparing the stochastic and continuum models is available on GitHub (https://github.com/elliotcarr/Carr2024c) with simulation results demonstrating good agreement for several example problems., Comment: 17 pages, 3 figures

Neural architecture search (NAS) finds high performing networks for a given task. Yet the results of NAS are fairly prosaic; they did not e.g. create a shift from convolutional structures to transformers. This is not least because the search spaces in NAS often aren't diverse enough to include such transformations a priori. Instead, for NAS to provide greater potential for fundamental design shifts, we need a novel expressive search space design which is built from more fundamental operations. To this end, we introduce einspace, a search space based on a parameterised probabilistic context-free grammar. Our space is versatile, supporting architectures of various sizes and complexities, while also containing diverse network operations which allow it to model convolutions, attention components and more. It contains many existing competitive architectures, and provides flexibility for discovering new ones. Using this search space, we perform experiments to find novel architectures as well as improvements on existing ones on the diverse Unseen NAS datasets. We show that competitive architectures can be obtained by searching from scratch, and we consistently find large improvements when initialising the search with strong baselines. We believe that this work is an important advancement towards a transformative NAS paradigm where search space expressivity and strategic search initialisation play key roles., Comment: NeurIPS 2024. Project page at https://linusericsson.github.io/einspace/

In continual learning (CL) -- where a learner trains on a stream of data -- standard hyperparameter optimisation (HPO) cannot be applied, as a learner does not have access to all of the data at the same time. This has prompted the development of CL-specific HPO frameworks. The most popular way to tune hyperparameters in CL is to repeatedly train over the whole data stream with different hyperparameter settings. However, this end-of-training HPO is unrealistic as in practice a learner can only see the stream once. Hence, there is an open question: what HPO framework should a practitioner use for a CL problem in reality? This paper answers this question by evaluating several realistic HPO frameworks. We find that all the HPO frameworks considered, including end-of-training HPO, perform similarly. We therefore advocate using the realistic and most computationally efficient method: fitting the hyperparameters on the first task and then fixing them throughout training., Comment: Preprint, 9 pages

We present EgoPoseFormer, a simple yet effective transformer-based model for stereo egocentric human pose estimation. The main challenge in egocentric pose estimation is overcoming joint invisibility, which is caused by self-occlusion or a limited field of view (FOV) of head-mounted cameras. Our approach overcomes this challenge by incorporating a two-stage pose estimation paradigm: in the first stage, our model leverages the global information to estimate each joint's coarse location, then in the second stage, it employs a DETR style transformer to refine the coarse locations by exploiting fine-grained stereo visual features. In addition, we present a Deformable Stereo Attention operation to enable our transformer to effectively process multi-view features, which enables it to accurately localize each joint in the 3D world. We evaluate our method on the stereo UnrealEgo dataset and show it significantly outperforms previous approaches while being computationally efficient: it improves MPJPE by 27.4mm (45% improvement) with only 7.9% model parameters and 13.1% FLOPs compared to the state-of-the-art. Surprisingly, with proper training settings, we find that even our first-stage pose proposal network can achieve superior performance compared to previous arts. We also show that our method can be seamlessly extended to monocular settings, which achieves state-of-the-art performance on the SceneEgo dataset, improving MPJPE by 25.5mm (21% improvement) compared to the best existing method with only 60.7% model parameters and 36.4% FLOPs. Code is available at: https://github.com/ChenhongyiYang/egoposeformer ., Comment: Accepted to ECCV 2024

We present PlainMamba: a simple non-hierarchical state space model (SSM) designed for general visual recognition. The recent Mamba model has shown how SSMs can be highly competitive with other architectures on sequential data and initial attempts have been made to apply it to images. In this paper, we further adapt the selective scanning process of Mamba to the visual domain, enhancing its ability to learn features from two-dimensional images by (i) a continuous 2D scanning process that improves spatial continuity by ensuring adjacency of tokens in the scanning sequence, and (ii) direction-aware updating which enables the model to discern the spatial relations of tokens by encoding directional information. Our architecture is designed to be easy to use and easy to scale, formed by stacking identical PlainMamba blocks, resulting in a model with constant width throughout all layers. The architecture is further simplified by removing the need for special tokens. We evaluate PlainMamba on a variety of visual recognition tasks, achieving performance gains over previous non-hierarchical models and is competitive with hierarchical alternatives. For tasks requiring high-resolution inputs, in particular, PlainMamba requires much less computing while maintaining high performance. Code and models are available at: https://github.com/ChenhongyiYang/PlainMamba ., Comment: Accepted to BMVC 2024

We present WidthFormer, a novel transformer-based module to compute Bird's-Eye-View (BEV) representations from multi-view cameras for real-time autonomous-driving applications. WidthFormer is computationally efficient, robust and does not require any special engineering effort to deploy. We first introduce a novel 3D positional encoding mechanism capable of accurately encapsulating 3D geometric information, which enables our model to compute high-quality BEV representations with only a single transformer decoder layer. This mechanism is also beneficial for existing sparse 3D object detectors. Inspired by the recently proposed works, we further improve our model's efficiency by vertically compressing the image features when serving as attention keys and values, and then we develop two modules to compensate for potential information loss due to feature compression. Experimental evaluation on the widely-used nuScenes 3D object detection benchmark demonstrates that our method outperforms previous approaches across different 3D detection architectures. More importantly, our model is highly efficient. For example, when using $256\times 704$ input images, it achieves 1.5 ms and 2.8 ms latency on NVIDIA 3090 GPU and Horizon Journey-5 computation solutions. Furthermore, WidthFormer also exhibits strong robustness to different degrees of camera perturbations. Our study offers valuable insights into the deployment of BEV transformation methods in real-world, complex road environments. Code is available at https://github.com/ChenhongyiYang/WidthFormer ., Comment: IROS 2024 Oral Presentation

Deep Neural Networks (DNNs) are extremely computationally demanding, which presents a large barrier to their deployment on resource-constrained devices. Since such devices are where many emerging deep learning applications lie (e.g., drones, vision-based medical technology), significant bodies of work from both the machine learning and systems communities have attempted to provide optimizations to accelerate DNNs. To help unify these two perspectives, in this paper we combine machine learning and systems techniques within the Deep Learning Acceleration Stack (DLAS), and demonstrate how these layers can be tightly dependent on each other with an across-stack perturbation study. We evaluate the impact on accuracy and inference time when varying different parameters of DLAS across two datasets, seven popular DNN architectures, four DNN compression techniques, three algorithmic primitives with sparse and dense variants, untuned and auto-scheduled code generation, and four hardware platforms. Our evaluation highlights how perturbations across DLAS parameters can cause significant variation and across-stack interactions. The highest level observation from our evaluation is that the model size, accuracy, and inference time are not guaranteed to be correlated. Overall we make 13 key observations, including that speedups provided by compression techniques are very hardware dependent, and that compiler auto-tuning can significantly alter what the best algorithm to use for a given configuration is. With DLAS, we aim to provide a reference framework to aid machine learning and systems practitioners in reasoning about the context in which their respective DNN acceleration solutions exist in. With our evaluation strongly motivating the need for co-design, we believe that DLAS can be a valuable concept for exploring the next generation of co-designed accelerated deep learning solutions.

Despite recent advancements in image generation, diffusion models still remain largely underexplored in Earth Observation. In this paper we show that state-of-the-art pretrained diffusion models can be conditioned on cartographic data to generate realistic satellite images. We provide two large datasets of paired OpenStreetMap images and satellite views over the region of Mainland Scotland and the Central Belt. We train a ControlNet model and qualitatively evaluate the results, demonstrating that both image quality and map fidelity are possible. Finally, we provide some insights on the opportunities and challenges of applying these models for remote sensing. Our model weights and code for creating the dataset are publicly available at https://github.com/miquel-espinosa/map-sat., Comment: 13 pages, 6 figures. preprint

Advances in material design has led to the rapid development of novel materials with increasing complexity and functions in bioengineering. In particular, functionally graded materials (FGMs) offer important advantages in various fields of application. In this work, we consider a heterogeneous reaction-diffusion model for an FGM spherical drug releasing system that generalizes the multi-layer configuration to arbitrary spatially-variable coefficients. Our model proposes a possible form for the drug diffusivity and reaction rate functions exhibiting fixed average material properties and a drug release profile that can be continuously varied between the limiting cases of a homogeneous system (constant coefficients) and two-layer system (stepwise coefficients). A hybrid analytical-numerical solution is then used to solve the model, which provides closed-form expressions for the drug concentration and drug release profiles in terms of generalized Fourier series. The resulting concentration and mass profiles show how the release rate can be controlled and continuously varied between a fast (homogeneous) and slow (two-layer) release., Comment: 16 pages, 10 figures, submitted

Building a fault-tolerant quantum computer will require vast numbers of physical qubits. For qubit technologies based on solid state electronic devices, integrating millions of qubits in a single processor will require device fabrication to reach a scale comparable to that of the modern CMOS industry. Equally importantly, the scale of cryogenic device testing must keep pace to enable efficient device screening and to improve statistical metrics like qubit yield and voltage variation. Spin qubits based on electrons in Si have shown impressive control fidelities but have historically been challenged by yield and process variation. Here we present a testing process using a cryogenic 300 mm wafer prober to collect high-volume data on the performance of hundreds of industry-manufactured spin qubit devices at 1.6 K. This testing method provides fast feedback to enable optimization of the CMOS-compatible fabrication process, leading to high yield and low process variation. Using this system, we automate measurements of the operating point of spin qubits and probe the transitions of single electrons across full wafers. We analyze the random variation in single-electron operating voltages and find that the optimized fabrication process leads to low levels of disorder at the 300 mm scale. Together these results demonstrate the advances that can be achieved through the application of CMOS industry techniques to the fabrication and measurement of spin qubit devices., Comment: 33 pages, 4 figures, 10 extended data figures. Supplementary Information available under "Ancillary files."

Likelihood profiling is an efficient and powerful frequentist approach for parameter estimation, uncertainty quantification and practical identifiablity analysis. Unfortunately, these methods cannot be easily applied for stochastic models without a tractable likelihood function. Such models are typical in many fields of science, rendering these classical approaches impractical in these settings. To address this limitation, we develop a new approach to generalising the methods of likelihood profiling for situations when the likelihood cannot be evaluated but stochastic simulations of the assumed data generating process are possible. Our approach is based upon recasting developments from generalised Bayesian inference into a frequentist setting. We derive a method for constructing generalised likelihood profiles and calibrating these profiles to achieve desired frequentist coverage for a given coverage level. We demonstrate the performance of our method on realistic examples from the literature and highlight the capability of our approach for the purpose of practical identifability analysis for models with intractable likelihoods.

We consider diffusion-controlled release of particles from $d$-dimensional radially-symmetric geometries. A quantity commonly used to characterise such diffusive processes is the proportion of particles remaining within the geometry over time, denoted as $P(t)$. The stochastic approach for computing $P(t)$ is time-consuming and lacks analytical insight into key parameters while the continuum approach yields complicated expressions for $P(t)$ that obscure the influence of key parameters and complicate the process of fitting experimental release data. In this work, to address these issues, we develop several simple surrogate models to approximate $P(t)$ by matching moments with the continuum analogue of the stochastic diffusion model. Surrogate models are developed for homogeneous slab, circular, annular, spherical and spherical shell geometries with a constant particle movement probability and heterogeneous slab, circular, annular and spherical geometries, comprised of two concentric layers with different particle movement probabilities. Each model is easy to evaluate, agrees well with both stochastic and continuum calculations of $P(t)$ and provides analytical insight into the key parameters of the diffusive transport system: dimension, diffusivity, geometry and boundary conditions., Comment: 22 pages, 3 figures, accepted version

In this paper, we present a new method for calculating linear absorption spectra for large molecular aggregates, called dyadic adaptive HOPS (DadHOPS). This method combines the adaptive HOPS (adHOPS) framework, which uses locality to improve computational scaling, with the dyadic HOPS method previously developed to calculate linear and non-linear spectroscopic signals. To construct a local representation of dyadic HOPS, we introduce an initial state decomposition which reconstructs the linear absorption spectra from a sum over locally excited initial conditions. We demonstrate the sum over initial conditions can be efficiently Monte Carlo sampled, that the corresponding calculations achieve size-invariant (i.e. $\mathcal{O}(1)$) scaling for sufficiently large aggregates, and that it allows for the trivial inclusion of static disorder in the Hamiltonian. We present calculations on the photosystem I core complex to explore the behavior of the initial state decomposition in complex molecular aggregates, and proof-of-concept DadHOPS calculations on an artificial molecular aggregate inspired by perylene bis-imide.

We present the Group Propagation Vision Transformer (GPViT): a novel nonhierarchical (i.e. non-pyramidal) transformer model designed for general visual recognition with high-resolution features. High-resolution features (or tokens) are a natural fit for tasks that involve perceiving fine-grained details such as detection and segmentation, but exchanging global information between these features is expensive in memory and computation because of the way self-attention scales. We provide a highly efficient alternative Group Propagation Block (GP Block) to exchange global information. In each GP Block, features are first grouped together by a fixed number of learnable group tokens; we then perform Group Propagation where global information is exchanged between the grouped features; finally, global information in the updated grouped features is returned back to the image features through a transformer decoder. We evaluate GPViT on a variety of visual recognition tasks including image classification, semantic segmentation, object detection, and instance segmentation. Our method achieves significant performance gains over previous works across all tasks, especially on tasks that require highresolution outputs, for example, our GPViT-L3 outperforms Swin Transformer-B by 2.0 mIoU on ADE20K semantic segmentation with only half as many parameters. Project page: chenhongyiyang.com/projects/GPViT/GPViT, Comment: Accepted by ICLR 2023. Project Page: chenhongyiyang.com/projects/GPViT/GPViT. Code: https://github.com/ChenhongyiYang/GPViT

Annotating datasets for object detection is an expensive and time-consuming endeavor. To minimize this burden, active learning (AL) techniques are employed to select the most informative samples for annotation within a constrained "annotation budget". Traditional AL strategies typically rely on model uncertainty or sample diversity for query sampling, while more advanced methods have focused on developing AL-specific object detector architectures to enhance performance. However, these specialized approaches are not readily adaptable to different object detectors due to the significant engineering effort required for integration. To overcome this challenge, we introduce Plug and Play Active Learning (PPAL), a simple and effective AL strategy for object detection. PPAL is a two-stage method comprising uncertainty-based and diversity-based sampling phases. In the first stage, our Difficulty Calibrated Uncertainty Sampling leverage a category-wise difficulty coefficient that combines both classification and localisation difficulties to re-weight instance uncertainties, from which we sample a candidate pool for the subsequent diversity-based sampling. In the second stage, we propose Category Conditioned Matching Similarity to better compute the similarities of multi-instance images as ensembles of their instance similarities, which is used by the k-Means++ algorithm to sample the final AL queries. PPAL makes no change to model architectures or detector training pipelines; hence it can be easily generalized to different object detectors. We benchmark PPAL on the MS-COCO and Pascal VOC datasets using different detector architectures and show that our method outperforms prior work by a large margin. Code is available at https://github.com/ChenhongyiYang/PPAL, Comment: CVPR 2024

Electrochemical random-access memory (ECRAM) is a recently developed and highly promising analog resistive memory element for in-memory computing. One longstanding challenge of ECRAM is attaining retention time beyond a few hours. This short retention has precluded ECRAM from being considered for inference classification in deep neural networks, which is likely the largest opportunity for in-memory computing. In this work, we develop an ECRAM cell with orders of magnitude longer retention than previously achieved, and which we anticipate to exceed 10 years at 85C. We hypothesize that the origin of this exceptional retention is phase separation, which enables the formation of multiple effectively equilibrium resistance states. This work highlights the promises and opportunities to use phase separation to yield ECRAM cells with exceptionally long, and potentially permanent, retention times., Comment: 14 pages, 4 figures

Calculating the thermal diffusivity of solid materials is commonly carried out using the laser flash experiment. This classical experiment considers a small (usually thin disc-shaped) sample of the material with parallel front and rear surfaces, applying a heat pulse to the front surface and recording the resulting rise in temperature over time on the rear surface. Recently, Carr and Wood [Int J Heat Mass Transf, 144 (2019) 118609] showed that the thermal diffusivity can be expressed analytically in terms of the heat flux function applied at the front surface and the temperature rise history at the rear surface. In this paper, we generalise this result to radial unidirectional heat flow, developing new analytical formulas for calculating the thermal diffusivity for cylindrical shell and spherical shell shaped samples. Two configurations are considered: (i) heat pulse applied on the inner surface and temperature rise recorded on the outer surface and (ii) heat pulse applied on the outer surface and temperature rise recorded on the inner surface. Code implementing and verifying the thermal diffusivity formulas for both configurations is made available., Comment: 10 pages, 3 figures, accepted version

Emerging concepts for neuromorphic computing, bioelectronics, and brain-computer interfacing inspire new research avenues aimed at understanding the relationship between oxidation state and conductivity in unexplored materials. Here, we present ruthenium Prussian blue analogue (RuPBA), a mixed valence coordination compound with an open framework structure and ability to conduct both ionic and electronic charge, for flexible artificial synapses that reversibly switch conductance by more than four orders of magnitude based on electrochemically tunable oxidation state. Retention of programmed states is improved by nearly two orders of magnitude compared to the extensively studied organic polymers, thus reducing the frequency, complexity and energy costs associated with error correction schemes. We demonstrate dopamine detection using RuPBA synapses and biocompatibility with neuronal cells, evoking prospective application for brain-computer interfacing. By application of electron transfer theory to in-situ spectroscopic probing of intervalence charge transfer, we elucidate a switching mechanism whereby the degree of mixed valency between N-coordinated Ru sites controls the carrier concentration and mobility, as supported by DFT.