Search

Existing deep learning models have achieved promising performance in recognizing skin diseases from dermoscopic images. However, these models can only recognize samples from predefined categories, when they are deployed in the clinic, data from new unknown categories are constantly emerging. Therefore, it is crucial to automatically discover and identify new semantic categories from new data. In this paper, we propose a new novel class discovery framework for automatically discovering new semantic classes from dermoscopy image datasets based on the knowledge of known classes. Specifically, we first use contrastive learning to learn a robust and unbiased feature representation based on all data from known and unknown categories. We then propose an uncertainty-aware multi-view cross pseudo-supervision strategy, which is trained jointly on all categories of data using pseudo labels generated by a self-labeling strategy. Finally, we further refine the pseudo label by aggregating neighborhood information through local sample similarity to improve the clustering performance of the model for unknown categories. We conducted extensive experiments on the dermatology dataset ISIC 2019, and the experimental results show that our approach can effectively leverage knowledge from known categories to discover new semantic categories. We also further validated the effectiveness of the different modules through extensive ablation experiments. Our code will be released soon., Comment: 10 pages, 1 figure,Accepted by miccai 2023

Monocular depth estimation is a crucial task to measure distance relative to a camera, which is important for applications, such as robot navigation and self-driving. Traditional frame-based methods suffer from performance drops due to the limited dynamic range and motion blur. Therefore, recent works leverage novel event cameras to complement or guide the frame modality via frame-event feature fusion. However, event streams exhibit spatial sparsity, leaving some areas unperceived, especially in regions with marginal light changes. Therefore, direct fusion methods, e.g., RAMNet, often ignore the contribution of the most confident regions of each modality. This leads to structural ambiguity in the modality fusion process, thus degrading the depth estimation performance. In this paper, we propose a novel Spatial Reliability-oriented Fusion Network (SRFNet), that can estimate depth with fine-grained structure at both daytime and nighttime. Our method consists of two key technical components. Firstly, we propose an attention-based interactive fusion (AIF) module that applies spatial priors of events and frames as the initial masks and learns the consensus regions to guide the inter-modal feature fusion. The fused feature are then fed back to enhance the frame and event feature learning. Meanwhile, it utilizes an output head to generate a fused mask, which is iteratively updated for learning consensual spatial priors. Secondly, we propose the Reliability-oriented Depth Refinement (RDR) module to estimate dense depth with the fine-grained structure based on the fused features and masks. We evaluate the effectiveness of our method on the synthetic and real-world datasets, which shows that, even without pretraining, our method outperforms the prior methods, e.g., RAMNet, especially in night scenes. Our project homepage: https://vlislab22.github.io/SRFNet., Comment: Accepted to ICRA 2024

The ability to detect objects in all lighting (i.e., normal-, over-, and under-exposed) conditions is crucial for real-world applications, such as self-driving.Traditional RGB-based detectors often fail under such varying lighting conditions.Therefore, recent works utilize novel event cameras to supplement or guide the RGB modality; however, these methods typically adopt asymmetric network structures that rely predominantly on the RGB modality, resulting in limited robustness for all-day detection. In this paper, we propose EOLO, a novel object detection framework that achieves robust and efficient all-day detection by fusing both RGB and event modalities. Our EOLO framework is built based on a lightweight spiking neural network (SNN) to efficiently leverage the asynchronous property of events. Buttressed by it, we first introduce an Event Temporal Attention (ETA) module to learn the high temporal information from events while preserving crucial edge information. Secondly, as different modalities exhibit varying levels of importance under diverse lighting conditions, we propose a novel Symmetric RGB-Event Fusion (SREF) module to effectively fuse RGB-Event features without relying on a specific modality, thus ensuring a balanced and adaptive fusion for all-day detection. In addition, to compensate for the lack of paired RGB-Event datasets for all-day training and evaluation, we propose an event synthesis approach based on the randomized optical flow that allows for directly generating the event frame from a single exposure image. We further build two new datasets, E-MSCOCO and E-VOC based on the popular benchmarks MSCOCO and PASCAL VOC. Extensive experiments demonstrate that our EOLO outperforms the state-of-the-art detectors,e.g.,RENet,by a substantial margin (+3.74% mAP50) in all lighting conditions.Our code and datasets will be available at https://vlislab22.github.io/EOLO/, Comment: Accepted by ICRA 2024

Efficiently optimizing multi-model inference pipelines for fast, accurate, and cost-effective inference is a crucial challenge in machine learning production systems, given their tight end-to-end latency requirements. To simplify the exploration of the vast and intricate trade-off space of latency, accuracy, and cost in inference pipelines, providers frequently opt to consider one of them. However, the challenge lies in reconciling latency, accuracy, and cost trade-offs. To address this challenge and propose a solution to efficiently manage model variants in inference pipelines, we present IPA, an online deep learning Inference Pipeline Adaptation system that efficiently leverages model variants for each deep learning task. Model variants are different versions of pre-trained models for the same deep learning task with variations in resource requirements, latency, and accuracy. IPA dynamically configures batch size, replication, and model variants to optimize accuracy, minimize costs, and meet user-defined latency Service Level Agreements (SLAs) using Integer Programming. It supports multi-objective settings for achieving different trade-offs between accuracy and cost objectives while remaining adaptable to varying workloads and dynamic traffic patterns. Navigating a wider variety of configurations allows \namex{} to achieve better trade-offs between cost and accuracy objectives compared to existing methods. Extensive experiments in a Kubernetes implementation with five real-world inference pipelines demonstrate that IPA improves end-to-end accuracy by up to 21% with a minimal cost increase. The code and data for replications are available at https://github.com/reconfigurable-ml-pipeline/ipa.

The layered ATMPn$_2$ (A = alkali earth or rare earth atom, TM = transition metal, Pn = Sb, Bi) compounds are widely studied for their rich magnetism and electronic structure topology. Here, we characterize the physical properties of LaMn$_x$Sb$_2$, an understudied member of the ATMPn$_2$ family. LaMn$_x$Sb$_2$ forms with intrinsic Mn vacancies, and we demonstrate synthetic control of the Mn occupancy to produce single crystals with x = 0.74-0.97. Magnetization and transport measurements indicate LaMn$_x$Sb$_2$ has a rich temperature-composition (T-x) magnetic phase diagram with physical properties strongly influenced by the Mn occupancy. LaMn$_x$Sb$_2$ orders antiferromagnetically at T$_{1}$ = 130--180 K, where T$_{1}$ increases with x. Below T$_{1}$, the T-x phase diagram is complicated. At high x, there is a second transition T$_2$ that decreases in temperature as x is lowered, vanishing below x $\leq$ 0.85. A third, first-order, transition T$_3$ is detected at x $\approx$ 0.92, and the transition temperature increases as x is lowered, crossing above T$_2$ near x $\approx$ 0.9. On moving below x $<$ 0.79, we find the crystal structure changes from the P4/nmm arrangement to a I$\bar{4}$2m structure with partially ordered Mn vacancies. The change in crystal structure results in the appearance of two new low temperature phases and a crossover between regimes of negative and positive magnetoresistance. Finally, we provide neutron diffraction for x = 0.93, and find that the high x compositions first adopt a G-type AFM structure with the Mn moments aligned within the ab-plane which is followed on cooling by a second transition to a different, non-collinear structure where the moments are rotated within the basal plane. Our results demonstrate that LaMn$_x$Sb$_2$ is a highly tunable material with six unique magnetically ordered phases, depending on T and x.

To achieve the true diffraction-limited emittance of a storage ring light source, such as ~10 pm.rad for medium-energy electron beams, within a limited circumference, it is generally necessary to increase the number of bending magnets in a multi-bend achromat (MBA) lattice, as in the future upgrade plan of MAX IV with a 19BA replacing the current 7BA. However, this comes with extremely strong quadrupole and sextupole magnets and very limited space. The former can result in very small vacuum chambers, increasing the coupling impedance and thus enhancing the beam instabilities, and the latter can pose significant challenges in accommodating the necessary diagnostics and vacuum components. Inspired by the hybrid MBA lattice concept, in this paper we propose a new unit lattice concept called the complex unit lattice cell, which can reduce the magnet strengths and also save space. The complex unit cell is numerically studied using a simplified model. Then as an example, a 17BA lattice based on the complex unit cell concept is designed for a 3 GeV storage ring light source with a circumference of 537.6 m, which has a natural emittance of 19.3 pm.rad. This 17BA lattice is also compared with the 17BA lattice designed with conventional unit cells to showcase the benefits of the complex unit cell concept. This 17BA lattice also suggests a new type of MBA lattice, which we call the MBA lattice with semi-distributed chromatic correction.

Omnidirectional images (ODIs) have become increasingly popular, as their large field-of-view (FoV) can offer viewers the chance to freely choose the view directions in immersive environments such as virtual reality. The M\"obius transformation is typically employed to further provide the opportunity for movement and zoom on ODIs, but applying it to the image level often results in blurry effect and aliasing problem. In this paper, we propose a novel deep learning-based approach, called \textbf{OmniZoomer}, to incorporate the M\"obius transformation into the network for movement and zoom on ODIs. By learning various transformed feature maps under different conditions, the network is enhanced to handle the increasing edge curvatures, which alleviates the blurry effect. Moreover, to address the aliasing problem, we propose two key components. Firstly, to compensate for the lack of pixels for describing curves, we enhance the feature maps in the high-resolution (HR) space and calculate the transformed index map with a spatial index generation module. Secondly, considering that ODIs are inherently represented in the spherical space, we propose a spherical resampling module that combines the index map and HR feature maps to transform the feature maps for better spherical correlation. The transformed feature maps are decoded to output a zoomed ODI. Experiments show that our method can produce HR and high-quality ODIs with the flexibility to move and zoom in to the object of interest. Project page is available at http://vlislab22.github.io/OmniZoomer/., Comment: Accepted by ICCV 2023

Endeavors have been recently made to transfer knowledge from the labeled pinhole image domain to the unlabeled panoramic image domain via Unsupervised Domain Adaptation (UDA). The aim is to tackle the domain gaps caused by the style disparities and distortion problem from the non-uniformly distributed pixels of equirectangular projection (ERP). Previous works typically focus on transferring knowledge based on geometric priors with specially designed multi-branch network architectures. As a result, considerable computational costs are induced, and meanwhile, their generalization abilities are profoundly hindered by the variation of distortion among pixels. In this paper, we find that the pixels' neighborhood regions of the ERP indeed introduce less distortion. Intuitively, we propose a novel UDA framework that can effectively address the distortion problems for panoramic semantic segmentation. In comparison, our method is simpler, easier to implement, and more computationally efficient. Specifically, we propose distortion-aware attention (DA) capturing the neighboring pixel distribution without using any geometric constraints. Moreover, we propose a class-wise feature aggregation (CFA) module to iteratively update the feature representations with a memory bank. As such, the feature similarity between two domains can be consistently optimized. Extensive experiments show that our method achieves new state-of-the-art performance while remarkably reducing 80% parameters., Comment: Accepted to ICCV 2023

We report the single crystal growth and characterization of EuIn$_2$, a magnetic topological semimetal candidate according to our density functional theory (DFT) calculations. We present results from electrical resistance, magnetization, M\"ossbauer spectroscopy, and X-ray resonant magnetic scattering (XRMS) measurements. We observe three magnetic transitions at $T_{\text{N}1}\sim 14.2~$K, $T_{\text{N}2}\sim12.8~$K and $T_{\text{N}3}\sim 11~$K, signatures of which are consistently seen in anisotropic temperature dependent magnetic susceptibility and electrical resistance data. M\"ossbauer spectroscopy measurements on ground crystals suggest an incommensurate sinusoidally modulated magnetic structure below the transition at $T_{\text{N}1}\sim 14~$K, followed by the appearance of higher harmonics in the modulation on further cooling roughly below $T_{\text{N}2}\sim13~$K, before the moment distribution squaring up below the lowest transition around $T_{\text{N}3}\sim 11~$K. XRMS measurements showed the appearance of magnetic Bragg peaks below $T_{\text{N}1}\sim14~$K, with a propagation vector of $\bm{\tau}$ $=(\tau_h,\bar{\tau}_h,0)$, with $\tau_h$varying with temperature, and showing a jump at $T_{\text{N}3}\sim11$~K. The temperature dependence of $\tau_h$ between $\sim11$~K and $14$~K shows incommensurate values consistent with the M\"{o}ssbauer data. XRMS data indicate that $\tau_h$ remains incommensurate at low temperatures and locks into $\tau_h=0.3443(1)$.

In this paper, we propose EventBind, a novel and effective framework that unleashes the potential of vision-language models (VLMs) for event-based recognition to compensate for the lack of large-scale event-based datasets. In particular, due to the distinct modality gap with the image-text data and the lack of large-scale datasets, learning a common representation space for images, texts, and events is non-trivial.Intuitively, we need to address two key challenges: 1) how to generalize CLIP's visual encoder to event data while fully leveraging events' unique properties, e.g., sparsity and high temporal resolution; 2) how to effectively align the multi-modal embeddings, i.e., image, text, and events. Accordingly, we first introduce a novel event encoder that subtly models the temporal information from events and meanwhile, generates event prompts for modality bridging. We then design a text encoder that generates content prompts and utilizes hybrid text prompts to enhance EventBind's generalization ability across diverse datasets.With the proposed event encoder, text encoder, and image encoder, a novel Hierarchical Triple Contrastive Alignment (HTCA) module is introduced to jointly optimize the correlation and enable efficient knowledge transfer among the three modalities. We evaluate various settings, including fine-tuning and few-shot on three benchmarks, and our EventBind achieves new state-of-the-art accuracy compared with the previous methods, such as on N-Caltech101 (+5.34% and +1.70%) and N-Imagenet (+5.65% and +1.99%) with fine-tuning and 20-shot settings, respectively. Moreover, our EventBind can be flexibly extended to the event retrieval task using text or image queries, showing plausible performance. Project page:https://vlislab22.github.io/EventBind/., Comment: ECCV 2024 Accepted. Camera-ready version with supplementary

For Y$_2$C$_3$ with a superconducting critical temperature (T$_c$) $\sim$18 K, zone-center imaginary optical phonon modes have been found for the high-symmetry $I$-$43d$ structure due to C dimer wobbling motion and electronic instability from a flat band near Fermi energy. After lattice distortion to the more stable lowest symmetry $P1$ structure, these stabilized low-energy phonon modes with a mixed C and Y character carry a strong electron-phonon coupling to give arise to the observed sizable T$_c$. Our work shows that compounds with the calculated dynamical instability should not be simply excluded in high-throughput search for new phonon-mediated superconductors. Moreover, we have studied the phase stability of the $I$-$43d$ structure by calculating the enthalpy of different structural motifs of binary compounds containing group IV elements at the 2:3 composition and also exploring the energy landscapes via $ab$ $initio$ molecular dynamics near and out of the $I$-$43d$ structure. Our results show that the $I$-$43d$ type structures with C dimers are preferred in the low to medium pressure range. Because of the wobbling motion of the C dimers, there are many local energy minimums with degenerated energies. Thus, the ensemble average of many $I$-$43d$-distorted structures with C dimer wobbling motion at finite temperature still gives an overall $I$-$43d$ structure., Comment: 6 pages, 9 figures and 1 table

Deep Neural Networks (DNNs) for 3D point cloud recognition are vulnerable to adversarial examples, threatening their practical deployment. Despite the many research endeavors have been made to tackle this issue in recent years, the diversity of adversarial examples on 3D point clouds makes them more challenging to defend against than those on 2D images. For examples, attackers can generate adversarial examples by adding, shifting, or removing points. Consequently, existing defense strategies are hard to counter unseen point cloud adversarial examples. In this paper, we first establish a comprehensive, and rigorous point cloud adversarial robustness benchmark to evaluate adversarial robustness, which can provide a detailed understanding of the effects of the defense and attack methods. We then collect existing defense tricks in point cloud adversarial defenses and then perform extensive and systematic experiments to identify an effective combination of these tricks. Furthermore, we propose a hybrid training augmentation methods that consider various types of point cloud adversarial examples to adversarial training, significantly improving the adversarial robustness. By combining these tricks, we construct a more robust defense framework achieving an average accuracy of 83.45\% against various attacks, demonstrating its capability to enabling robust learners. Our codebase are open-sourced on: \url{https://github.com/qiufan319/benchmark_pc_attack.git}., Comment: 8 pages 6 figures

The explicit neural radiance field (NeRF) has gained considerable interest for its efficient training and fast inference capabilities, making it a promising direction such as virtual reality and gaming. In particular, PlenOctree (POT)[1], an explicit hierarchical multi-scale octree representation, has emerged as a structural and influential framework. However, POT's fixed structure for direct optimization is sub-optimal as the scene complexity evolves continuously with updates to cached color and density, necessitating refining the sampling distribution to capture signal complexity accordingly. To address this issue, we propose the dynamic PlenOctree DOT, which adaptively refines the sample distribution to adjust to changing scene complexity. Specifically, DOT proposes a concise yet novel hierarchical feature fusion strategy during the iterative rendering process. Firstly, it identifies the regions of interest through training signals to ensure adaptive and efficient refinement. Next, rather than directly filtering out valueless nodes, DOT introduces the sampling and pruning operations for octrees to aggregate features, enabling rapid parameter learning. Compared with POT, our DOT outperforms it by enhancing visual quality, reducing over $55.15$/$68.84\%$ parameters, and providing 1.7/1.9 times FPS for NeRF-synthetic and Tanks $\&$ Temples, respectively. Project homepage:https://vlislab22.github.io/DOT. [1] Yu, Alex, et al. "Plenoctrees for real-time rendering of neural radiance fields." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021., Comment: Accepted by ICCV2023

In this paper, we strive to answer the question "how to collaboratively learn convolutional neural network (CNN)-based and vision transformer (ViT)-based models by selecting and exchanging the reliable knowledge between them for semantic segmentation?" Accordingly, we propose an online knowledge distillation (KD) framework that can simultaneously learn compact yet effective CNN-based and ViT-based models with two key technical breakthroughs to take full advantage of CNNs and ViT while compensating their limitations. Firstly, we propose heterogeneous feature distillation (HFD) to improve students' consistency in low-layer feature space by mimicking heterogeneous features between CNNs and ViT. Secondly, to facilitate the two students to learn reliable knowledge from each other, we propose bidirectional selective distillation (BSD) that can dynamically transfer selective knowledge. This is achieved by 1) region-wise BSD determining the directions of knowledge transferred between the corresponding regions in the feature space and 2) pixel-wise BSD discerning which of the prediction knowledge to be transferred in the logit space. Extensive experiments on three benchmark datasets demonstrate that our proposed framework outperforms the state-of-the-art online distillation methods by a large margin, and shows its efficacy in learning collaboratively between ViT-based and CNN-based models., Comment: ICCV 2023

Integrated sensing and communication (ISAC) will greatly enhance the efficiency of physical resource utilization. The design of ISAC signal based on the orthogonal frequency division multiplex (OFDM) signal is the mainstream. However, when detecting the long-range target, the delay of echo signal exceeds CP duration, which will result in inter-symbol interference (ISI) and inter-carrier interference (ICI), limiting the sensing range. Facing the above problem, we propose to increase useful signal power through coherent compensation and improve the signal to interference plus noise power ratio (SINR) of each OFDM block. Compared with the traditional 2D-FFT algorithm, the improvement of SINR of range-doppler map (RDM) is verified by simulation, which will expand the sensing range.

The ability to scene understanding in adverse visual conditions, e.g., nighttime, has sparked active research for RGB-Thermal (RGB-T) semantic segmentation. However, it is essentially hampered by two critical problems: 1) the day-night gap of RGB images is larger than that of thermal images, and 2) the class-wise performance of RGB images at night is not consistently higher or lower than that of thermal images. we propose the first test-time adaptation (TTA) framework, dubbed Night-TTA, to address the problems for nighttime RGBT semantic segmentation without access to the source (daytime) data during adaptation. Our method enjoys three key technical parts. Firstly, as one modality (e.g., RGB) suffers from a larger domain gap than that of the other (e.g., thermal), Imaging Heterogeneity Refinement (IHR) employs an interaction branch on the basis of RGB and thermal branches to prevent cross-modal discrepancy and performance degradation. Then, Class Aware Refinement (CAR) is introduced to obtain reliable ensemble logits based on pixel-level distribution aggregation of the three branches. In addition, we also design a specific learning scheme for our TTA framework, which enables the ensemble logits and three student logits to collaboratively learn to improve the quality of predictions during the testing phase of our Night TTA. Extensive experiments show that our method achieves state-of-the-art (SoTA) performance with a 13.07% boost in mIoU.

GNN inference is a non-trivial task, especially in industrial scenarios with giant graphs, given three main challenges, i.e., scalability tailored for full-graph inference on huge graphs, inconsistency caused by stochastic acceleration strategies (e.g., sampling), and the serious redundant computation issue. To address the above challenges, we propose a scalable system named InferTurbo to boost the GNN inference tasks in industrial scenarios. Inspired by the philosophy of ``think-like-a-vertex", a GAS-like (Gather-Apply-Scatter) schema is proposed to describe the computation paradigm and data flow of GNN inference. The computation of GNNs is expressed in an iteration manner, in which a vertex would gather messages via in-edges and update its state information by forwarding an associated layer of GNNs with those messages and then send the updated information to other vertexes via out-edges. Following the schema, the proposed InferTurbo can be built with alternative backends (e.g., batch processing system or graph computing system). Moreover, InferTurbo introduces several strategies like shadow-nodes and partial-gather to handle nodes with large degrees for better load balancing. With InferTurbo, GNN inference can be hierarchically conducted over the full graph without sampling and redundant computation. Experimental results demonstrate that our system is robust and efficient for inference tasks over graphs containing some hub nodes with many adjacent edges. Meanwhile, the system gains a remarkable performance compared with the traditional inference pipeline, and it can finish a GNN inference task over a graph with tens of billions of nodes and hundreds of billions of edges within 2 hours., Comment: Accepted by ICDE 2023