Search

Predicting genetic mutations from whole slide images is indispensable for cancer diagnosis. However, existing work training multiple binary classification models faces two challenges: (a) Training multiple binary classifiers is inefficient and would inevitably lead to a class imbalance problem. (b) The biological relationships among genes are overlooked, which limits the prediction performance. To tackle these challenges, we innovatively design a Biological-knowledge enhanced PathGenomic multi-label Transformer to improve genetic mutation prediction performances. BPGT first establishes a novel gene encoder that constructs gene priors by two carefully designed modules: (a) A gene graph whose node features are the genes' linguistic descriptions and the cancer phenotype, with edges modeled by genes' pathway associations and mutation consistencies. (b) A knowledge association module that fuses linguistic and biomedical knowledge into gene priors by transformer-based graph representation learning, capturing the intrinsic relationships between different genes' mutations. BPGT then designs a label decoder that finally performs genetic mutation prediction by two tailored modules: (a) A modality fusion module that firstly fuses the gene priors with critical regions in WSIs and obtains gene-wise mutation logits. (b) A comparative multi-label loss that emphasizes the inherent comparisons among mutation status to enhance the discrimination capabilities. Sufficient experiments on The Cancer Genome Atlas benchmark demonstrate that BPGT outperforms the state-of-the-art., Comment: 16 pages, 8 figures, and 3 tables

Understanding the real world through point cloud video is a crucial aspect of robotics and autonomous driving systems. However, prevailing methods for 4D point cloud recognition have limitations due to sensor resolution, which leads to a lack of detailed information. Recent advances have shown that Vision-Language Models (VLM) pre-trained on web-scale text-image datasets can learn fine-grained visual concepts that can be transferred to various downstream tasks. However, effectively integrating VLM into the domain of 4D point clouds remains an unresolved problem. In this work, we propose the Vision-Language Models Goes 4D (VG4D) framework to transfer VLM knowledge from visual-text pre-trained models to a 4D point cloud network. Our approach involves aligning the 4D encoder's representation with a VLM to learn a shared visual and text space from training on large-scale image-text pairs. By transferring the knowledge of the VLM to the 4D encoder and combining the VLM, our VG4D achieves improved recognition performance. To enhance the 4D encoder, we modernize the classic dynamic point cloud backbone and propose an improved version of PSTNet, im-PSTNet, which can efficiently model point cloud videos. Experiments demonstrate that our method achieves state-of-the-art performance for action recognition on both the NTU RGB+D 60 dataset and the NTU RGB+D 120 dataset. Code is available at \url{https://github.com/Shark0-0/VG4D}., Comment: ICRA 2024

We report experimental and theoretical studies of MoTe$_2$-MoSe$_2$ heterobilayers with rigid moir\'e superlattices controlled by the twist angle. Using an effective continuum model that combines resonant interlayer electron tunneling with stacking-dependent moir\'e potentials, we identify the nature of moir\'e excitons and the dependence of their energies, oscillator strengths and Land\'e $g$-factors on the twist angle. Within the same framework, we interpret distinct signatures of bound complexes among electrons and moir\'e excitons in nearly collinear heterostacks. Our work provides fundamental understanding of hybrid moir\'e excitons and trions in MoTe$_2$-MoSe$_2$ heterobilayers, and establishes the material system as a prime candidate for optical studies of correlated phenomena in moir\'e lattices.

The radio-frequency quadrupole (RFQ) at the Facility for Rare Isotope Beams (FRIB) is a critical device to accelerate heavy ion beams from 12 keV/u to 0.5 MeV/u for state-of-the-art nuclear physics experiments. Efficient control of the RFQ resonance frequency detuning still remains a challenge because the temperature-sensitive frequency is solely control by a cooling water system, exhibiting complicated transport delay and nonlinearity in the heat transfer processes. In this work, we propose a long-short term memory (LSTM)-based Koopman network model that can simultaneously learn the time-delayed and non-delayed correlations hidden in the historical operating data. It is proven that the model can effectively predict the behavior of the RFQ resonance frequency using historical data as inputs. With this model, a model predictive control (MPC) framework based on the Newton-Raphson method is proposed and tested. We demonstrate that the MPC framework utilizing deep learning model is able to provide precise and rapid control for the RFQ frequency detuning, reducing the control time by half compared to the proportional-integral-derivative (PID) controller.

Multimodal-based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, we introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts, while filtering out most irrelevant noise. We propose a new dual-branch framework called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing festures via convolutional backbones. The two high-level features will be effectively combined through a late fusion strategy for better action recognition. Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently verify the effectiveness of our proposed EPP-Net, which outperforms the existing action recognition methods. Our code is available at: https://github.com/liujf69/EPP-Net-Action., Comment: arXiv admin note: text overlap with arXiv:2307.07977

We report on the first detailed beam test attesting the fundamental principle behind the development of high-current-efficiency ultrafast electron microscope systems where a radio-frequency cavity is incorporated as a condenser lens in the beam delivery system. To allow the experiment to be carried out with a sufficient resolution to probe the performance at the emittance floor, a new cascade loop RF controller system is developed to reduce the RF noise floor. Temporal resolution at 50 femtoseconds in full-width-at-half-maximum and detection sensitivity better than 1% are demonstrated on exfoliated 1T-TaSe$_2$ layers where the multi-order edge-mode coherent phonon excitation is employed as the standard candle to benchmark the performance. The high temporal resolution and the significant visibility to very low dynamical contrast in diffraction signals give strong support to the working principle of the high-brightness beam delivery via phase-space manipulation in the electron microscope system.

Deep neural networks (DNNs) have been deployed for many image segmentation tasks and achieved outstanding performance. However, preparing a dataset for training segmentation DNNs is laborious and costly since typically pixel-level annotations are provided for each object of interest. To alleviate this issue, one can provide only weak labels such as bounding boxes or scribbles, or less accurate (noisy) annotations of the objects. These are significantly faster to generate and thus result in more annotated images given the same time budget. However, the reduction in quality might negatively affect the segmentation performance of the resulting model. In this study, we perform a thorough cost-effectiveness evaluation of several weak and noisy labels. We considered 11 variants of annotation strategies and 4 datasets. We conclude that the common practice of accurately outlining the objects of interest is virtually never the optimal approach when the annotation time is limited, even if notable annotation time is available (10s of hours). Annotation approaches that stood out in such scenarios were (1) contour-based annotation with rough continuous traces, (2) polygon-based annotation with few vertices, and (3) box annotations combined with the Segment Anything Model (SAM). In situations where unlimited annotation time was available, precise annotations still lead to the highest segmentation model performance.

With potential applications in fields including intelligent surveillance and human-robot interaction, the human motion prediction task has become a hot research topic and also has achieved high success, especially using the recent Graph Convolutional Network (GCN). Current human motion prediction task usually focuses on predicting human motions for atomic actions. Observing that atomic actions can happen at the same time and thus formulating the composite actions, we propose the composite human motion prediction task. To handle this task, we first present a Composite Action Generation (CAG) module to generate synthetic composite actions for training, thus avoiding the laborious work of collecting composite action samples. Moreover, we alleviate the effect of composite actions on demand for a more complicated model by presenting a Dynamic Compositional Graph Convolutional Network (DC-GCN). Extensive experiments on the Human3.6M dataset and our newly collected CHAMP dataset consistently verify the efficiency of our DC-GCN method, which achieves state-of-the-art motion prediction accuracies and meanwhile needs few extra computational costs than traditional GCN-based human motion methods.

Exciton-exciton interactions are key to understanding non-linear optical and transport phenomena in van der Waals heterobilayers, which emerged as versatile platforms to study correlated electronic states. We present a combined theory-experiment study of excitonic many-body effects based on first-principle band structures and Coulomb interaction matrix elements. Key to our approach is the explicit treatment of the fermionic substructure of excitons and dynamical screening effects for density-induced energy renormalization and dissipation. We demonstrate that dipolar blue shifts are almost perfectly compensated by many-body effects, mainly by screening-induced self-energy corrections. Moreover, we identify a crossover between attractive and repulsive behavior at elevated exciton densities. Theoretical findings are supported by experimental studies of spectrally-narrow interlayer excitons in atomically-reconstructed, hBN-encapsulated MoSe$_2$/WSe$_2$ heterobilayers. Both theory and experiment show energy renormalization on a scale of a few meV even for high injection densities in the vicinity of the Mott transition. Our results revise the established picture of dipolar repulsion dominating exciton-exciton interactions in van der Waals heterostructures and open up opportunities for their external design.

Cross-modal pre-training has shown impressive performance on a wide range of downstream tasks, benefiting from massive image-text pairs collected from the Internet. In practice, online data are growing constantly, highlighting the importance of the ability of pre-trained model to learn from data that is continuously growing. Existing works on cross-modal pre-training mainly focus on training a network with fixed architecture. However, it is impractical to limit the model capacity when considering the continuously growing nature of pre-training data in real-world applications. On the other hand, it is important to utilize the knowledge in the current model to obtain efficient training and better performance. To address the above issues, in this paper, we propose GrowCLIP, a data-driven automatic model growing algorithm for contrastive language-image pre-training with continuous image-text pairs as input. Specially, we adopt a dynamic growth space and seek out the optimal architecture at each growth step to adapt to online learning scenarios. And the shared encoder is proposed in our growth space to enhance the degree of cross-modal fusion. Besides, we explore the effect of growth in different dimensions, which could provide future references for the design of cross-modal model architecture. Finally, we employ parameter inheriting with momentum (PIM) to maintain the previous knowledge and address the issue of the local minimum dilemma. Compared with the existing methods, GrowCLIP improves 2.3% average top-1 accuracy on zero-shot image classification of 9 downstream tasks. As for zero-shot image retrieval, GrowCLIP can improve 1.2% for top-1 image-to-text recall on Flickr30K dataset., Comment: Accepted by ICCV2023

Existing Graph Convolutional Networks to achieve human motion prediction largely adopt a one-step scheme, which output the prediction straight from history input, failing to exploit human motion patterns. We observe that human motions have transitional patterns and can be split into snippets representative of each transition. Each snippet can be reconstructed from its starting and ending poses referred to as the transitional poses. We propose a snippet-to-motion multi-stage framework that breaks motion prediction into sub-tasks easier to accomplish. Each sub-task integrates three modules: transitional pose prediction, snippet reconstruction, and snippet-to-motion prediction. Specifically, we propose to first predict only the transitional poses. Then we use them to reconstruct the corresponding snippets, obtaining a close approximation to the true motion sequence. Finally we refine them to produce the final prediction output. To implement the network, we propose a novel unified graph modeling, which allows for direct and effective feature propagation compared to existing approaches which rely on separate space-time modeling. Extensive experiments on Human 3.6M, CMU Mocap and 3DPW datasets verify the effectiveness of our method which achieves state-of-the-art performance.

Medical artificial general intelligence (MAGI) enables one foundation model to solve different medical tasks, which is very practical in the medical domain. It can significantly reduce the requirement of large amounts of task-specific data by sufficiently sharing medical knowledge among different tasks. However, due to the challenges of designing strongly generalizable models with limited and complex medical data, most existing approaches tend to develop task-specific models. To take a step towards MAGI, we propose a new paradigm called Medical-knOwledge-enhanced mulTimOdal pretRaining (MOTOR). In MOTOR, we combine two kinds of basic medical knowledge, i.e., general and specific knowledge, in a complementary manner to boost the general pretraining process. As a result, the foundation model with comprehensive basic knowledge can learn compact representations from pretraining radiographic data for better cross-modal alignment. MOTOR unifies the understanding and generation, which are two kinds of core intelligence of an AI system, into a single medical foundation model, to flexibly handle more diverse medical tasks. To enable a comprehensive evaluation and facilitate further research, we construct a medical multimodal benchmark including a wide range of downstream tasks, such as chest x-ray report generation and medical visual question answering. Extensive experiments on our benchmark show that MOTOR obtains promising results through simple task-oriented adaptation. The visualization shows that the injected knowledge successfully highlights key information in the medical data, demonstrating the excellent interpretability of MOTOR. Our MOTOR successfully mimics the human practice of fulfilling a "medical student" to accelerate the process of becoming a "specialist". We believe that our work makes a significant stride in realizing MAGI., Comment: Project page: https://github.com/chenzcv7/MOTOR

Their outstanding electrical and optical properties make semiconducting single-walled carbon nanotubes (SWCNTs) highly suitable for charge transport and emissive layers in near-infrared optoelectronic devices. However, the luminescence spectra of SWCNT thin films on commonly used glass and Si/SiO2 substrates are often compromised by broadening of the main excitonic emission and unwanted low-energy sidebands. Surface passivation with a commercially available, low dielectric constant, cross-linked bis-benzocyclobutene-based polymer (BCB) enhances the emission properties of SWCNTs to the same level as hexagonal boron nitride (h-BN) flakes do. The presence of BCB suppresses sideband emission, especially from the Y1 band, which is attributed to defects introduced by the interaction of the nanotube lattice with oxygen-containing terminal groups of the substrate surface. The facile and reproducible deposition of homogeneous BCB films over large areas combined with their resistance against common solvents and chemicals employed during photolithography make them compatible with standard semiconductor device fabrication. Utilizing this approach, light-emitting (6,5) SWCNT network field-effect transistors are fabricated on BCB-treated glass substrates with excellent electrical characteristics and near-intrinsic electroluminescence. Hence, passivation with BCB is proposed as a standard treatment for substrates used for spectroscopic investigations of and optoelectronic devices with SWCNTs and other low-dimensional emitters.

In recent years, Graph Convolutional Networks (GCNs) have been widely used in human motion prediction, but their performance remains unsatisfactory. Recently, MLP-Mixer, initially developed for vision tasks, has been leveraged into human motion prediction as a promising alternative to GCNs, which achieves both better performance and better efficiency than GCNs. Unlike GCNs, which can explicitly capture human skeleton's bone-joint structure by representing it as a graph with edges and nodes, MLP-Mixer relies on fully connected layers and thus cannot explicitly model such graph-like structure of human's. To break this limitation of MLP-Mixer's, we propose \textit{Graph-Guided Mixer}, a novel approach that equips the original MLP-Mixer architecture with the capability to model graph structure. By incorporating graph guidance, our \textit{Graph-Guided Mixer} can effectively capture and utilize the specific connectivity patterns within human skeleton's graph representation. In this paper, first we uncover a theoretical connection between MLP-Mixer and GCN that is unexplored in existing research. Building on this theoretical connection, next we present our proposed \textit{Graph-Guided Mixer}, explaining how the original MLP-Mixer architecture is reinvented to incorporate guidance from graph structure. Then we conduct an extensive evaluation on the Human3.6M, AMASS, and 3DPW datasets, which shows that our method achieves state-of-the-art performance.

Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding medical reports according to the given radiology image. However, due to the spurious correlations within image-text data induced by visual and linguistic biases, it is challenging to generate accurate reports reliably describing lesion areas. Moreover, the cross-modal confounders are usually unobservable and challenging to be eliminated explicitly. In this paper, we aim to mitigate the cross-modal data bias for MRG from a new perspective, i.e., cross-modal causal intervention, and propose a novel Visual-Linguistic Causal Intervention (VLCI) framework for MRG, which consists of a visual deconfounding module (VDM) and a linguistic deconfounding module (LDM), to implicitly mitigate the visual-linguistic confounders by causal front-door intervention. Specifically, due to the absence of a generalized semantic extractor, the VDM explores and disentangles the visual confounders from the patch-based local and global features without expensive fine-grained annotations. Simultaneously, due to the lack of knowledge encompassing the entire field of medicine, the LDM eliminates the linguistic confounders caused by salient visual features and high-frequency context without constructing a terminology database. Extensive experiments on IU-Xray and MIMIC-CXR datasets show that our VLCI significantly outperforms the state-of-the-art MRG methods. The code and models are available at https://github.com/WissingChen/VLCI.

Benefiting from large-scale vision-language pre-training on image-text pairs, open-world detection methods have shown superior generalization ability under the zero-shot or few-shot detection settings. However, a pre-defined category space is still required during the inference stage of existing methods and only the objects belonging to that space will be predicted. To introduce a "real" open-world detector, in this paper, we propose a novel method named CapDet to either predict under a given category list or directly generate the category of predicted bounding boxes. Specifically, we unify the open-world detection and dense caption tasks into a single yet effective framework by introducing an additional dense captioning head to generate the region-grounded captions. Besides, adding the captioning task will in turn benefit the generalization of detection performance since the captioning dataset covers more concepts. Experiment results show that by unifying the dense caption task, our CapDet has obtained significant performance improvements (e.g., +2.1% mAP on LVIS rare classes) over the baseline method on LVIS (1203 classes). Besides, our CapDet also achieves state-of-the-art performance on dense captioning tasks, e.g., 15.44% mAP on VG V1.2 and 13.98% on the VG-COCO dataset., Comment: Accepted by CVPR2023

Vertical van der Waals heterostructures of semiconducting transition metal dichalcogenides realize moir\'e systems with rich correlated electron phases and moir\'e exciton phenomena. For material combinations with small lattice mismatch and twist angles as in MoSe$_2$-WSe$_2$, however, lattice reconstruction eliminates the canonical moir\'e pattern and instead gives rise to arrays of periodically reconstructed nanoscale domains and mesoscopically extended areas of one atomic registry. Here, we elucidate the role of atomic reconstruction in MoSe$_2$-WSe$_2$ heterostructures synthesized by chemical vapor deposition. With complementary imaging down to the atomic scale, simulations, and optical spectroscopy methods we identify the coexistence of moir\'e-type cores and extended moir\'e-free regions in heterostacks with parallel and antiparallel alignment. Our work highlights the potential of chemical vapor deposition for applications requiring laterally extended heterosystems of one atomic registry or exciton-confining heterostack arrays., Comment: 10 pages, 5 figures

Imbalanced training data is a significant challenge for medical image classification. In this study, we propose a novel Progressive Class-Center Triplet (PCCT) framework to alleviate the class imbalance issue particularly for diagnosis of rare diseases, mainly by carefully designing the triplet sampling strategy and the triplet loss formation. Specifically, the PCCT framework includes two successive stages. In the first stage, PCCT trains the diagnosis system via a class-balanced triplet loss to coarsely separate distributions of different classes. In the second stage, the PCCT framework further improves the diagnosis system via a class-center involved triplet loss to cause a more compact distribution for each class. For the class-balanced triplet loss, triplets are sampled equally for each class at each training iteration, thus alleviating the imbalanced data issue. For the class-center involved triplet loss, the positive and negative samples in each triplet are replaced by their corresponding class centers, which enforces data representations of the same class closer to the class center. Furthermore, the class-center involved triplet loss is extended to the pair-wise ranking loss and the quadruplet loss, which demonstrates the generalization of the proposed framework. Extensive experiments support that the PCCT framework works effectively for medical image classification with imbalanced training images. On two skin image datasets and one chest X-ray dataset, the proposed approach respectively obtains the mean F1 score 86.2, 65.2, and 90.66 over all classes and 81.4, 63.87, and 81.92 for rare classes, achieving state-of-the-art performance and outperforming the widely used methods for the class imbalance issue.

Continual learning is a challenging real-world problem for constructing a mature AI system when data are provided in a streaming fashion. Despite recent progress in continual classification, the researches of continual object detection are impeded by the diverse sizes and numbers of objects in each image. Different from previous works that tune the whole network for all tasks, in this work, we present a simple and flexible framework for continual object detection via pRotOtypical taSk corrElaTion guided gaTing mechAnism (ROSETTA). Concretely, a unified framework is shared by all tasks while task-aware gates are introduced to automatically select sub-models for specific tasks. In this way, various knowledge can be successively memorized by storing their corresponding sub-model weights in this system. To make ROSETTA automatically determine which experience is available and useful, a prototypical task correlation guided Gating Diversity Controller(GDC) is introduced to adaptively adjust the diversity of gates for the new task based on class-specific prototypes. GDC module computes class-to-class correlation matrix to depict the cross-task correlation, and hereby activates more exclusive gates for the new task if a significant domain gap is observed. Comprehensive experiments on COCO-VOC, KITTI-Kitchen, class-incremental detection on VOC and sequential learning of four tasks show that ROSETTA yields state-of-the-art performance on both task-based and class-based continual object detection.

Moir\'e effects in twisted or lattice-incommensurate vertical assemblies of two-dimensional crystals give rise to a new class of quantum materials with rich transport and optical phenomena, including correlated electron physics in flat bands of bilayer graphene and moir\'e excitons in semiconductor heterostructures. These phenomena arise from modulations of interlayer hybridization on the nanoscale of spatially varying atomic registries of moir\'e supercells. Due to finite elasticity, however, lattices of marginally-twisted homobilayers and heterostructures can transform from moir\'e to periodically reconstructed patterns with triangular or hexagonal tiling. Here, we expand the notion of nanoscale lattice reconstruction to the mesoscopic scale of extended samples and demonstrate rich consequences in optical studies of excitons in MoSe$_2$-WSe$_2$ heterostructures with parallel and antiparallel alignment. Our results provide a unified perspective on diverse and partly controversial signatures of moir\'e excitons in semiconductor heterostructures by identifying domains with exciton properties of distinct effective dimensionality and establish mesoscopic reconstruction as a compelling feature of real samples and devices with inherent finite-size effects and disorder. Generalized to stacks of other two-dimensional materials, this notion of mesoscale domain formation with emergent topological defects and percolation networks will instructively expand our understanding of fundamental electronic, optical, and magnetic properties of van der Waals heterostructures.

In phase-contrast magnetic resonance imaging (PC-MRI), the velocity of spins at a voxel is encoded in the image phase. The strength of the velocity encoding gradient offers a trade-off between the velocity-to-noise ratio (VNR) and the extent of phase aliasing. Phase differences provide invariance to an unknown background phase. Existing literature proposes processing a reduced set of phase difference equations, simplifying the phase unwrapping problem at the expense of VNR or unaliased range of velocities, or both. Here, we demonstrate that the fullest unambiguous range of velocities is a parallelepiped, which can be accessed by jointly processing all phase differences. The joint processing also minimizes the velocity-to-noise ratio. The simple understanding of the unambiguous parallelepiped provides the potential for analyzing new multi-point acquisitions for an enhanced range of unaliased velocities; two examples are given., Comment: 5 pages, 5 figures

In phase-contrast magnetic resonance imaging (PC-MRI), spin velocity contributes to the phase measured at each voxel. Therefore, estimating velocity from potentially wrapped phase measurements is the task of solving a system of noisy congruence equations. We propose Phase Recovery from Multiple Wrapped Measurements (PRoM) as a fast, approximate maximum likelihood estimator of velocity from multi-coil data with possible amplitude attenuation due to dephasing. The estimator can recover the fullest possible extent of unambiguous velocities, which can greatly exceed twice the highest venc. The estimator uses all pairwise phase differences and the inherent correlations among them to minimize the estimation error. Correlations are directly estimated from multi-coil data without requiring knowledge of coil sensitivity maps, dephasing factors, or the actual per-voxel signal-to-noise ratio. Derivation of the estimator yields explicit probabilities of unwrapping errors and the probability distribution for the velocity estimate; this, in turn, allows for optimized design of the phase-encoded acquisition. These probabilities are also incorporated into spatial post-processing to further mitigate wrapping errors. Simulation, phantom, and in vivo results for three-point PC-MRI acquisitions validate the benefits of reduced estimation error, increased recovered velocity range, optimized acquisition, and fast computation. A phantom study at 1.5T demonstrates 48.5% decrease in root mean squared error using PRoM with post-processing versus a conventional "dual-venc" technique. Simulation and 3T in vivo results likewise demonstrate the proposed benefits., Comment: 13 pages, 11 figures