Search

Panoramic X-ray (PX) is a prevalent modality in dental practice for its wide availability and low cost. However, as a 2D projection image, PX does not contain 3D anatomical information, and therefore has limited use in dental applications that can benefit from 3D information, e.g., tooth angular misa-lignment detection and classification. Reconstructing 3D structures directly from 2D PX has recently been explored to address limitations with existing methods primarily reliant on Convolutional Neural Networks (CNNs) for direct 2D-to-3D mapping. These methods, however, are unable to correctly infer depth-axis spatial information. In addition, they are limited by the in-trinsic locality of convolution operations, as the convolution kernels only capture the information of immediate neighborhood pixels. In this study, we propose a progressive hybrid Multilayer Perceptron (MLP)-CNN pyra-mid network (3DPX) for 2D-to-3D oral PX reconstruction. We introduce a progressive reconstruction strategy, where 3D images are progressively re-constructed in the 3DPX with guidance imposed on the intermediate recon-struction result at each pyramid level. Further, motivated by the recent ad-vancement of MLPs that show promise in capturing fine-grained long-range dependency, our 3DPX integrates MLPs and CNNs to improve the semantic understanding during reconstruction. Extensive experiments on two large datasets involving 464 studies demonstrate that our 3DPX outperforms state-of-the-art 2D-to-3D oral reconstruction methods, including standalone MLP and transformers, in reconstruction quality, and also im-proves the performance of downstream angular misalignment classification tasks., Comment: accepted by MICCAI 2024

Deformable image registration is a fundamental step for medical image analysis. Recently, transformers have been used for registration and outperformed Convolutional Neural Networks (CNNs). Transformers can capture long-range dependence among image features, which have been shown beneficial for registration. However, due to the high computation/memory loads of self-attention, transformers are typically used at downsampled feature resolutions and cannot capture fine-grained long-range dependence at the full image resolution. This limits deformable registration as it necessitates precise dense correspondence between each image pixel. Multi-layer Perceptrons (MLPs) without self-attention are efficient in computation/memory usage, enabling the feasibility of capturing fine-grained long-range dependence at full resolution. Nevertheless, MLPs have not been extensively explored for image registration and are lacking the consideration of inductive bias crucial for medical registration tasks. In this study, we propose the first correlation-aware MLP-based registration network (CorrMLP) for deformable medical image registration. Our CorrMLP introduces a correlation-aware multi-window MLP block in a novel coarse-to-fine registration architecture, which captures fine-grained multi-range dependence to perform correlation-aware coarse-to-fine registration. Extensive experiments with seven public medical datasets show that our CorrMLP outperforms state-of-the-art deformable registration methods., Comment: Accepted at CVPR2024 as Oral Presentation && Best Paper Candidate

Medical image representations can be learned through medical vision-language contrastive learning (mVLCL) where medical imaging reports are used as weak supervision through image-text alignment. These learned image representations can be transferred to and benefit various downstream medical vision tasks such as disease classification and segmentation. Recent mVLCL methods attempt to align image sub-regions and the report keywords as local-matchings. However, these methods aggregate all local-matchings via simple pooling operations while ignoring the inherent relations between them. These methods therefore fail to reason between local-matchings that are semantically related, e.g., local-matchings that correspond to the disease word and the location word (semantic-relations), and also fail to differentiate such clinically important local-matchings from others that correspond to less meaningful words, e.g., conjunction words (importance-relations). Hence, we propose a mVLCL method that models the inter-matching relations between local-matchings via a relation-enhanced contrastive learning framework (RECLF). In RECLF, we introduce a semantic-relation reasoning module (SRM) and an importance-relation reasoning module (IRM) to enable more fine-grained report supervision for image representation learning. We evaluated our method using four public benchmark datasets on four downstream tasks, including segmentation, zero-shot classification, supervised classification, and cross-modal retrieval. Our results demonstrated the superiority of our RECLF over the state-of-the-art mVLCL methods with consistent improvements across single-modal and cross-modal tasks. These results suggest that our RECLF, by modelling the inter-matching relations, can learn improved medical image representations with better generalization capabilities., Comment: 11 pages, 5 figures. Under review

The clinical diagnosis of skin lesion involves the analysis of dermoscopic and clinical modalities. Dermoscopic images provide a detailed view of the surface structures whereas clinical images offer a complementary macroscopic information. The visual diagnosis of melanoma is also based on seven-point checklist which involves identifying different visual attributes. Recently, supervised learning approaches such as convolutional neural networks (CNNs) have shown great performances using both dermoscopic and clinical modalities (Multi-modality). The seven different visual attributes in the checklist are also used to further improve the the diagnosis. The performances of these approaches, however, are still reliant on the availability of large-scaled labeled data. The acquisition of annotated dataset is an expensive and time-consuming task, more so with annotating multi-attributes. To overcome this limitation, we propose a self-supervised learning (SSL) algorithm for multi-modality skin lesion classification. Our algorithm enables the multi-modality learning by maximizing the similarities between paired dermoscopic and clinical images from different views. In addition, we generate surrogate pseudo-multi-labels that represent seven attributes via clustering analysis. We also propose a label-relation-aware module to refine each pseudo-label embedding and capture the interrelationships between pseudo-multi-labels. We validated the effectiveness of our algorithm using well-benchmarked seven-point skin lesion dataset. Our results show that our algorithm achieved better performances than other state-of-the-art SSL counterparts.

Positron emission tomography (PET) is a widely used, highly sensitive molecular imaging in clinical diagnosis. There is interest in reducing the radiation exposure from PET but also maintaining adequate image quality. Recent methods using convolutional neural networks (CNNs) to generate synthesized high-quality PET images from low-dose counterparts have been reported to be state-of-the-art for low-to-high image recovery methods. However, these methods are prone to exhibiting discrepancies in texture and structure between synthesized and real images. Furthermore, the distribution shift between low-dose PET and standard PET has not been fully investigated. To address these issues, we developed a self-supervised adaptive residual estimation generative adversarial network (SS-AEGAN). We introduce (1) An adaptive residual estimation mapping mechanism, AE-Net, designed to dynamically rectify the preliminary synthesized PET images by taking the residual map between the low-dose PET and synthesized output as the input, and (2) A self-supervised pre-training strategy to enhance the feature representation of the coarse generator. Our experiments with a public benchmark dataset of total-body PET images show that SS-AEGAN consistently outperformed the state-of-the-art synthesis methods with various dose reduction factors., Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Deformable image registration aims to find a dense non-linear spatial correspondence between a pair of images, which is a crucial step for many medical tasks such as tumor growth monitoring and population analysis. Recently, Deep Neural Networks (DNNs) have been widely recognized for their ability to perform fast end-to-end registration. However, DNN-based registration needs to explore the spatial information of each image and fuse this information to characterize spatial correspondence. This raises an essential question: what is the optimal fusion strategy to characterize spatial correspondence? Existing fusion strategies (e.g., early fusion, late fusion) were empirically designed to fuse information by manually defined prior knowledge, which inevitably constrains the registration performance within the limits of empirical designs. In this study, we depart from existing empirically-designed fusion strategies and develop a data-driven fusion strategy for deformable image registration. To achieve this, we propose an Automatic Fusion network (AutoFuse) that provides flexibility to fuse information at many potential locations within the network. A Fusion Gate (FG) module is also proposed to control how to fuse information at each potential network location based on training data. Our AutoFuse can automatically optimize its fusion strategy during training and can be generalizable to both unsupervised registration (without any labels) and semi-supervised registration (with weak labels provided for partial training data). Extensive experiments on two well-benchmarked medical registration tasks (inter- and intra-patient registration) with eight public datasets show that our AutoFuse outperforms state-of-the-art unsupervised and semi-supervised registration methods., Comment: Under Review

Survival prediction is crucial for cancer patients as it provides early prognostic information for treatment planning. Recently, deep survival models based on deep learning and medical images have shown promising performance for survival prediction. However, existing deep survival models are not well developed in utilizing multi-modality images (e.g., PET-CT) and in extracting region-specific information (e.g., the prognostic information in Primary Tumor (PT) and Metastatic Lymph Node (MLN) regions). In view of this, we propose a merging-diverging learning framework for survival prediction from multi-modality images. This framework has a merging encoder to fuse multi-modality information and a diverging decoder to extract region-specific information. In the merging encoder, we propose a Hybrid Parallel Cross-Attention (HPCA) block to effectively fuse multi-modality features via parallel convolutional layers and cross-attention transformers. In the diverging decoder, we propose a Region-specific Attention Gate (RAG) block to screen out the features related to lesion regions. Our framework is demonstrated on survival prediction from PET-CT images in Head and Neck (H&N) cancer, by designing an X-shape merging-diverging hybrid transformer network (named XSurv). Our XSurv combines the complementary information in PET and CT images and extracts the region-specific prognostic information in PT and MLN regions. Extensive experiments on the public dataset of HEad and neCK TumOR segmentation and outcome prediction challenge (HECKTOR 2022) demonstrate that our XSurv outperforms state-of-the-art survival prediction methods., Comment: Early Accepted at International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023)

Image registration is a fundamental requirement for medical image analysis. Deep registration methods based on deep learning have been widely recognized for their capabilities to perform fast end-to-end registration. Many deep registration methods achieved state-of-the-art performance by performing coarse-to-fine registration, where multiple registration steps were iterated with cascaded networks. Recently, Non-Iterative Coarse-to-finE (NICE) registration methods have been proposed to perform coarse-to-fine registration in a single network and showed advantages in both registration accuracy and runtime. However, existing NICE registration methods mainly focus on deformable registration, while affine registration, a common prerequisite, is still reliant on time-consuming traditional optimization-based methods or extra affine registration networks. In addition, existing NICE registration methods are limited by the intrinsic locality of convolution operations. Transformers may address this limitation for their capabilities to capture long-range dependency, but the benefits of using transformers for NICE registration have not been explored. In this study, we propose a Non-Iterative Coarse-to-finE Transformer network (NICE-Trans) for image registration. Our NICE-Trans is the first deep registration method that (i) performs joint affine and deformable coarse-to-fine registration within a single network, and (ii) embeds transformers into a NICE registration framework to model long-range relevance between images. Extensive experiments with seven public datasets show that our NICE-Trans outperforms state-of-the-art registration methods on both registration accuracy and runtime., Comment: Accepted at International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023)

Survival prediction is a major concern for cancer management. Deep survival models based on deep learning have been widely adopted to perform end-to-end survival prediction from medical images. Recent deep survival models achieved promising performance by jointly performing tumor segmentation with survival prediction, where the models were guided to extract tumor-related information through Multi-Task Learning (MTL). However, these deep survival models have difficulties in exploring out-of-tumor prognostic information. In addition, existing deep survival models are unable to effectively leverage multi-modality images. Empirically-designed fusion strategies were commonly adopted to fuse multi-modality information via task-specific manually-designed networks, thus limiting the adaptability to different scenarios. In this study, we propose an Adaptive Multi-modality Segmentation-to-Survival model (AdaMSS) for survival prediction from PET/CT images. Instead of adopting MTL, we propose a novel Segmentation-to-Survival Learning (SSL) strategy, where our AdaMSS is trained for tumor segmentation and survival prediction sequentially in two stages. This strategy enables the AdaMSS to focus on tumor regions in the first stage and gradually expand its focus to include other prognosis-related regions in the second stage. We also propose a data-driven strategy to fuse multi-modality information, which realizes adaptive optimization of fusion strategies based on training data during training. With the SSL and data-driven fusion strategies, our AdaMSS is designed as an adaptive model that can self-adapt its focus regions and fusion strategy for different training stages. Extensive experiments with two large clinical datasets show that our AdaMSS outperforms state-of-the-art survival prediction methods., Comment: Under Review

Positron emission tomography (PET) is the most sensitive molecular imaging modality routinely applied in our modern healthcare. High radioactivity caused by the injected tracer dose is a major concern in PET imaging and limits its clinical applications. However, reducing the dose leads to inadequate image quality for diagnostic practice. Motivated by the need to produce high quality images with minimum low-dose, Convolutional Neural Networks (CNNs) based methods have been developed for high quality PET synthesis from its low-dose counterparts. Previous CNNs-based studies usually directly map low-dose PET into features space without consideration of different dose reduction level. In this study, a novel approach named CG-3DSRGAN (Classification-Guided Generative Adversarial Network with Super Resolution Refinement) is presented. Specifically, a multi-tasking coarse generator, guided by a classification head, allows for a more comprehensive understanding of the noise-level features present in the low-dose data, resulting in improved image synthesis. Moreover, to recover spatial details of standard PET, an auxiliary super resolution network - Contextual-Net - is proposed as a second-stage training to narrow the gap between coarse prediction and standard PET. We compared our method to the state-of-the-art methods on whole-body PET with different dose reduction factors (DRFs). Experiments demonstrate our method can outperform others on all DRF.

In this study, we focus on brain tumor sequence registration between pre-operative and follow-up Magnetic Resonance Imaging (MRI) scans of brain glioma patients, in the context of Brain Tumor Sequence Registration challenge (BraTS-Reg 2022). Brain tumor registration is a fundamental requirement in brain image analysis for quantifying tumor changes. This is a challenging task due to large deformations and missing correspondences between pre-operative and follow-up scans. For this task, we adopt our recently proposed Non-Iterative Coarse-to-finE registration Networks (NICE-Net) - a deep learning-based method for coarse-to-fine registering images with large deformations. To overcome missing correspondences, we extend the NICE-Net by introducing dual deep supervision, where a deep self-supervised loss based on image similarity and a deep weakly-supervised loss based on manually annotated landmarks are deeply embedded into the NICE-Net. At the BraTS-Reg 2022, our method achieved a competitive result on the validation set (mean absolute error: 3.387) and placed 4th in the final testing phase (Score: 0.3544)., Comment: Brain Tumor Sequence Registration challenge (BraTS-Reg 2022)

Outcome prediction is crucial for head and neck cancer patients as it can provide prognostic information for early treatment planning. Radiomics methods have been widely used for outcome prediction from medical images. However, these methods are limited by their reliance on intractable manual segmentation of tumor regions. Recently, deep learning methods have been proposed to perform end-to-end outcome prediction so as to remove the reliance on manual segmentation. Unfortunately, without segmentation masks, these methods will take the whole image as input, such that makes them difficult to focus on tumor regions and potentially unable to fully leverage the prognostic information within the tumor regions. In this study, we propose a radiomics-enhanced deep multi-task framework for outcome prediction from PET/CT images, in the context of HEad and neCK TumOR segmentation and outcome prediction challenge (HECKTOR 2022). In our framework, our novelty is to incorporate radiomics as an enhancement to our recently proposed Deep Multi-task Survival model (DeepMTS). The DeepMTS jointly learns to predict the survival risk scores of patients and the segmentation masks of tumor regions. Radiomics features are extracted from the predicted tumor regions and combined with the predicted survival risk scores for final outcome prediction, through which the prognostic information in tumor regions can be further leveraged. Our method achieved a C-index of 0.681 on the testing set, placing the 2nd on the leaderboard with only 0.00068 lower in C-index than the 1st place., Comment: HEad and neCK TumOR segmentation and outcome prediction challenge (HECKTOR 2022)

[18F]-Fluorodeoxyglucose (FDG) positron emission tomography - computed tomography (PET-CT) has become the imaging modality of choice for diagnosing many cancers. Co-learning complementary PET-CT imaging features is a fundamental requirement for automatic tumor segmentation and for developing computer aided cancer diagnosis systems. In this study, we propose a hyper-connected transformer (HCT) network that integrates a transformer network (TN) with a hyper connected fusion for multi-modality PET-CT images. The TN was leveraged for its ability to provide global dependencies in image feature learning, which was achieved by using image patch embeddings with a self-attention mechanism to capture image-wide contextual information. We extended the single-modality definition of TN with multiple TN based branches to separately extract image features. We also introduced a hyper connected fusion to fuse the contextual and complementary image features across multiple transformers in an iterative manner. Our results with two clinical datasets show that HCT achieved better performance in segmentation accuracy when compared to the existing methods., Comment: EMBC 2023

Beam combination with high efficiency is desirable to overcome the power limit of single electromagnetic sources, enabling long-distance optical communication and high-power laser. The efficiency of coherent beam combination is severely limited by the phase correlation between different input light beams. Here, we theoretically proposed and experimentally demonstrated a new mechanism for beam combining, the topology-enabled beam combination (TEBC), from multiple spatial channels with high efficiency based on a unidirectional topological edge state. We show that the topologically protected power orthogonal excitation arising from both the unidirectional edge states and the energy conservation ensures -0.31dB (93%) efficiency experimentally for a multi-channel combination of coherent microwaves at 9.1-9.3 GHz. Moreover, we demonstrate broadband, phase insensitive, and high-efficiency beam combination using the TEBC mechanism with one single topological photonic crystal device, which significantly reduces the device footprint and design complexity. Our scheme transcends the limits of the required phase correlations in the scenario of coherent beam combination and the number of combined channels in the scenario of incoherent beam combination., Comment: 15 pages, 4 figures

Multi-modality Fluorodeoxyglucose (FDG) positron emission tomography / computed tomography (PET/CT) has been routinely used in the assessment of common cancers, such as lung cancer, lymphoma, and melanoma. This is mainly attributed to the fact that PET/CT combines the high sensitivity for tumor detection of PET and anatomical information from CT. In PET/CT image assessment, automatic tumor segmentation is an important step, and in recent years, deep learning based methods have become the state-of-the-art. Unfortunately, existing methods tend to over-segment the tumor regions and include regions such as the normal high uptake organs, inflammation, and other infections. In this study, we introduce a false positive reduction network to overcome this limitation. We firstly introduced a self-supervised pre-trained global segmentation module to coarsely delineate the candidate tumor regions using a self-supervised pre-trained encoder. The candidate tumor regions were then refined by removing false positives via a local refinement module. Our experiments with the MICCAI 2022 Automated Lesion Segmentation in Whole-Body FDG-PET/CT (AutoPET) challenge dataset showed that our method achieved a dice score of 0.9324 with the preliminary testing data and was ranked 1st place in dice on the leaderboard. Our method was also ranked in the top 7 methods on the final testing data, the final ranking will be announced during the 2022 MICCAI AutoPET workshop. Our code is available at: https://github.com/YigePeng/AutoPET_False_Positive_Reduction., Comment: Pre-print paper for 2022 MICCAI AutoPET Challenge

Non-collinear antiferromagnet Weyl semimetal $Mn_3Sn$ have attracted great research interest recently. Although large anomalous Hall effect, anomalous Nernst effect and magneto-optical effect have been observed in $Mn_3Sn$, most studies are based on single crystals. So far, it is still challenging to grow high quality epitaxial $Mn_3Sn$ thin films with transport and optical properties comparable to their single crystal counterparts. Here, we report the structure, magneto-optical and transport properties of epitaxial $Mn_3Sn$ thin films fabricated by pulsed laser deposition (PLD). Highly oriented $Mn_{3+x}Sn_{1-x}$ (0001) and (11$\bar2$0) epitaxial films are successfully growth on single crystalline $Al_2O_3$ and MgO substrates. Large anomalous Hall effect (AHE) up to $\left| \Delta R_H\right|$=3.02 $\mu\Omega\cdot cm$, and longitudinal magneto-optical Kerr effect (LMOKE) with $\theta_K$ = 38.1 mdeg at 633 nm wavelength are measured at 300 K temperature, which are comparable to $Mn_3Sn$ single crystals. Our work demonstrates that high quality $Mn_3Sn$ epitaxial thin films can be fabricated by PLD, paving the way for future device applications.

Deformable image registration is a crucial step in medical image analysis for finding a non-linear spatial transformation between a pair of fixed and moving images. Deep registration methods based on Convolutional Neural Networks (CNNs) have been widely used as they can perform image registration in a fast and end-to-end manner. However, these methods usually have limited performance for image pairs with large deformations. Recently, iterative deep registration methods have been used to alleviate this limitation, where the transformations are iteratively learned in a coarse-to-fine manner. However, iterative methods inevitably prolong the registration runtime, and tend to learn separate image features for each iteration, which hinders the features from being leveraged to facilitate the registration at later iterations. In this study, we propose a Non-Iterative Coarse-to-finE registration Network (NICE-Net) for deformable image registration. In the NICE-Net, we propose: (i) a Single-pass Deep Cumulative Learning (SDCL) decoder that can cumulatively learn coarse-to-fine transformations within a single pass (iteration) of the network, and (ii) a Selectively-propagated Feature Learning (SFL) encoder that can learn common image features for the whole coarse-to-fine registration process and selectively propagate the features as needed. Extensive experiments on six public datasets of 3D brain Magnetic Resonance Imaging (MRI) show that our proposed NICE-Net can outperform state-of-the-art iterative deep registration methods while only requiring similar runtime to non-iterative methods., Comment: Accepted at International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022)

Zero-index metamaterials (ZIMs) feature a uniform electromagnetic mode over a large area in arbitrary shapes, enabling many applications including high-transmission supercouplers with arbitrary shapes, direction-independent phase matching for nonlinear optics, and collective emission of many quantum emitters. However, most ZIMs reported till date are passive, with no method for the dynamic modulation of their electromagnetic properties. Here, we design and fabricate a magnetically tunable ZIM consisting of yttrium iron garnet (YIG) pillars sandwiched between two copper clad laminates in the microwave regime. By harnessing the Cotton-Mouton effect of YIG, the metamaterial was successfully toggled between gapless and bandgap states, leading to a "phase transition" between a zero-index phase and a single negative phase of the metamaterial. Using an S-shaped ZIM supercoupler, we experimentally demonstrated a tunable supercoupling state with a low intrinsic loss of 0.95 dB and a high extinction ratio of up to 30.63 dB at 9 GHz. Our work enables dynamic modulation of the electromagnetic characteristics of ZIMs, enabling various applications in tunable linear, nonlinear, quantum and nonreciprocal electromagnetic devices.

Unconstrained by Lorentz reciprocity, nonreciprocal metasurfaces are uniquely capable of encoding distinctive optical functions on forward- and backward-propagating waves. The nonreciprocal metasurfaces reported to date require external electric or magnetic field biasing or rely on nonlinear effects, both of which are challenging to practically implement. Here, we propose and experimentally realize a magnet-free, linear, and passive nonreciprocal metasurface based on self-biased magnetic meta-atoms. Record transmittance up to 77% and operation angle reaching 64 degree are experimentally demonstrated. Moreover, on-demand bidirectional phase modulation in a "LEGO-like" manner is theoretically proposed and experimentally demonstrated, enabling a cohort of nonreciprocal functionalities such as microwave isolation, nonreciprocal beam steering, nonreciprocal focusing, and nonreciprocal holography. The design can also be extended to MHz and optical frequencies, taking advantage of the wide variety of self-biased gyrotropic materials available. We foresee that the nonreciprocal metasurfaces demonstrated in this work will have a significant practical impact for applications ranging from nonreciprocal antennas and radomes to full-duplex wireless communication and radar systems., Comment: 18 pages, 5 figures

Spectral/angular emissivity $e$ and absorptivity ${\alpha}$ of an object are widely believed to be identical by Kirchhoff's law of thermal radiation in reciprocal systems, but this introduces an intrinsic and inevitable energy loss for energy conversion and harvesting devices. So far, experimental evidences of breaking this well-known balance are still absent, and previous theoretical proposals are restricted to narrow single-band nonreciprocal radiation. Here we observe for the first time, to our knowledge, the violation of Kirchhoff's law using ultrathin ($<{\lambda}/40$, ${\lambda}$ is the working wavelength) magnetized InAs semiconductor films at epsilon-near-zero (ENZ) frequencies. Large difference of $|{\alpha}-e|>0.6$ has been experimentally demonstrated under a moderate external magnetic field. Moreover, based on magnetized ENZ building blocks supporting asymmetrically radiative Berreman and surface ENZ modes, we show versatile shaping of nonreciprocal thermal radiation: single-band, dual-band, and broadband nonreciprocal emission spectra at different wavebands. Our findings of breaking Kirchhoff's law will advance the conventional understanding of emission and absorption processes of natural objects, and lay a solid foundation for more comprehensive studies in designing various nonreciprocal thermal emitters. The reported recipe of diversely shaping nonreciprocal emission will also breed new possibilities in renovating next-generation nonreciprocal energy devices in the areas of solar cells, thermophotovoltaic, radiative cooling, etc., Comment: 21 pages, 4 figures

Registration of longitudinal brain MRI scans containing pathologies is challenging due to dramatic changes in tissue appearance. Although there has been progress in developing general-purpose medical image registration techniques, they have not yet attained the requisite precision and reliability for this task, highlighting its inherent complexity. Here we describe the Brain Tumor Sequence Registration (BraTS-Reg) challenge, as the first public benchmark environment for deformable registration algorithms focusing on estimating correspondences between pre-operative and follow-up scans of the same patient diagnosed with a diffuse brain glioma. The BraTS-Reg data comprise de-identified multi-institutional multi-parametric MRI (mpMRI) scans, curated for size and resolution according to a canonical anatomical template, and divided into training, validation, and testing sets. Clinical experts annotated ground truth (GT) landmark points of anatomical locations distinct across the temporal domain. Quantitative evaluation and ranking were based on the Median Euclidean Error (MEE), Robustness, and the determinant of the Jacobian of the displacement field. The top-ranked methodologies yielded similar performance across all evaluation metrics and shared several methodological commonalities, including pre-alignment, deep neural networks, inverse consistency analysis, and test-time instance optimization per-case basis as a post-processing step. The top-ranked method attained the MEE at or below that of the inter-rater variability for approximately 60% of the evaluated landmarks, underscoring the scope for further accuracy and robustness improvements, especially relative to human experts. The aim of BraTS-Reg is to continue to serve as an active resource for research, with the data and online evaluation tools accessible at https://bratsreg.github.io/.

Metamaterials with artificial optical properties have attracted significant research interest. In particular, artificial magnetic resonances in non-unity permeability tensor at optical frequencies in metamaterials have been reported. However, only non-unity diagonal elements of the permeability tensor have been demonstrated to date. A gyromagnetic permeability tensor with non-zero off-diagonal elements has not been observed at the optical frequencies. Here we report the observation of gyromagnetic properties in the near-infrared wavelength range in a magneto-plasmonic metamaterial. The non-zero off-diagonal permeability tensor element causes the transverse magneto-optical Kerr effect (TMOKE) under s-polarized incidence that otherwise vanishes if the permeability tensor is not gyromagnetic. By retrieving the permeability tensor elements from reflection, transmission, and TMOKE spectra, we show that the effective off-diagonal permeability tensor elements reach the 10-3 level at the resonance wavelength (~900 nm) of the split-ring resonators that is at least two orders of magnitude higher than that of magneto-optical materials at the same wavelength. The artificial gyromagnetic permeability is attributed to the change in the local electric field direction modulated by the split-ring resonators. Our study demonstrates the possibility of engineering the permeability and permittivity tensors in metamaterials at arbitrary frequencies, thereby promising a variety of applications of next-generation nonreciprocal photonic devices, magneto-plasmonic sensors, and active metamaterials.

Motion estimation is a fundamental step in dynamic medical image processing for the assessment of target organ anatomy and function. However, existing image-based motion estimation methods, which optimize the motion field by evaluating the local image similarity, are prone to produce implausible estimation, especially in the presence of large motion. In this study, we provide a novel motion estimation framework of Dense-Sparse-Dense (DSD), which comprises two stages. In the first stage, we process the raw dense image to extract sparse landmarks to represent the target organ anatomical topology and discard the redundant information that is unnecessary for motion estimation. For this purpose, we introduce an unsupervised 3D landmark detection network to extract spatially sparse but representative landmarks for the target organ motion estimation. In the second stage, we derive the sparse motion displacement from the extracted sparse landmarks of two images of different time points. Then, we present a motion reconstruction network to construct the motion field by projecting the sparse landmarks displacement back into the dense image domain. Furthermore, we employ the estimated motion field from our two-stage DSD framework as initialization and boost the motion estimation quality in light-weight yet effective iterative optimization. We evaluate our method on two dynamic medical imaging tasks to model cardiac motion and lung respiratory motion, respectively. Our method has produced superior motion estimation accuracy compared to existing comparative methods. Besides, the extensive experimental results demonstrate that our solution can extract well representative anatomical landmarks without any requirement of manual annotation. Our code is publicly available online., Comment: accepted by IEEE Transactions on Cybernetics

Nasopharyngeal Carcinoma (NPC) is a malignant epithelial cancer arising from the nasopharynx. Survival prediction is a major concern for NPC patients, as it provides early prognostic information to plan treatments. Recently, deep survival models based on deep learning have demonstrated the potential to outperform traditional radiomics-based survival prediction models. Deep survival models usually use image patches covering the whole target regions (e.g., nasopharynx for NPC) or containing only segmented tumor regions as the input. However, the models using the whole target regions will also include non-relevant background information, while the models using segmented tumor regions will disregard potentially prognostic information existing out of primary tumors (e.g., local lymph node metastasis and adjacent tissue invasion). In this study, we propose a 3D end-to-end Deep Multi-Task Survival model (DeepMTS) for joint survival prediction and tumor segmentation in advanced NPC from pretreatment PET/CT. Our novelty is the introduction of a hard-sharing segmentation backbone to guide the extraction of local features related to the primary tumors, which reduces the interference from non-relevant background information. In addition, we also introduce a cascaded survival network to capture the prognostic information existing out of primary tumors and further leverage the global tumor information (e.g., tumor size, shape, and locations) derived from the segmentation backbone. Our experiments with two clinical datasets demonstrate that our DeepMTS can consistently outperform traditional radiomics-based survival prediction models and existing deep survival models., Comment: Accepted at IEEE Journal of Biomedical and Health Informatics (JBHI)

Dielectric Mie nanoresonators showing strong light-matter interaction at the nanoscale may enable new functionality in photonic devices. Recently, strong magneto-optical effects have been observed in magneto-optical nanophotonic devices due to the electromagnetic field localization. However, most reports so far have been focused on the enhancement of conventional magneto-optical effects. Here, we report the observation of circular displacement current induced anomalous magneto-optical effects in high-index-contrast Si/Ce:YIG/YIG/SiO2 Mie resonators. In particular, giant modulation of light intensity in transverse magnetic configuration up to 6.4 % under s-polarized incidence appears, which is non-existent in planar magneto-optical thin films. Apart from that, we observe a large rotation of transmitted light polarization in the longitudinal magnetic configuration under near normal incidence conditions, which is two orders of magnitude higher than for a planar magneto-optical thin film. These phenomena are essentially originated from the unique circular displacement current when exciting the magnetic resonance modes in the Mie resonators, which changes the incident electric field direction locally. Our work indicates an uncharted territory of light polarization control based on the complex modal profiles in all-dielectric magneto-optical Mie resonators and metasurfaces, which opens the door for versatile control of light propagation by magnetization for a variety of applications in vectoral magnetic field and biosensing, free space non-reciprocal photonic devices, magneto-optical imaging and optomagnetic memories.

Non-reciprocal photonic devices are essential components of classical optical information processing. It is interesting and important to investigate their feasibility in the quantum world. In this work, the quantum properties of an on-chip silicon nitride (SiN)-based magneto-optical (MO) isolator were studied using a single-photon non-reciprocal dynamical transmission experiment. The measured isolation ratio for single photons achieved was 12.33 dB, which proved the functionality of our on-chip isolator. The quantum coherence of the passing single photons was further verified using high-visibility quantum interference. Our work will promote on-chip isolators within the integrated quantum circuits and help introduce novel phenomena in quantum information processes., Comment: 5 pages, 4figures

Sequential whole-body 18F-Fluorodeoxyglucose (FDG) positron emission tomography (PET) scans are regarded as the imaging modality of choice for the assessment of treatment response in the lymphomas because they detect treatment response when there may not be changes on anatomical imaging. Any computerized analysis of lymphomas in whole-body PET requires automatic segmentation of the studies so that sites of disease can be quantitatively monitored over time. State-of-the-art PET image segmentation methods are based on convolutional neural networks (CNNs) given their ability to leverage annotated datasets to derive high-level features about the disease process. Such methods, however, focus on PET images from a single time-point and discard information from other scans or are targeted towards specific organs and cannot cater for the multiple structures in whole-body PET images. In this study, we propose a spatio-temporal 'dual-stream' neural network (ST-DSNN) to segment sequential whole-body PET scans. Our ST-DSNN learns and accumulates image features from the PET images done over time. The accumulated image features are used to enhance the organs / structures that are consistent over time to allow easier identification of sites of active lymphoma. Our results show that our method outperforms the state-of-the-art PET image segmentation methods., Comment: 16 pages