Search

In the rapidly evolving landscape of artificial intelligence, multimodal learning systems (MMLS) have gained traction for their ability to process and integrate information from diverse modality inputs. Their expanding use in vital sectors such as healthcare has made safety assurance a critical concern. However, the absence of systematic research into their safety is a significant barrier to progress in this field. To bridge the gap, we present the first taxonomy that systematically categorizes and assesses MMLS safety. This taxonomy is structured around four fundamental pillars that are critical to ensuring the safety of MMLS: robustness, alignment, monitoring, and controllability. Leveraging this taxonomy, we review existing methodologies, benchmarks, and the current state of research, while also pinpointing the principal limitations and gaps in knowledge. Finally, we discuss unique challenges in MMLS safety. In illuminating these challenges, we aim to pave the way for future research, proposing potential directions that could lead to significant advancements in the safety protocols of MMLS.

Object detection in visible (RGB) and infrared (IR) images has been widely applied in recent years. Leveraging the complementary characteristics of RGB and IR images, the object detector provides reliable and robust object localization from day to night. Most existing fusion strategies directly input RGB and IR images into deep neural networks, leading to inferior detection performance. However, the RGB and IR features have modality-specific noise, these strategies will exacerbate the fused features along with the propagation. Inspired by the mechanism of the human brain processing multimodal information, in this paper, we introduce a new coarse-to-fine perspective to purify and fuse two modality features. Specifically, following this perspective, we design a Redundant Spectrum Removal module to coarsely remove interfering information within each modality and a Dynamic Feature Selection module to finely select the desired features for feature fusion. To verify the effectiveness of the coarse-to-fine fusion strategy, we construct a new object detector called the Removal and Selection Detector (RSDet). Extensive experiments on three RGB-IR object detection datasets verify the superior performance of our method., Comment: 11pages, 11figures

Semantic analysis on visible (RGB) and infrared (IR) images has gained attention for its ability to be more accurate and robust under low-illumination and complex weather conditions. Due to the lack of pre-trained foundation models on the large-scale infrared image datasets, existing methods prefer to design task-specific frameworks and directly fine-tune them with pre-trained foundation models on their RGB-IR semantic relevance datasets, which results in poor scalability and limited generalization. In this work, we propose a scalable and efficient framework called UniRGB-IR to unify RGB-IR downstream tasks, in which a novel adapter is developed to efficiently introduce richer RGB-IR features into the pre-trained RGB-based foundation model. Specifically, our framework consists of a vision transformer (ViT) foundation model, a Multi-modal Feature Pool (MFP) module and a Supplementary Feature Injector (SFI) module. The MFP and SFI modules cooperate with each other as an adpater to effectively complement the ViT features with the contextual multi-scale features. During training process, we freeze the entire foundation model to inherit prior knowledge and only optimize the MFP and SFI modules. Furthermore, to verify the effectiveness of our framework, we utilize the ViT-Base as the pre-trained foundation model to perform extensive experiments. Experimental results on various RGB-IR downstream tasks demonstrate that our method can achieve state-of-the-art performance. The source code and results are available at https://github.com/PoTsui99/UniRGB-IR.git.

Graph Neural Networks (GNNs) are powerful tools for learning representations on graphs, such as social networks. However, their vulnerability to privacy inference attacks restricts their practicality, especially in high-stake domains. To address this issue, privacy-preserving GNNs have been proposed, focusing on preserving node and/or link privacy. This work takes a step back and investigates how GNNs contribute to privacy leakage. Through theoretical analysis and simulations, we identify message passing under structural bias as the core component that allows GNNs to \textit{propagate} and \textit{amplify} privacy leakage. Building upon these findings, we propose a principled privacy-preserving GNN framework that effectively safeguards both node and link privacy, referred to as dual-privacy preservation. The framework comprises three major modules: a Sensitive Information Obfuscation Module that removes sensitive information from node embeddings, a Dynamic Structure Debiasing Module that dynamically corrects the structural bias, and an Adversarial Learning Module that optimizes the privacy-utility trade-off. Experimental results on four benchmark datasets validate the effectiveness of the proposed model in protecting both node and link privacy while preserving high utility for downstream tasks, such as node classification., Comment: CIKM 2023

Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to inject spatial details from panchromatic images into multispectral images (MS) to obtain high-resolution multispectral images. Since deep learning has received widespread attention because of its powerful fitting ability and efficient feature extraction, a variety of pan-sharpening methods have been proposed to achieve remarkable performance. However, current pan-sharpening methods usually require the paired panchromatic (PAN) and MS images as input, which limits their usage in some scenarios. To address this issue, in this paper we observe that the spatial details from PAN images are mainly high-frequency cues, i.e., the edges reflect the contour of input PAN images. This motivates us to develop a PAN-agnostic representation to store some base edges, so as to compose the contour for the corresponding PAN image via them. As a result, we can perform the pan-sharpening task with only the MS image when inference. To this end, a memory-based network is adapted to extract and memorize the spatial details during the training phase and is used to replace the process of obtaining spatial information from PAN images when inference, which is called Memory-based Spatial Details Network (MSDN). Finally, we integrate the proposed MSDN module into the existing deep learning-based pan-sharpening methods to achieve an end-to-end pan-sharpening network. With extensive experiments on the Gaofen1 and WorldView-4 satellites, we verify that our method constructs good spatial details without PAN images and achieves the best performance. The code is available at https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git.

Background: Multiple sclerosis (MS) is a central nervous system disease with a high disability rate. Modern molecular biology techniques have identified a number of key genes and diagnostic markers to MS, but the etiology and pathogenesis of MS remain unknown. Results: In this study, the integration of three peripheral blood mononuclear cell (PBMC) microarray datasets and one peripheral blood T cells microarray dataset allowed comprehensive network and pathway analyses of the biological functions of MS-related genes. Differential expression analysis identified 78 significantly aberrantly expressed genes in MS, and further functional enrichment analysis showed that these genes were associated with innate immune response-activating signal transduction (p = 0.0017), neutrophil mediated immunity (p = 0.002), positive regulation of innate immune response (p = 0.004), IL-17 signaling pathway (p < 0.035) and other immune-related signaling pathways. In addition, a network of MS-specific protein–protein interactions (PPI) was constructed based on differential genes. Subsequent analysis of network topology properties identified the up-regulated CXCR4, ITGAM, ACTB, RHOA, RPS27A, UBA52, and RPL8 genes as the hub genes of the network, and they were also potential biomarkers of MS through Rap1 signaling pathway or leukocyte transendothelial migration. RT-qPCR results demonstrated that CXCR4 was obviously up-regulated, while ACTB, RHOA, and ITGAM were down-regulated in MS patient PBMC in comparison with normal samples. Finally, support vector machine was employed to establish a diagnostic model of MS with a high prediction performance in internal and external datasets (mean AUC = 0.97) and in different chip platform datasets (AUC = (0.93). Conclusion: This study provides new understanding for the etiology/pathogenesis of MS, facilitating an early identification and prediction of MS.

The pancreatic disease taxonomy includes ten types of masses (tumors or cysts)[20,8]. Previous work focuses on developing segmentation or classification methods only for certain mass types. Differential diagnosis of all mass types is clinically highly desirable [20] but has not been investigated using an automated image understanding approach. We exploit the feasibility to distinguish pancreatic ductal adenocarcinoma (PDAC) from the nine other nonPDAC masses using multi-phase CT imaging. Both image appearance and the 3D organ-mass geometry relationship are critical. We propose a holistic segmentation-mesh-classification network (SMCN) to provide patient-level diagnosis, by fully utilizing the geometry and location information, which is accomplished by combining the anatomical structure and the semantic detection-by-segmentation network. SMCN learns the pancreas and mass segmentation task and builds an anatomical correspondence-aware organ mesh model by progressively deforming a pancreas prototype on the raw segmentation mask (i.e., mask-to-mesh). A new graph-based residual convolutional network (Graph-ResNet), whose nodes fuse the information of the mesh model and feature vectors extracted from the segmentation network, is developed to produce the patient-level differential classification results. Extensive experiments on 661 patients' CT scans (five phases per patient) show that SMCN can improve the mass segmentation and detection accuracy compared to the strong baseline method nnUNet (e.g., for nonPDAC, Dice: 0.611 vs. 0.478; detection rate: 89% vs. 70%), achieve similar sensitivity and specificity in differentiating PDAC and nonPDAC as expert radiologists (i.e., 94% and 90%), and obtain results comparable to a multimodality test [20] that combines clinical, imaging, and molecular testing for clinical management of patients.

SMARCAD1 is a matrix-associated actin-dependent regulator of chromatin that encodes SWI/SNF subfamily of helicase proteins. It has been shown that the functions of SMARCAD1 are linked to histone 3 citrullination, a specific type of histone post-translational modifications. Histone citrullination can lead to alterations in protein functions, and affect gene expressions. This peptidylarginine deiminases (PADIs) catalyzed post-translational modification also increases in the progression of cancer. Interestingly, human renal cancer cells exhibit low to non-detection of SMARCAD1 protein expression. It is anticipated that an association between SMARCAD1 and histone 3 citrullination function critically in renal cancer and tumor formation. Western blot was performed on mouse cells fist followed by multiple renal cancer and non-cancer human cell lines, to determine SMARCAD1 and histone 3 citrullination protein expression levels and their potential correlation. Multiple cell treatments were introduced and CRISPR/Cas9 gene editing system was used to create SMARCAD1 gene knocking out condition. A negative correlation was determined between SMARCAD1 and citrullination expression level, with a potential positive feedback loop in between SMARCAD1 and histone 3 citrulline enzyme PADI4 in certain cell lines. Moreover, this negative correlation between two protein levels serves as an indicator of renal cancer existence; decrement of SMARCAD1 protein level holds true in renal cancer cells, in parallel with an increment of the histone 3 citrulline protein expression.

SMARCAD1 is a matrix-associated actin-dependent regulator of chromatin that encodes SWI/SNF subfamily of helicase proteins. It has been shown that the functions of SMARCAD1 are linked to histone 3 citrullination, a specific type of histone post-translational modifications. Histone citrullination can lead to alterations in protein functions, and affect gene expressions. This peptidylarginine deiminases (PADIs) catalyzed post-translational modification also increases in the progression of cancer. Interestingly, human renal cancer cells exhibit low to non-detection of SMARCAD1 protein expression. It is anticipated that an association between SMARCAD1 and histone 3 citrullination function critically in renal cancer and tumor formation. Western blot was performed on mouse cells fist followed by multiple renal cancer and non-cancer human cell lines, to determine SMARCAD1 and histone 3 citrullination protein expression levels and their potential correlation. Multiple cell treatments were introduced and CRISPR/Cas9 gene editing system was used to create SMARCAD1 gene knocking out condition. A negative correlation was determined between SMARCAD1 and citrullination expression level, with a potential positive feedback loop in between SMARCAD1 and histone 3 citrulline enzyme PADI4 in certain cell lines. Moreover, this negative correlation between two protein levels serves as an indicator of renal cancer existence; decrement of SMARCAD1 protein level holds true in renal cancer cells, in parallel with an increment of the histone 3 citrulline protein expression.

With the seawater temperature rising more than 1.5°C (IPCC) from the pre-industrial time, marine organisms are facing more and more severe climate changes. As temperature is an important factor influencing the physiology of animals, species specific adaptations has been well observed. Subtidal species are one of the most seawater temperature influenced animals. In previous researches, NMR metabolic profiling has been proved to be a decent technique of animal physiological studies. In this work, the king scallop, Pecten maximus was studied to test if (1) consuming labeled phytoplankton would be a stable way of 13C labeling marine filter feeders such as scallops; (2) the metabolism of P. maximus would also change with increasing temperature, which reflects as the different filtration rates from the outside and changing metabolic pathway inside organs. The scallop P. maximus were incubated under two different temperatures, 15°C and 20°C, fed with 13C labeled diatom Phaeodactylum tricornutum. After three days’ filtration rate measurement, the tissue samples of digestive gland and striated adductor muscle were dissected and extracted. Both qualitatively and quantitatively metabolic profiling was done via 13C NMR analyzation. The performance of experiment animal, Pecten maximus were quite different under two temperature treatments. Higher filtration rate was observed at 20°C whereas faster digestion and incorporation of algal lipids was also found inside the digestive gland from 20°C treatment. As for the muscle tissues, incorporation of 13C labeling was observed in both temperature groups, proving this labeling technique is applicable for marine filter feeders.

With the seawater temperature rising more than 1.5°C (IPCC) from the pre-industrial time, marine organisms are facing more and more severe climate changes. As temperature is an important factor influencing the physiology of animals, species specific adaptations has been well observed. Subtidal species are one of the most seawater temperature influenced animals. In previous researches, NMR metabolic profiling has been proved to be a decent technique of animal physiological studies. In this work, the king scallop, Pecten maximus was studied to test if (1) consuming labeled phytoplankton would be a stable way of 13C labeling marine filter feeders such as scallops; (2) the metabolism of P. maximus would also change with increasing temperature, which reflects as the different filtration rates from the outside and changing metabolic pathway inside organs. The scallop P. maximus were incubated under two different temperatures, 15°C and 20°C, fed with 13C labeled diatom Phaeodactylum tricornutum. After three days’ filtration rate measurement, the tissue samples of digestive gland and striated adductor muscle were dissected and extracted. Both qualitatively and quantitatively metabolic profiling was done via 13C NMR analyzation. The performance of experiment animal, Pecten maximus were quite different under two temperature treatments. Higher filtration rate was observed at 20°C whereas faster digestion and incorporation of algal lipids was also found inside the digestive gland from 20°C treatment. As for the muscle tissues, incorporation of 13C labeling was observed in both temperature groups, proving this labeling technique is applicable for marine filter feeders.

Histone citrullination regulates diverse cellular processes. Here, we report that SMARCAD1 preferentially associates with H3 arginine 26 citrullination (H3R26Cit) peptides present on arrays composed of 384 histone peptides harboring distinct post-transcriptional modifications. Among ten histone modifications assayed by ChIP-seq, H3R26Cit exhibited the most extensive genomewide co-localization with SMARCAD1 binding. Increased Smarcad1 expression correlated with naive pluripotency in pre-implantation embryos. In the presence of LIF, Smarcad1 knockdown (KD) embryonic stem cells lost naive state phenotypes but remained pluripotent, as suggested by morphology, gene expression, histone modifications, alkaline phosphatase activity, energy metabolism, embryoid bodies, teratoma, and chimeras. The majority of H3R26Cit ChIP-seq peaks occupied by SMARCAD1 were associated with increased levels of H3K9me3 in Smarcad1 KD cells. Inhibition of H3Cit induced H3K9me3 at the overlapping regions of H3R26Cit peaks and SMARCAD1 peaks. These data suggest a model in which SMARCAD1 regulates naive pluripotency by interacting with H3R26Cit and suppressing heterochromatin formation.

In this paper, a level-wise mixture model (LMM) is developed by embedding visual hierarchy with deep networks to support large-scale visual recognition (i.e., recognizing thousands or even tens of thousands of object classes), and a Bayesian approach is used to adapt a pre-trained visual hierarchy automatically to the improvements of deep features (that are used for image and object class representation) when more representative deep networks are learned along the time. Our LMM model can provide an end-to-end approach for jointly learning: (a) the deep networks to extract more discriminative deep features for image and object class representation; (b) the tree classifier for recognizing large numbers of object classes hierarchically; and (c) the visual hierarchy adaptation for achieving more accurate indexing of large numbers of object classes hierarchically. By supporting joint learning of the tree classifier, the deep networks and the visual hierarchy adaptation, our LMM algorithm can provide an effective approach for controlling inter-level error propagation effectively, thus it can achieve better accuracy rates on large-scale visual recognition. Our experiments are carried on ImageNet1K and ImageNet10K image sets, and our LMM algorithm can achieve very competitive results on both the accuracy rates and the computation efficiency as compared with the baseline methods.

In this paper, a deep mixture of diverse experts algorithm is developed for seamlessly combining a set of base deep CNNs (convolutional neural networks) with diverse outputs (task spaces), e.g., such base deep CNNs are trained to recognize different subsets of tens of thousands of atomic object classes. First, a two-layer (category layer and object class layer) ontology is constructed to achieve more effective solution for task group generation, e.g., assigning the semantically-related atomic object classes at the sibling leaf nodes into the same task group because they may share similar learning complexities. Second, one particular base deep CNNs with $M+1$ ($M \leq 1,000$) outputs is learned for each task group to recognize its $M$ atomic object classes effectively and identify one special class of "not-in-group" automatically, and the network structure (numbers of layers and units in each layer) of the well-designed AlexNet is directly used to configure such base deep CNNs. A deep multi-task learning algorithm is developed to leverage the inter-class visual similarities to learn more discriminative base deep CNNs and multi-task softmax for enhancing the separability of the atomic object classes in the same task group. Finally, all these base deep CNNs with diverse outputs (task spaces) are seamlessly combined to form a deep mixture of diverse experts for recognizing tens of thousands of atomic object classes. Our experimental results have demonstrated that our deep mixture of diverse experts algorithm can achieve very competitive results on large-scale visual recognition.

Robots such as drones have been leveraged to perform structure health inspection such as bridge inspection. Big data of inspection videos can be collected by cameras mounted on drones. In this project, we develop image analysis algorithms to support bridge engineers to analyze the big video data. Bridge engineers define the region of interest initially, then the algorithm retrieves all related regions in the video, which facilitates the engineers to inspect the bridge rather than exhaustively check every frame of the video. To perform this task, we propose a Multi-scale Siamese Neural Network. The network is initially trained by one-shot learning and is fine-tuned iteratively with human in the loop. Our neural network is evaluated on three bridge inspection videos with promising performances.