Search

• A novel Cluster-Prototype Matching (CPM) strategy is proposed to solve the domain shift problem of zero-shot learning. • The distribution information of samples in the embedding space, i.e., cluster structure, is utilized to assist in zero-shot learning classification. • The CPM strategy can be applied to most existing ZSL models in a plug-and-play style. • Extensive experiments show the effectiveness and robustness of the proposed method. Given the descriptions of classes, Zero-Shot Learning (ZSL) aims to recognize unseen samples by learning a projection between the visual features of samples and the semantic descriptions (prototypes) of classes from seen data. However, due to the inherent distribution gap between seen and unseen domains, the learned projection is generally biased to seen classes and may produce misleading relationships between unseen samples and prototypes (sample-prototype relationship). To tackle this problem, we propose a Cluster-Prototype Matching (CPM) framework which exploits the distribution information of samples to explore the cluster structure of samples and then use the robust cluster-prototype relationship to correct the biased sample-prototype relationship. Specifically, we first use an iterative cluster generation module to identify the underlying cluster structure of samples based on their embedding features, which are acquired via a basic ZSL model. Then each identified cluster will be matched with a specific class prototype through the Kuhn-Munkres algorithm, based on which we can export a sharp cluster-prototype similarity. Finally, the cluster-prototype similarity is combined with the sample-prototype similarity to determine the class labels of test samples. We apply CPM to five well-established ZSL methods and the experimental results show that CPM can significantly improve the performance of basic models and enable them achieve or beyond the state-of-the-art. [ABSTRACT FROM AUTHOR]

• Firstly, we consider a new yet practical DA scenario, called sparsely-labeled source assisted domain adaptation. • Secondly, we propose a unified framework to jointly seek cluster centroids, source and target labels, and domain-invariant features. • Thirdly, we construct an optimization strategy to solve the objective function efficiently. Domain Adaptation (DA) aims to generalize the classifier learned from a well-labeled source domain to an unlabeled target domain. Existing DA methods usually assume that rich labels could be available in the source domain. However, we usually confront the source domain with a large number of unlabeled data but only a few labeled data, and thus, how to transfer knowledge from this sparsely-labeled source domain to the target domain is still a challenge, which greatly limits its application in the wild. This paper proposes a novel Sparsely-Labeled Source Assisted Domain Adaptation (SLSA-DA) algorithm to address the challenge with limited labeled source domain samples. Specifically, due to the label scarcity problem, the projected clustering is first conducted on both the source and target domains, so that the discriminative structures of data could be exploited elegantly. Then label propagation is adopted to propagate the labels from those limited labeled source samples to the whole unlabeled data progressively, so that the cluster labels are revealed correctly. Finally, we jointly align the marginal and conditional distributions to mitigate the cross-domain mismatching problem, and optimize those three procedures iteratively. However, it is nontrivial to incorporate the above three procedures into a unified optimization framework seamlessly since some variables to be optimized are implicitly involved in their formulas, thus they could not benefit to each other. Remarkably, we prove that the projected clustering and conditional distribution alignment could be reformulated into other formulations, thus the implicit variables are embedded in different optimization steps. As such, the variables related to those three quantities could be optimized in a unified optimization framework and benefit to each other, and improve the recognition performance obviously. Extensive experiments have verified that our approach could deal with the challenge in the SLSA-DA setting, and achieve the best performances across different real-world cross-domain visual recognition tasks. Our preliminary Matlab code is available at https://github.com/WWLoveTransfer/SLSA-DA/. [ABSTRACT FROM AUTHOR]

Spatial transcriptomics (ST) revolutionizes RNA quantification with high spatial resolution. Hematoxylin and eosin (H&E) images, the gold standard in medical diagnosis, offer insights into tissue structure, correlating with gene expression patterns. Current methods for predicting spatial gene expression from H&E images often overlook spatial relationships. We introduce ResSAT (Residual networks - Self-Attention Transformer), a framework generating spatially resolved gene expression profiles from H&E images by capturing tissue structures and using a self-attention transformer to enhance prediction.Benchmarking on 10× Visium datasets, ResSAT significantly outperformed existing methods, promising reduced ST profiling costs and rapid acquisition of numerous profiles., Competing Interests: Competing interests The authors declare that they have no competing interests.

Integration of heterogeneous and high-dimensional multi-omics data is becoming increasingly important in understanding etiology of complex genetic diseases. Each omics technique only provides a limited view of the underlying biological process and integrating heterogeneous omics layers simultaneously would lead to a more comprehensive and detailed understanding of diseases and phenotypes. However, one obstacle faced when performing multi-omics data integration is the existence of unpaired multi-omics data due to instrument sensitivity and cost. Studies may fail if certain aspects of the subjects are missing or incomplete. In this paper, we propose a deep learning method for multi-omics integration with incomplete data by Cross-omics Linked unified embedding with Contrastive Learning and Self Attention (CLCLSA). Utilizing complete multi-omics data as supervision, the model employs cross-omics autoencoders to learn the feature representation across different types of biological data. The multi-omics contrastive learning is employed, which maximizes the mutual information between different types of omics. In addition, the feature-level self-attention and omics-level self-attention are employed to dynamically identify the most informative features for multi-omics data integration. Finally, a Softmax classifier is employed to perform multi-omics data classification. Extensive experiments were conducted on four public multi-omics datasets. The experimental results indicate that our proposed CLCLSA produces promising results in multi-omics data classification using both complete and incomplete multi-omics data., Competing Interests: Declaration of competing interest The authors declare that there are no conflicts of interest., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

Zero-shot learning (ZSL) aims to recognize novel categories by merely utilizing disjoint seen samples. It is a challenging task as the knowledge of unseen objects is forbidden in the training stage, which easily leads to unseen samples degrading to mismatched categories. In order to alleviate the biased recognition problem, in this article, we propose a differential refinement network (DRNet) for ZSL, which aims to explore robust semantic-to-visual embedding. Our DRNet model consists of two subnetworks: basic network and differential network. The basic network targets to generate initial class-specific visual centers conditioned on corresponding semantic prototypes. The differential network is designed to predict class-unrelated differences between visual centers of arbitrary semantic prototype pairs, which are applied to further polish the initial visual centers. The motivation is that, by comparing different prototypes, interactions between various categories will be characterized, benefiting the generation of authentic and discriminative visual centers. Moreover, a modified episode-based training paradigm is explored to optimize the two subnetworks actively. In the training stage, we form a collection of episodes, each of which is an imitated ZSL task. Our DRNet is optimized by those sampled tasks rather than individual samples, which progressively learns skills to adapt and generalize to novel classes. Experiments on four challenging datasets demonstrate the effectiveness of our method.

Many proteoforms can be produced from a gene due to genetic mutations, alternative splicing, post-translational modifications (PTMs), and other variations. PTMs in proteoforms play critical roles in cell signaling, protein degradation, and other biological processes. Mass spectrometry (MS) is the primary technique for investigating PTMs in proteoforms, and two alternative MS approaches, top-down and bottom-up, have complementary strengths. The combination of the two approaches has the potential to increase the sensitivity and accuracy in PTM identification and characterization. In addition, protein and PTM knowledge bases, such as UniProt, provide valuable information for PTM characterization and verification. Here, we present a software pipeline PTM-TBA (PTM characterization by Top-down and Bottom-up MS and Annotations) for identifying and localizing PTMs in proteoforms by integrating top-down and bottom-up MS as well as PTM annotations. We assessed PTM-TBA using a technical triplicate of bottom-up and top-down MS data of SW480 cells. On average, database search of the top-down MS data identified 2000 mass shifts, 814.5 (40.7%) of which were matched to 11 common PTMs and 423 of which were localized. Of the mass shifts identified by top-down MS, PTM-TBA verified 435 mass shifts using the bottom-up MS data and UniProt annotations.

Background: Deep learning, which is a part of a broader concept of artificial intelligence (AI) and/or machine learning has achieved remarkable success in vision tasks. While there is growing interest in the use of this technology in diagnostic support for skin-related neglected tropical diseases (skin NTDs), there have been limited studies in this area and fewer focused on dark skin. In this study, we aimed to develop deep learning based AI models with clinical images we collected for five skin NTDs, namely, Buruli ulcer, leprosy, mycetoma, scabies, and yaws, to understand how diagnostic accuracy can or cannot be improved using different models and training patterns., Methodology: This study used photographs collected prospectively in Côte d'Ivoire and Ghana through our ongoing studies with use of digital health tools for clinical data documentation and for teledermatology. Our dataset included a total of 1,709 images from 506 patients. Two convolutional neural networks, ResNet-50 and VGG-16 models were adopted to examine the performance of different deep learning architectures and validate their feasibility in diagnosis of the targeted skin NTDs., Principal Findings: The two models were able to correctly predict over 70% of the diagnoses, and there was a consistent performance improvement with more training samples. The ResNet-50 model performed better than the VGG-16 model. A model trained with PCR confirmed cases of Buruli ulcer yielded 1-3% increase in prediction accuracy across all diseases, except, for mycetoma, over a model which training sets included unconfirmed cases., Conclusions: Our approach was to have the deep learning model distinguish between multiple pathologies simultaneously-which is close to real-world practice. The more images used for training, the more accurate the diagnosis became. The percentages of correct diagnosis increased with PCR-positive cases of Buruli ulcer. This demonstrated that it may be better to input images from the more accurately diagnosed cases in the training models also for achieving better accuracy in the generated AI models. However, the increase was marginal which may be an indication that the accuracy of clinical diagnosis alone is reliable to an extent for Buruli ulcer. Diagnostic tests also have their flaws, and they are not always reliable. One hope for AI is that it will objectively resolve this gap between diagnostic tests and clinical diagnoses with the addition of another tool. While there are still challenges to be overcome, there is a potential for AI to address the unmet needs where access to medical care is limited, like for those affected by skin NTDs., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2023 Yotsu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Domain adaptation (DA) has recently drawn a lot of attention, as it facilitates unlabeled target learning by borrowing knowledge from an external source domain. Most existing DA solutions seek to align feature representations between the labeled source and unlabeled target data. However, the scarcity of target data easily results in negative transfer, as it misleads the cross DA to the dominance of the source. To address the challenging few-shot domain adaptation (FSDA) problem, in this article, we propose a novel marginalized augmented FSDA (MAF) approach to address the cross-domain distribution disparity and insufficiency of target data simultaneously. On the one hand, cross-domain continuity augmentation (CCA) synthesizes abundant intermediate patterns across domains leading to a continuous domain-invariant latent space. On the other hand, sufficient source-supervised semantic augmentation (SSA) is explored to progressively diversify the conditional distribution within and across domains. Moreover, the proposed augmentation strategies are implemented efficiently via an expected transferable cross-entropy (CE) loss over the augmented distribution instead of explicit data synthesis, and minimizing the upper bound of the expected loss introduces negligible extra computing cost. Experimentally, our method outperforms the state of the art in various FSDA benchmarks, which demonstrates the effectiveness and contribution of our work. Our source code is provided at https://github.com/scottjingtt/MAF.git.

Domain generalization (DG) aims to learn transferable knowledge from multiple source domains and generalize it to the unseen target domain. To achieve such expectation, the intuitive solution is to seek domain-invariant representations via generative adversarial mechanism or minimization of cross-domain discrepancy. However, the widespread imbalanced data scale problem across source domains and category in real-world applications becomes the key bottleneck of improving generalization ability of model due to its negative effect on learning the robust classification model. Motivated by this observation, we first formulate a practical and challenging imbalance domain generalization (IDG) scenario, and then propose a straightforward but effective novel method generative inference network (GINet), which augments reliable samples for minority domain/category to promote discriminative ability of the learned model. Concretely, GINet utilizes the available cross-domain images from the identical category and estimates their common latent variable, which derives to discover domain-invariant knowledge for unseen target domain. According to these latent variables, our GINet further generates more novel samples with optimal transport constraint and deploys them to enhance the desired model with more robustness and generalization ability. Considerable empirical analysis and ablation studies on three popular benchmarks under normal DG and IDG setups suggests the advantage of our method over other DG methods on elevating model generalization. The source code is available in GitHub https://github.com/HaifengXia/IDG.

Existing domain adaptation approaches often try to reduce distribution difference between source and target domains and respect domain-specific discriminative structures by some distribution [e.g., maximum mean discrepancy (MMD)] and discriminative distances (e.g., intra-class and inter-class distances). However, they usually consider these losses together and trade off their relative importance by estimating parameters empirically. It is still under insufficient exploration so far to deeply study their relationships to each other so that we cannot manipulate them correctly and the model's performance degrades. To this end, this article theoretically proves two essential facts: 1) minimizing MMD equals to jointly minimizing their data variance with some implicit weights but, respectively, maximizing the source and target intra-class distances so that feature discriminability degrades and 2) the relationship between intra-class and inter-class distances is as one falls and another rises. Based on this, we propose a novel discriminative MMD with two parallel strategies to correctly restrain the degradation of feature discriminability or the expansion of intra-class distance; specifically: 1) we directly impose a tradeoff parameter on the intra-class distance that is implicit in the MMD according to 1) and 2) we reformulate the inter-class distance with special weights that are analogical to those implicit ones in the MMD and maximizing it can also lead to the intra-class distance falling according to 2). Notably, we do not consider the two strategies in one model due to 2). The experiments on several benchmark datasets not only prove the validity of our revealed theoretical results but also demonstrate that the proposed approach could perform better than some compared state-of-art methods substantially. Our preliminary MATLAB code will be available at https://github.com/WWLoveTransfer/.

Conventional machine learning algorithms suffer the problem that the model trained on existing data fails to generalize well to the data sampled from other distributions. To tackle this issue, unsupervised domain adaptation (UDA) transfers the knowledge learned from a well-labeled source domain to a different but related target domain where labeled data is unavailable. The majority of existing UDA methods assume that data from the source domain and the target domain are available and complete during training. Thus, the divergence between the two domains can be formulated and minimized. In this paper, we consider a more practical yet challenging UDA setting where either the source domain data or the target domain data are unknown. Conventional UDA methods would fail this setting since the domain divergence is agnostic due to the absence of the source data or the target data. Technically, we investigate UDA from a novel view-adversarial attack-and tackle the divergence-agnostic adaptive learning problem in a unified framework. Specifically, we first report the motivation of our approach by investigating the inherent relationship between UDA and adversarial attacks. Then we elaborately design adversarial examples to attack the training model and harness these adversarial examples. We argue that the generalization ability of the model would be significantly improved if it can defend against our attack, so as to improve the performance on the target domain. Theoretically, we analyze the generalization bound for our method based on domain adaptation theories. Extensive experimental results on multiple UDA benchmarks under conventional, source-absent and target-absent UDA settings verify that our method is able to achieve a favorable performance compared with previous ones. Notably, this work extends the scope of both domain adaptation and adversarial attack, and expected to inspire more ideas in the community.

One of the key tasks for an intelligent visual surveillance system is to automatically re-identify objects of interest, e.g., persons or vehicles, from nonoverlapping camera views. This demand incurs the vast investigation of person re-identification (re-ID) and vehicle re-ID techniques, especially those deep learning-based ones. While most recent algorithms focus on designing new convolutional neural networks, less attention is paid to the loss functions, which are of vital roles as well. Triplet loss and softmax loss are the two losses that are extensively used, both of which, however, have limitations. Triplet loss optimizes the model to produce features with which samples from the same class have higher similarity than those from different classes. The problem of triplet loss is that the number of triplets to be constructed grows cubically with training samples, which causes scalability issue, unstable performance, and slow convergence. Softmax loss has favorable scalable property and is widely used for large-scale classification problems. However, since Softmax loss only aims to separate well training classes, its performance for re-ID tasks is not desirable because the model is tested to measure the similarity of samples from unseen classes. We propose the support neighbor (SN) loss, which avoids the limitations of the abovementioned two losses. Unlike triplet loss that is calculated based on triplets, SN loss is derived from K -nearest neighbors (SNs) of anchor samples. The SNs of an anchor are unique, containing more valuable contextual information and neighborhood structure of the anchor, and thus contribute to more stable performance and reliable embedding from image space to feature space. Based on the SNs, a softmax-like separation term and a squeeze term are proposed, which encourage interclass separation and intraclass compactness, respectively. Experiments show that SN loss surpasses triplet and softmax losses with the same backbone network and reaches the state-of-the-art performance for both person and vehicle re-ID using a ResNet50 backbone when combined with training tricks.

Zero-shot learning casts light on lacking unseen class data by transferring knowledge from seen classes via a joint semantic space. However, the distributions of samples from seen and unseen classes are usually imbalanced. Many zero-shot learning methods fail to obtain satisfactory results in the generalized zero-shot learning task, where seen and unseen classes are all used for the test. Also, irregular structures of some classes may result in inappropriate mapping from visual features space to semantic attribute space. A novel generative mixup networks with semantic graph alignment is proposed in this article to mitigate such problems. To be specific, our model first attempts to synthesize samples conditioned with class-level semantic information as the prototype to recover the class-based feature distribution from the given semantic description. Second, the proposed model explores a mixup mechanism to augment training samples and improve the generalization ability of the model. Third, triplet gradient matching loss is developed to guarantee the class invariance to be more continuous in the latent space, and it can help the discriminator distinguish the real and fake samples. Finally, a similarity graph is constructed from semantic attributes to capture the intrinsic correlations and guides the feature generation process. Extensive experiments conducted on several zero-shot learning benchmarks from different tasks prove that the proposed model can achieve superior performance over the state-of-the-art generalized zero-shot learning.

Zero-shot sketch-based image retrieval (ZS-SBIR) has attracted great attention recently, due to the potential application of sketch-based retrieval under zero-shot scenarios, where the categories of query sketches and gallery photos are not observed in the training stage. However, it is still under insufficient exploration for the general and practical scenario when the query sketches and gallery photos contain both seen and unseen categories. Such a problem is defined as generalized zero-shot sketch-based image retrieval (GZS-SBIR), which is the focus of this work. To this end, we propose a novel Augmented Multi-modality Fusion (AMF) framework to generalize seen concepts to unobserved ones efficiently. Specifically, a novel knowledge discovery module named cross-domain augmentation is designed in both visual and semantic space to mimic novel knowledge unseen from the training stage, which is the key to handling the GZS-SBIR challenge. Moreover, a triplet domain alignment module is proposed to couple the cross-domain distribution between photo and sketch in visual space. To enhance the robustness of our model, we explore embedding propagation to refine both visual and semantic features by removing undesired noise. Eventually, visual-semantic fusion representations are concatenated for further domain discrimination and task-specific recognition, which tend to trigger the cross-domain alignment in both visual and semantic feature space. Experimental evaluations are conducted on popular ZS-SBIR benchmarks as well as a new evaluation protocol designed for GZS-SBIR from DomainNet dataset with more diverse sub-domains, and the promising results demonstrate the superiority of the proposed solution over other baselines. The source code is available at https://github.com/scottjingtt/AMF_GZS_SBIR.git.

Unsupervised domain adaptation aims to learn a classification model for the target domain without any labeled samples by transferring the knowledge from the source domain with sufficient labeled samples. The source and the target domains usually share the same label space but are with different data distributions. In this paper, we consider a more difficult but insufficient-explored problem named as few-shot domain adaptation, where a classifier should generalize well to the target domain given only a small number of examples in the source domain. In such a problem, we recast the link between the source and target samples by a mixup optimal transport model. The mixup mechanism is integrated into optimal transport to perform the few-shot adaptation by learning the cross-domain alignment matrix and domain-invariant classifier simultaneously to augment the source distribution and align the two probability distributions. Moreover, spectral shrinkage regularization is deployed to improve the transferability and discriminability of the mixup optimal transport model by utilizing all singular eigenvectors. Experiments conducted on several domain adaptation tasks demonstrate the effectiveness of our proposed model dealing with the few-shot domain adaptation problem compared with state-of-the-art methods.