Search

Rehabilitation training is essential for the recovery of patients with conditions such as stroke and Parkinson’s disease. However, traditional skeletal-based assessments often fail to capture the subtle movement qualities necessary for personalized care and are not optimized for scoring tasks. To address these limitations, we propose a hierarchical contrastive learning framework that integrates multi-view skeletal data, combining both positional and angular joint information. This integration enhances the framework’s ability to detect subtle variations in movement during rehabilitation exercises. In addition, we introduce a novel contrastive loss function specifically designed for regression tasks. This new approach yields substantial improvements over existing state-of-the-art models, achieving over a 30% reduction in mean absolute deviation on both the KIMORE and UIPRMD datasets. The framework demonstrates robustness in capturing both global and local movement characteristics, which are critical for accurate clinical evaluations. By precisely quantifying action quality, the framework supports the development of more targeted, personalized rehabilitation plans and shows strong potential for broad application in rehabilitation practices as well as in a wider range of motion assessment tasks.

Abstract Traditional methods for identifying “hit” molecules from a large collection of potential drug-like candidates rely on biophysical theory to compute approximations to the Gibbs free energy of the binding interaction between the drug and its protein target. These approaches have a significant limitation in that they require exceptional computing capabilities for even relatively small collections of molecules. Increasingly large and complex state-of-the-art deep learning approaches have gained popularity with the promise to improve the productivity of drug design, notorious for its numerous failures. However, as deep learning models increase in their size and complexity, their acceleration at the hardware level becomes more challenging. Hyperdimensional Computing (HDC) has recently gained attention in the computer hardware community due to its algorithmic simplicity relative to deep learning approaches. The HDC learning paradigm, which represents data with high-dimension binary vectors, allows the use of low-precision binary vector arithmetic to create models of the data that can be learned without the need for the gradient-based optimization required in many conventional machine learning and deep learning methods. This algorithmic simplicity allows for acceleration in hardware that has been previously demonstrated in a range of application areas (computer vision, bioinformatics, mass spectrometery, remote sensing, edge devices, etc.). To the best of our knowledge, our work is the first to consider HDC for the task of fast and efficient screening of modern drug-like compound libraries. We also propose the first HDC graph-based encoding methods for molecular data, demonstrating consistent and substantial improvement over previous work. We compare our approaches to alternative approaches on the well-studied MoleculeNet dataset and the recently proposed LIT-PCBA dataset derived from high quality PubChem assays. We demonstrate our methods on multiple target hardware platforms, including Graphics Processing Units (GPUs) and Field Programmable Gate Arrays (FPGAs), showing at least an order of magnitude improvement in energy efficiency versus even our smallest neural network baseline model with a single hidden layer. Our work thus motivates further investigation into molecular representation learning to develop ultra-efficient pre-screening tools. We make our code publicly available at https://github.com/LLNL/hdbind .

Abstract Graph neural networks (GNNs) have gained prominence as an effective technique for representation learning and have found wide application in tag recommendation tasks. Existing approaches aim to encode the hidden collaborative information among entities into embedding representations by propagating node information between connected nodes. However, in sparse observable graph structures, a significant number of connections are missing, leading to incomplete and biased propagation. To address these issues, we propose a novel model called Low-frequency Spectral Graph Convolution Networks with one-hop connections information for Personalized Tag Recommendation (LSGCNT). This model utilizes graph convolution in the spectral domain and incorporates a graph structure comprising two bipartite graphs, the user–tag interaction graph and the item–tag interaction graph. Our model aims to reduce information loss caused by propagation by utilizing graph convolution networks with trainable convolution kernels to recover preference information. In order to preserve useful low-frequency signals, we couple graph convolution with low-pass filters in the frequency domain. Through reconstructing the true rating tensor and ranking the tag scores within the tensor, we can achieve top-N recommendations. Furthermore, to preserve the one-hop connection information of the bipartite graphs, we treat the observed two bipartite graphs as two homogeneous graphs, where both users and tags contribute to the convolution of a node in the user–tag graph, and both items and tags contribute to the convolution of a node in the item–tag graph. Lastly, we analyze the impact of different internal components, pooling methods, parameter choices, and prediction approaches of LSGCNT on recommendation performance. Experimental results on two real-world datasets demonstrate that LSGCNT achieves superior recommendation performance compared with eight other state-of-the-art recommendation models.

Abstract Self-supervised learning has become the cornerstone of building generalizable and transferable artificial intelligence systems in medical imaging. In particular, contrastive representation learning techniques trained on large multi-modal datasets have demonstrated impressive capabilities of producing highly transferable representations for different downstream tasks. In ophthalmology, large multi-modal datasets are abundantly available and conveniently accessible as modern retinal imaging scanners acquire both 2D fundus images and 3D optical coherence tomography (OCT) scans to assess the eye. In this context, we introduce a novel multi-modal contrastive learning-based pipeline to facilitate learning joint representations for the two retinal imaging modalities. After self-supervised pre-training on 153,306 scan pairs, we show that such a pre-training framework can provide both a retrieval system and encoders that produce comprehensive OCT and fundus image representations that generalize well for various downstream tasks on three independent external datasets, explicitly focusing on clinically pertinent prediction tasks. In addition, we show that interchanging OCT with lower-cost fundus imaging can preserve the predictive power of the trained models.

Abstract Bitter taste is an unpleasant taste modality that affects food consumption. Bitter peptides are generated during enzymatic processes that produce functional, bioactive protein hydrolysates or during the aging process of fermented products such as cheese, soybean protein, and wine. Understanding the underlying peptide sequences responsible for bitter taste can pave the way for more efficient identification of these peptides. This paper presents BitterPep-GCN, a feature-agnostic graph convolution network for bitter peptide prediction. The graph-based model learns the embedding of amino acids in the bitter peptide sequences and uses mixed pooling for bitter classification. BitterPep-GCN was benchmarked using BTP640, a publicly available bitter peptide dataset. The latent peptide embeddings generated by the trained model were used to analyze the activity of sequence motifs responsible for the bitter taste of the peptides. Particularly, we calculated the activity for individual amino acids and dipeptide, tripeptide, and tetrapeptide sequence motifs present in the peptides. Our analyses pinpoint specific amino acids, such as F, G, P, and R, as well as sequence motifs, notably tripeptide and tetrapeptide motifs containing FF, as key bitter signatures in peptides. This work not only provides a new predictor of bitter taste for a more efficient identification of bitter peptides in various food products but also gives a hint into the molecular basis of bitterness. Scientific Contribution Our work provides the first application of Graph Neural Networks for the prediction of peptide bitter taste. The best-developed model, BitterPep-GCN, learns the embedding of amino acids in the bitter peptide sequences and uses mixed pooling for bitter classification. The embeddings were used to analyze the sequence motifs responsible for the bitter taste.

Abstract Background The rapid advancements in deep neural network models have significantly enhanced the ability to extract features from microbial sequence data, which is critical for addressing biological challenges. However, the scarcity and complexity of labeled microbial data pose substantial difficulties for supervised learning approaches. To address these issues, we propose DNASimCLR, an unsupervised framework designed for efficient gene sequence data feature extraction. Results DNASimCLR leverages convolutional neural networks and the SimCLR framework, based on contrastive learning, to extract intricate features from diverse microbial gene sequences. Pre-training was conducted on two classic large scale unlabelled datasets encompassing metagenomes and viral gene sequences. Subsequent classification tasks were performed by fine-tuning the pretrained model using the previously acquired model. Our experiments demonstrate that DNASimCLR is at least comparable to state-of-the-art techniques for gene sequence classification. For convolutional neural network-based approaches, DNASimCLR surpasses the latest existing methods, clearly establishing its superiority over the state-of-the-art CNN-based feature extraction techniques. Furthermore, the model exhibits superior performance across diverse tasks in analyzing biological sequence data, showcasing its robust adaptability. Conclusions DNASimCLR represents a robust and database-agnostic solution for gene sequence classification. Its versatility allows it to perform well in scenarios involving novel or previously unseen gene sequences, making it a valuable tool for diverse applications in genomics.

Abstract In this study, we introduce StructmRNA, a new BERT-based model that was designed for the detailed analysis of mRNA sequences and structures. The success of DNABERT in understanding the intricate language of non-coding DNA with bidirectional encoder representations is extended to mRNA with StructmRNA. This new model uses a special dual-level masking technique that covers both sequence and structure, along with conditional masking. This enables StructmRNA to adeptly generate meaningful embeddings for mRNA sequences, even in the absence of explicit structural data, by capitalizing on the intricate sequence-structure correlations learned during extensive pre-training on vast datasets. Compared to well-known models like those in the Stanford OpenVaccine project, StructmRNA performs better in important tasks such as predicting RNA degradation. Thus, StructmRNA can inform better RNA-based treatments by predicting the secondary structures and biological functions of unseen mRNA sequences. The proficiency of this model is further confirmed by rigorous evaluations, revealing its unprecedented ability to generalize across various organisms and conditions, thereby marking a significant advance in the predictive analysis of mRNA for therapeutic design. With this work, we aim to set a new standard for mRNA analysis, contributing to the broader field of genomics and therapeutic development.

Background: Deep 3D morphable models (deep 3DMMs) play an essential role in computer vision. They are used in facial synthesis, compression, reconstruction and animation, avatar creation, virtual try-on, facial recognition systems and medical imaging. These applications require high spatial and perceptual quality of synthesised meshes. Despite their significance, these models have not been compared with different mesh representations and evaluated jointly with point-wise distance and perceptual metrics. Methods: We compare the influence of different mesh representation features to various deep 3DMMs on spatial and perceptual fidelity of the reconstructed meshes. This paper proves the hypothesis that building deep 3DMMs from meshes represented with global representations leads to lower spatial reconstruction error measured with L1 and L2 norm metrics and underperforms on perceptual metrics. In contrast, using differential mesh representations which describe differential surface properties yields lower perceptual FMPD and DAME and higher spatial fidelity error. The influence of mesh feature normalisation and standardisation is also compared and analysed from perceptual and spatial fidelity perspectives. Results: The results presented in this paper provide guidance in selecting mesh representations to build deep 3DMMs accordingly to spatial and perceptual quality objectives and propose combinations of mesh representations and deep 3DMMs which improve either perceptual or spatial fidelity of existing methods.

In fashion e-commerce, predicting item compatibility using visual features remains a significant challenge. Current recommendation systems often struggle to incorporate high-dimensional visual data into graph-based learning models effectively. This limitation presents a substantial opportunity to enhance the precision and effectiveness of fashion recommendations. In this paper, we present the Visual-aware Graph Convolutional Network (VAGCN). This novel framework helps improve how visual features can be incorporated into graph-based learning systems for fashion item compatibility predictions. The VAGCN framework employs a deep-stacked autoencoder to convert the input image’s high-dimensional raw CNN visual features into more manageable low-dimensional representations. In addition to improving feature representation, the GCN can also reason more intelligently about predictions, which would not be possible without this compression. The GCN encoder processes nodes in the graph to capture structural and feature correlation. Following the GCN encoder, the refined embeddings are input to a multi-layer perceptron (MLP) to calculate compatibility scores. The approach extends to using neighborhood information only during the testing phase to help with training efficiency and generalizability in practical scenarios, a key characteristic of our model. By leveraging its ability to capture latent visual features and neighborhood-based learning, VAGCN thoroughly investigates item compatibility across various categories. This method significantly improves predictive accuracy, consistently outperforming existing benchmarks. These contributions tackle significant scalability and computational efficiency challenges, showcasing the potential transformation of recommendation systems through enhanced feature representation, paving the way for further innovations in the fashion domain.

Abstract Chemical engineers heavily rely on precise knowledge of physicochemical properties to model chemical processes. Despite the growing popularity of deep learning, it is only rarely applied for property prediction due to data scarcity and limited accuracy for compounds in industrially-relevant areas of the chemical space. Herein, we present a geometric deep learning framework for predicting gas- and liquid-phase properties based on novel quantum chemical datasets comprising 124,000 molecules. Our findings reveal that the necessity for quantum-chemical information in deep learning models varies significantly depending on the modeled physicochemical property. Specifically, our top-performing geometric model meets the most stringent criteria for “chemically accurate” thermochemistry predictions. We also show that by carefully selecting the appropriate model featurization and evaluating prediction uncertainties, the reliability of the predictions can be strongly enhanced. These insights represent a crucial step towards establishing deep learning as the standard property prediction workflow in both industry and academia. Scientific contribution We propose a flexible property prediction tool that can handle two-dimensional and three-dimensional molecular information. A thermochemistry prediction methodology that achieves high-level quantum chemistry accuracy for a broad application range is presented. Trained deep learning models and large novel molecular databases of real-world molecules are provided to offer a directly usable and fast property prediction solution to practitioners.

Recent studies have shown that combining Transformer and conditional strategies to deal with offline reinforcement learning can bring better results. However, in a conventional reinforcement learning scenario, the agent can receive a single frame of observations one by one according to its natural chronological sequence, but in Transformer, a series of observations are received at each step. Individual features cannot be extracted efficiently to make more accurate decisions, and it is still difficult to generalize effectively for data outside the distribution. We focus on the characteristic of few-shot learning in pre-trained models, and combine prompt learning to enhance the ability of real-time policy adjustment. By sampling the specific information in the offline dataset as trajectory samples, the task information is encoded to help the pre-trained model quickly understand the task characteristics and the sequence generation paradigm to quickly adapt to the downstream tasks. In order to understand the dependencies in the sequence more accurately, we also divide the fixed-size state information blocks in the input trajectory, extract the features of the segmented sub-blocks respectively, and finally encode the whole sequence into the GPT model to generate decisions more accurately. Experiments show that the proposed method achieves better performance than the baseline method in related tasks, can be generalized to new environments and tasks better, and effectively improves the stability and accuracy of agent decision making.

Abstract Deep learning (DL) has shown potential to provide powerful representations of bulk RNA-seq data in cancer research. However, there is no consensus regarding the impact of design choices of DL approaches on the performance of the learned representation, including the model architecture, the training methodology and the various hyperparameters. To address this problem, we evaluate the performance of various design choices of DL representation learning methods using TCGA and DepMap pan-cancer datasets and assess their predictive power for survival and gene essentiality predictions. We demonstrate that baseline methods achieve comparable or superior performance compared to more complex models on survival predictions tasks. DL representation methods, however, are the most efficient to predict the gene essentiality of cell lines. We show that auto-encoders (AE) are consistently improved by techniques such as masking and multi-head training. Our results suggest that the impact of DL representations and of pretraining are highly task- and architecture-dependent, highlighting the need for adopting rigorous evaluation guidelines. These guidelines for robust evaluation are implemented in a pipeline made available to the research community.

Abstract Background Detecting event triggers in biomedical texts, which contain domain knowledge and context-dependent terms, is more challenging than in general-domain texts. Most state-of-the-art models rely mainly on external resources such as linguistic tools and knowledge bases to improve system performance. However, they lack effective mechanisms to obtain semantic clues from label specification and sentence context. Given its success in image classification, label representation learning is a promising approach to enhancing biomedical event trigger detection models by leveraging the rich semantics of pre-defined event type labels. Results In this paper, we propose the Biomedical Label-based Synergistic representation Learning (BioLSL) model, which effectively utilizes event type labels by learning their correlation with trigger words and enriches the representation contextually. The BioLSL model consists of three modules. Firstly, the Domain-specific Joint Encoding module employs a transformer-based, domain-specific pre-trained architecture to jointly encode input sentences and pre-defined event type labels. Secondly, the Label-based Synergistic Representation Learning module learns the semantic relationships between input texts and event type labels, and generates a Label-Trigger Aware Representation (LTAR) and a Label-Context Aware Representation (LCAR) for enhanced semantic representations. Finally, the Trigger Classification module makes structured predictions, where each label is predicted with respect to its neighbours. We conduct experiments on three benchmark BioNLP datasets, namely MLEE, GE09, and GE11, to evaluate our proposed BioLSL model. Results show that BioLSL has achieved state-of-the-art performance, outperforming the baseline models. Conclusions The proposed BioLSL model demonstrates good performance for biomedical event trigger detection without using any external resources. This suggests that label representation learning and context-aware enhancement are promising directions for improving the task. The key enhancement is that BioLSL effectively learns to construct semantic linkages between the event mentions and type labels, which provide the latent information of label-trigger and label-context relationships in biomedical texts. Moreover, additional experiments on BioLSL show that it performs exceptionally well with limited training data under the data-scarce scenarios.

Background Subtraction (BgS) is a widely researched technique to develop online Change Detection algorithms for static video cameras. Many BgS methods have employed the unsupervised, adaptive approach of Gaussian Mixture Model (GMM) to produce decent backgrounds, but they lack proper consideration of scene semantics to produce better foregrounds. On the other hand, with considerable computational expenses, BgS with Deep Neural Networks (DNN) is able to produce accurate background and foreground segments. In our research, we blend both approaches for the best. First, we formulated a network called Convolutional Density Approximation (CDA) for direct density estimation of background models. Then, we propose a self-supervised training strategy for CDA to adaptively capture high-frequency color distributions for the corresponding backgrounds. Finally, we show that background models can indeed assist foreground extraction by an efficient Neural Motion Subtraction (NeMos) network. Our experiments verify competitive results in the balance between effectiveness and efficiency.

Person re-identification (re-ID) involves the cross-camera retrieval and matching of target pedestrian images, facilitating pedestrian association in scenarios where biometric features such as faces and fingerprints may prove ineffective. It has become a pivotal technology in intelligent video surveillance systems, playing a crucial role in domains like smart security and smart cities. Traditional re-ID algorithms typically employ either representation learning or metric learning methods. A novel approach was proposed which combined representation learning and metric learning methods based on the multi-task learning machine learning paradigm. By capitalizing on the advantages of both feature representation and distance metric, and concurrently training the model using classification loss and triplet loss, comprehensive training for both feature extraction and similarity measurement was ensured. Experimental results validate the effectiveness of the proposed approach, demonstrating superior performance in re-ID tasks and underscoring the robustness and superior generalization capability.

Abstract Background Compared with the time-consuming and labor-intensive for biological validation in vitro or in vivo, the computational models can provide high-quality and purposeful candidates in an instant. Existing computational models face limitations in effectively utilizing sparse local structural information for accurate predictions in circRNA-disease associations. This study addresses this challenge with a proposed method, CDA-DGRL (Prediction of CircRNA-Disease Association based on Double-line Graph Representation Learning), which employs a deep learning framework leveraging graph networks and a dual-line representation model integrating graph node features. Method CDA-DGRL comprises several key steps: initially, the integration of diverse biological information to compute integrated similarities among circRNAs and diseases, leading to the construction of a heterogeneous network specific to circRNA-disease associations. Subsequently, circRNA and disease node features are derived using sparse autoencoders. Thirdly, a graph convolutional neural network is employed to capture the local graph network structure by inputting the circRNA-disease heterogeneous network alongside node features. Fourthly, the utilization of node2vec facilitates depth-first sampling of the circRNA-disease heterogeneous network to grasp the global graph network structure, addressing issues associated with sparse raw data. Finally, the fusion of local and global graph network structures is inputted into an extra trees classifier to identify potential circRNA-disease associations. Results The results, obtained through a rigorous five-fold cross-validation on the circR2Disease dataset, demonstrate the superiority of CDA-DGRL with an AUC value of 0.9866 and an AUPR value of 0.9897 compared to existing state-of-the-art models. Notably, the hyper-random tree classifier employed in this model outperforms other machine learning classifiers. Conclusion Thus, CDA-DGRL stands as a promising methodology for reliably identifying circRNA-disease associations, offering potential avenues to alleviate the necessity for extensive traditional biological experiments. The source code and data for this study are available at https://github.com/zywait/CDA-DGRL .

Person re-identification (re-ID) involves the cross-camera retrieval and matching of target pedestrian images, facilitating pedestrian association in scenarios where biometric features such as faces and fingerprints may prove ineffective. It has become a pivotal technology in intelligent video surveillance systems, playing a crucial role in domains like smart security and smart cities. Traditional re-ID algorithms typically employ either representation learning or metric learning methods. A novel approach was proposed which combined representation learning and metric learning methods based on the multi-task learning machine learning paradigm. By capitalizing on the advantages of both feature representation and distance metric, and concurrently training the model using classification loss and triplet loss, comprehensive training for both feature extraction and similarity measurement was ensured. Experimental results validate the effectiveness of the proposed approach, demonstrating superior performance in re-ID tasks and underscoring the robustness and superior generalization capability.

Motivated by the growing threat of distributed denial-of-service (DDoS) attacks and the emergence of quantum computing, this study introduces a novel “quanvolutional autoencoder” architecture for learning representations. The architecture leverages the computational advantages of quantum mechanics to improve upon traditional machine learning techniques. Specifically, the quanvolutional autoencoder employs randomized quantum circuits to analyze time-series data from DDoS attacks, offering a robust alternative to classical convolutional neural networks. Experimental results suggest that the quanvolutional autoencoder performs similarly to classical models in visualizing and learning from DDoS hive plots and leads to faster convergence and learning stability. These findings suggest that quantum machine learning holds significant promise for advancing data analysis and visualization in cybersecurity. The study highlights the need for further research in this fast-growing field, particularly for unsupervised anomaly detection.

Due to spectacular gains during periods of rapid price increase and unpredictably large drops, Bitcoin has become a popular emergent asset class over the past few years. In this paper, we are interested in predicting the crashes of Bitcoin market. To tackle this task, we propose a framework for deep learning time series classification based on contrastive learning. The proposed framework is evaluated against six machine learning (ML) and deep learning (DL) baseline models, and outperforms them by 15.8% in balanced accuracy. Thus, we conclude that the contrastive learning strategy significantly enhance the model’s ability of extracting informative representations, and our proposed framework performs well in predicting Bitcoin crashes.

Transferring the ImageNet pre-trained weights to the various remote sensing tasks has produced acceptable results and reduced the need for labeled samples. However, the domain differences between ground imageries and remote sensing images cause the performance of such transfer learning to be limited. The difficulty of annotating remote sensing images is well-known as it requires domain experts and more time, whereas unlabeled data is readily available. Recently, self-supervised learning, which is a subset of unsupervised learning, emerged and significantly improved representation learning. Recent research has demonstrated that self-supervised learning methods capture visual features that are more discriminative and transferable than the supervised ImageNet weights. We are motivated by these facts to pre-train the in-domain representations of remote sensing imagery using contrastive self-supervised learning and transfer the learned features to other related remote sensing datasets. Specifically, we used the SimSiam algorithm to pre-train the in-domain knowledge of remote sensing datasets and then transferred the obtained weights to the other scene classification datasets. Thus, we have obtained state-of-the-art results on five land cover classification datasets with varying numbers of classes and spatial resolutions. In addition, by conducting appropriate experiments, including feature pre-training using datasets with different attributes, we have identified the most influential factors that make a dataset a good choice for obtaining in-domain features. We have transferred the features obtained by pre-training SimSiam on remote sensing datasets to various downstream tasks and used them as initial weights for fine-tuning. Moreover, we have linearly evaluated the obtained representations in cases where the number of samples per class is limited. Our experiments have demonstrated that using a higher-resolution dataset during the self-supervised pre-training stage results in learning more discriminative and general representations.

Abstract Effective use of the wealth of information provided by Distributed Acoustic Sensing (DAS) for mass movement monitoring remains a challenge. We propose a semi‐supervised neural network tailored to screen DAS data related to a series of rock collapses leading to a major failure of approximately 1.2 million m3 on 15 June 2023 in Brienz, Eastern Switzerland. Besides DAS, the dataset from 16 May to 30 June 2023 includes Doppler radar data for partially ground‐truth labeling. The proposed algorithm is capable of distinguishing between rock‐slope failures and background noise, including road and train traffic, with a detection precision of over 95%. It identifies hundreds of precursory failures and shows sustained detection hours before and during the major collapse. Event size and signal‐to‐noise ratio (SNR) are the key performance dependencies. As a critical part of our algorithm operates unsupervised, we suggest that it is suitable for general monitoring of natural hazards.

Abstract Stakeholders of machine learning models desire explainable artificial intelligence (XAI) to produce human-understandable and consistent interpretations. In computational toxicity, augmentation of text-based molecular representations has been used successfully for transfer learning on downstream tasks. Augmentations of molecular representations can also be used at inference to compare differences between multiple representations of the same ground-truth. In this study, we investigate the robustness of eight XAI methods using test-time augmentation for a molecular-representation model in the field of computational toxicity prediction. We report significant differences between explanations for different representations of the same ground-truth, and show that randomized models have similar variance. We hypothesize that text-based molecular representations in this and past research reflect tokenization more than learned parameters. Furthermore, we see a greater variance between in-domain predictions than out-of-domain predictions, indicating XAI measures something other than learned parameters. Finally, we investigate the relative importance given to expert-derived structural alerts and find similar importance given irregardless of applicability domain, randomization and varying training procedures. We therefore caution future research to validate their methods using a similar comparison to human intuition without further investigation. Scientific contribution In this research we critically investigate XAI through test-time augmentation, contrasting previous assumptions about using expert validation and showing inconsistencies within models for identical representations. SMILES augmentation has been used to increase model accuracy, but was here adapted from the field of image test-time augmentation to be used as an independent indication of the consistency within SMILES-based molecular representation models. Graphical Abstract

Abstract Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning, leveraging large amounts of unlabelled data. This review summarizes recent research into its usage in X-ray, computed tomography, magnetic resonance, and ultrasound imaging, concentrating on studies that compare self-supervised pretraining to fully supervised learning for diagnostic tasks such as classification and segmentation. The most pertinent finding is that self-supervised pretraining generally improves downstream task performance compared to full supervision, most prominently when unlabelled examples greatly outnumber labelled examples. Based on the aggregate evidence, recommendations are provided for practitioners considering using self-supervised learning. Motivated by limitations identified in current research, directions and practices for future study are suggested, such as integrating clinical knowledge with theoretically justified self-supervised learning methods, evaluating on public datasets, growing the modest body of evidence for ultrasound, and characterizing the impact of self-supervised pretraining on generalization.

Abstract This paper proposes a novel GAN framework with self-clustering approach for precipitation nowcasting (ClusterCast). Previous studies have primarily captured the motion vector using only a single latent space, making the models difficult to adapt to disparate space-time distribution of precipitation. Environmental factors (e.g., regional characteristics and precipitation scale) have an impact on precipitation systems and can cause non-stationary distribution. To tackle this problem, our key idea is to train a generator network to predict future radar frames by learning a sub-network that automatically labels precipitation types from a generative model. The training process consists of (i) clustering the hierarchical features derived from the generator stem using a sub-network and (ii) predicting future radar frames according to the self-supervised labels, enabling heterogeneous latent representation. Additionally, we attempt an ensemble forecast that prescribes random perturbations to improve performance. With the flexibility of representation learning, ClusterCast enables the model to learn precipitation distribution more accurately. Results indicate that our method generates non-blurry future frames by preventing mode collapse, and the proposed method demonstrates robustness across various precipitation scenarios. Extensive experiments demonstrate that our method outperforms four benchmarks on a 2-h prediction basis with a mean squared error (MSE) of 8.9% on unseen datasets.

Most few-shot knowledge graph completion models have some problems, such as low ability to learn relation representation and rarely attaching importance to the relative location and interaction between query entity pair when the relation between entities is complex or triples’ neighborhood is sparse. A selective attention mechanism and interaction awareness (SAIA) based few-shot knowledge graph completion algorithm is proposed. Firstly, by introducing selective attention mechanism in the process of aggregating neighbor information, the neighbor encoder pays more attention to important neighbors to reduce adverse effects of noise neighbors. Secondly, SAIA utilizes the information related to task relation in the background knowledge graph to learn more accurate relation embedding in the process of relationship representation learning. Finally, in order to mine the interaction information and location information between entities in knowledge graph, a common interaction rate index (CIR) of entity pair is designed to measure the degree of association between entities in 3-hop path. Then, SAIA combines entity pair semantic information to predict new fact. Experimental results show that SAIA outperforms the state-of-the-art few-shot knowledge graph completion methods. Compared with the optimal results of baseline models, the proposed method achieves 5-shot link prediction performance improvement of 0.038, 0.011, 0.028 and 0.052 on NELL-one dataset and 0.034，0.037，0.029 and 0.027 on Wiki-one dataset by the metric MRR, Hits@10, Hits@5 as well as Hits@1, which verifies the effectiveness and feasibility of SAIA.

Abstract Transfer learning plays a pivotal role in addressing the paucity of data, expediting training processes, and enhancing model performance. Nonetheless, the prevailing practice of transfer learning predominantly relies on pre-trained models designed for the natural image domain, which may not be well-suited for the medical image domain in grayscale. Recognizing the significance of leveraging transfer learning in medical research, we undertook the construction of class-balanced pediatric radiograph datasets collectively referred to as PedXnets, grounded in radiographic views using the pediatric radiographs collected over 24 years at Asan Medical Center. For PedXnets pre-training, approximately 70,000 X-ray images were utilized. Three different pre-training weights of PedXnet were constructed using Inception V3 for various radiation perspective classifications: Model-PedXnet-7C, Model-PedXnet-30C, and Model-PedXnet-68C. We validated the transferability and positive effects of transfer learning of PedXnets through pediatric downstream tasks including fracture classification and bone age assessment (BAA). The evaluation of transfer learning effects through classification and regression metrics showed superior performance of Model-PedXnets in quantitative assessments. Additionally, visual analyses confirmed that the Model-PedXnets were more focused on meaningful regions of interest.

Abstract Media convergence works by processing information from different modalities and applying them to different domains. It is difficult for the conventional knowledge graph to utilise multi‐media features because the introduction of a large amount of information from other modalities reduces the effectiveness of representation learning and makes knowledge graph inference less effective. To address the issue, an inference method based on Media Convergence and Rule‐guided Joint Inference model (MCRJI) has been proposed. The authors not only converge multi‐media features of entities but also introduce logic rules to improve the accuracy and interpretability of link prediction. First, a multi‐headed self‐attention approach is used to obtain the attention of different media features of entities during semantic synthesis. Second, logic rules of different lengths are mined from knowledge graph to learn new entity representations. Finally, knowledge graph inference is performed based on representing entities that converge multi‐media features. Numerous experimental results show that MCRJI outperforms other advanced baselines in using multi‐media features and knowledge graph inference, demonstrating that MCRJI provides an excellent approach for knowledge graph inference with converged multi‐media features.

Abstract Network embedding is a technique used to generate low-dimensional vectors representing each node in a network while maintaining the original topology and properties of the network. This technology enables a wide range of learning tasks, including node classification and link prediction. However, the current landscape of network embedding approaches predominantly revolves around static networks, neglecting the dynamic nature that characterizes real social networks. Dynamics at both the micro- and macrolevels are fundamental drivers of network evolution. Microlevel dynamics provide a detailed account of the network topology formation process, while macrolevel dynamics reveal the evolutionary trends of the network. Despite recent dynamic network embedding efforts, a few approaches accurately capture the evolution patterns of nodes at the microlevel or effectively preserve the crucial dynamics of both layers. Our study introduces a novel method for embedding networks, i.e., bilayer evolutionary pattern-preserving embedding for dynamic networks (Bi-DNE), that preserves the evolutionary patterns at both the micro- and macrolevels. The model utilizes strengthened triadic closure to represent the network structure formation process at the microlevel, while a dynamic equation constrains the network structure to adhere to the densification power-law evolution pattern at the macrolevel. The proposed Bi-DNE model exhibits significant performance improvements across a range of tasks, including link prediction, reconstruction, and temporal link analysis. These improvements are demonstrated through comprehensive experiments carried out on both simulated and real-world dynamic network datasets. The consistently superior results to those of the state-of-the-art methods provide empirical evidence for the effectiveness of Bi-DNE in capturing complex evolutionary patterns and learning high-quality node representations. These findings validate the methodological innovations presented in this work and mark valuable progress in the emerging field of dynamic network representation learning. Further exploration demonstrates that Bi-DNE is sensitive to the analysis task parameters, leading to a more accurate representation of the natural evolution process during dynamic network embedding.

Federated learning ensures the privacy of clients by conducting distributed training on individual client devices and sharing only the model weights with a central server. However, in real-world scenarios, especially in IoT scenarios where devices have varying capabilities and data heterogeneity exists among IoT clients, appropriate personalization methods are necessary. In this paper, this work aims to address this heterogeneity using a form of parameter decoupling known as representation learning. Representation learning divides deep learning models into ‘base’ and ‘head’ components. The base component, capturing common features across all clients, is shared with the server, while the head component, capturing unique features specific to individual clients, remains local. This work proposes a new representation learning-based approach, named FedSeq, that suggests decoupling the entire deep learning model into more densely divided parts with the application of suitable scheduling methods, which can benefit not only data heterogeneity but also class heterogeneity. FedSeq has two different layer scheduling approaches, namely forward (Vanilla) and backward (Anti), in the context of data and class heterogeneity among clients. Our experimental results show that FedSeq, when compared to existing personalized federated learning algorithms, achieves increased accuracy, especially under challenging conditions, while reducing computation costs. The study introduces a novel personalized federated learning approach that integrates sequential layer expansion and dynamic scheduling methods, demonstrating a 7.31% improvement in classification accuracy on the CIFAR-100 dataset and a 4.1% improvement on the Tiny-ImageNet dataset compared to existing methods, while also reducing computation costs by up to 15%. Furthermore, Anti Scheduling achieves a computational efficiency improvement of 3.91% compared to FedAvg and 3.06% compared to FedBABU, while Vanilla Scheduling achieves a significant efficiency improvement of 63.93% compared to FedAvg and 63.61% compared to FedBABU.

Data augmentation methods offer a cost-effective and efficient alternative to the acquisition of additional data, significantly enhancing data diversity and model generalization, making them particularly favored in object detection tasks. However, existing data augmentation techniques primarily focus on the visible spectrum and are directly applied to RGB-T object detection tasks, overlooking the inherent differences in image data between the two tasks. Visible images capture rich color and texture information during the daytime, while infrared images are capable of imaging under low-light complex scenarios during the nighttime. By integrating image information from both modalities, their complementary characteristics can be exploited to improve the overall effectiveness of data augmentation methods. To address this, we propose a cross-modality data augmentation method tailored for RGB-T object detection, leveraging masked image modeling within representation learning. Specifically, we focus on the temporal consistency of infrared images and combine them with visible images under varying lighting conditions for joint data augmentation, thereby enhancing the realism of the augmented images. Utilizing the masked image modeling method, we reconstruct images by integrating multimodal features, achieving cross-modality data augmentation in feature space. Additionally, we investigate the differences and complementarities between data augmentation methods in data space and feature space. Building upon existing theoretical foundations, we propose an integrative framework that combines these methods for improved augmentation effectiveness. Furthermore, we address the slow convergence observed with the existing Mosaic method in aerial imagery by introducing a multi-scale training strategy and proposing a full-scale Mosaic method as a complement. This optimization significantly accelerates network convergence. The experimental results validate the effectiveness of our proposed method and highlight its potential for further advancements in cross-modality object detection tasks.

Detecting mastitis on time in dairy cows is crucial for maintaining milk production and preventing significant economic losses, and machine learning has recently gained significant attention as a promising solution to address this issue. Most studies have detected mastitis on time series data using a supervised learning model, which requires the scale of labeled data; however, annotating the onset of mastitis in milking data from dairy cows is very difficult and costly, while supervised learning relies on accurate labels for ensuring the performance. Therefore, this study proposed a mastitis classification based on weakly supervised representation learning using an autoencoder on time series milking data, which allows for concurrent milking representation learning and weakly supervision with low-cost labels. The proposed method employed a structure where the classifier branches from the latent space of a 1D-convolutional autoencoder, enabling representation learning of milking data to be conducted from the perspective of reconstructing the original information and detecting mastitis. The branched classifier backpropagate the mastitis symptoms, which are less costly than mastitis diagnosis, during the encoder’s representation learning. The results showed that the proposed method achieved an F1-score of 0.6 that demonstrates performance comparable to previous studies despite using low-cost labels. Our method has the advantage of being easily reproducible across various data domains through low-cost annotation for supervised learning and is practical as it can be implemented with just milking data and weak labels, which can be collected in the field.

To tackle the challenges of cold start and data sparsity in recommendation systems, an increasing number of researchers are integrating item features, resulting in the emergence of multimodal recommendation systems. Although graph convolutional network-based approaches have achieved significant success, they still face two limitations: (1) Users have different preferences for various types of features, but existing methods often treat these preferences equally or fail to specifically address this issue. (2) They do not effectively distinguish the similarity between different modality item features, overlook the unique characteristics of each type, and fail to fully exploit their complementarity. To solve these issues, we propose the user perception-guided graph convolutional network for multimodal recommendation (UPGCN). This model consists of two main parts: the user perception-guided representation enhancement module (UPEM) and the multimodal two-step enhanced fusion method, which are designed to capture user preferences for different modalities to enhance user representation. At the same time, by distinguishing the similarity between different modalities, the model filters out noise and fully leverages their complementarity to achieve more accurate item representations. We performed comprehensive experiments on the proposed model, and the results indicate that it outperforms other baseline models in recommendation performance, strongly demonstrating its effectiveness.

While encryption enhances data security, it also presents significant challenges for network traffic analysis, especially in detecting malicious activities. To tackle this challenge, this paper introduces combined Attention-aware Feature Fusion and Communication Graph Embedding Learning (AFF_CGE), an advanced representation learning framework designed for detecting encrypted malicious traffic. By leveraging an attention mechanism and graph neural networks, AFF_CGE extracts rich semantic information from encrypted traffic and captures complex relations between communicating nodes. Experimental results reveal that AFF_CGE substantially outperforms traditional methods, improving F1-scores by 5.3% through 22.8%. The framework achieves F1-scores ranging from 0.903 to 0.929 across various classifiers, exceeding the performance of state-of-the-art techniques. These results underscore the effectiveness and robustness of AFF_CGE in detecting encrypted malicious traffic, demonstrating its superior performance.

Time series is a typical data type in numerous domains; however, labeling large amounts of time series data can be costly and time-consuming. Learning effective representation from unlabeled time series data is a challenging task. Contrastive learning stands out as a promising method to acquire representations of unlabeled time series data. Therefore, we propose a self-supervised time-series representation learning framework via Time-Frequency Fusion Contrasting (TF-FC) to learn time-series representation from unlabeled data. Specifically, TF-FC combines time-domain augmentation with frequency-domain augmentation to generate the diverse samples. For time-domain augmentation, the raw time series data pass through the time-domain augmentation bank (such as jitter, scaling, permutation, and masking) and get time-domain augmentation data. For frequency-domain augmentation, first, the raw time series undergoes conversion into frequency domain data following Fast Fourier Transform (FFT) analysis. Then, the frequency data passes through the frequency-domain augmentation bank (such as low pass filter, remove frequency, add frequency, and phase shift) and gets frequency-domain augmentation data. The fusion method of time-domain augmentation data and frequency-domain augmentation data is kernel PCA, which is useful for extracting nonlinear features in high-dimensional spaces. By capturing both the time and frequency domains of the time series, the proposed approach is able to extract more informative features from the data, enhancing the model's capacity to distinguish between different time series. To verify the effectiveness of the TF-FC method, we conducted experiments on four time series domain datasets (i.e., SleepEEG, HAR, Gesture, and Epilepsy). Experimental results show that TF-FC significantly improves in recognition accuracy compared with other SOTA methods.

Portfolio optimization is one of the essential fields of focus in finance. There has been an increasing demand for novel computational methods in this area to compute portfolios with better returns and lower risks in recent years. We present a novel computational method called Representation Portfolio Selection by redefining the distance matrix of financial assets using Representation Learning and Clustering algorithms for portfolio selection to increase diversification. RPS proposes a heuristic for getting closer to the optimal subset of assets. Using empirical results in this paper, we demonstrate that widely used portfolio optimization algorithms, such as Mean-Variance Optimization, Critical Line Algorithm, and Hierarchical Risk Parity can benefit from our asset subset selection.

IntroductionSelf-supervised learning (SSL) is a strategy for addressing the paucity of labelled data in medical imaging by learning representations from unlabelled images. Contrastive and non-contrastive SSL methods produce learned representations that are similar for pairs of related images. Such pairs are commonly constructed by randomly distorting the same image twice. The videographic nature of ultrasound offers flexibility for defining the similarity relationship between pairs of images.MethodsWe investigated the effect of utilizing proximal, distinct images from the same B-mode ultrasound video as pairs for SSL. Additionally, we introduced a sample weighting scheme that increases the weight of closer image pairs and demonstrated how it can be integrated into SSL objectives.ResultsNamed Intra-Video Positive Pairs (IVPP), the method surpassed previous ultrasound-specific contrastive learning methods' average test accuracy on COVID-19 classification with the POCUS dataset by ≥ 1.3%. Detailed investigations of IVPP's hyperparameters revealed that some combinations of IVPP hyperparameters can lead to improved or worsened performance, depending on the downstream task.DiscussionGuidelines for practitioners were synthesized based on the results, such as the merit of IVPP with task-specific hyperparameters, and the improved performance of contrastive methods for ultrasound compared to non-contrastive counterparts.

Few-shot action recognition aims to train a model to classify actions in videos using only a few examples, known as “shots,” per action class. This learning approach is particularly useful but challenging due to the limited availability of labeled video data in practice. Although significant progress has been made in developing few-shot learners, existing methods still face several limitations. Firstly, current methods have not sufficiently explored the effectiveness of 3D feature extractors (e.g., 3D CNNs or Video Transformers), thereby failing to exploit spatiotemporal dynamics in videos. Secondly, the need for a large video dataset to train the model in a centralized manner raises privacy concerns and results in high storage costs and communication overheads. Thirdly, the existing solutions based on local deployment lack the capability to benefit global prior knowledge from a wide variety of real-world action samples. To address these limitations, we propose a federated learning (FL) framework named FedFSLAR++ to collaboratively train few-shot learners with 3D feature extractors. Specifically, we perform few-shot action recognition tasks under FL settings, enhancing privacy protection while maintaining efficient communication and storage. Moreover, FL allows us to effectively learn meta-knowledge from a large set of action videos among heterogeneous clients. Within our framework, we establish a unified benchmark to systematically and fairly compare different components, including feature extraction, meta-learning, and FL for model update and aggregation. This type of benchmark is still lacking in the literature. Notably, we thoroughly examine six 3D CNN and Transformer models for extracting spatiotemporal video features needed to adapt to new tasks quickly during the meta-learning process. We further propose a hybrid feature extractor that combines the advantages of 3D CNNs and Transformers to produce strong video representations. Additionally, we explore three meta-learning paradigms and three FL algorithms to investigate their effectiveness and suggest the optimal choices for performance improvement. Results from extensive experiments on four action datasets verify the robustness of the FedFSLAR++ framework. Our comprehensive study provides a solid foundation for future research advancements in action recognition.

Knowledge graph embedding models represent entities as vectors in continuous spaces and their relations by geometric transformations, mainly translation, and rotation. However, multi-relational knowledge graphs contain complex sub-graph structures, for which these two families of models fail to solely preserve. The complexities of these sub-graph structures often require embedding vector spaces with additional structures and composite transformations. A typical intrinsic additional algebraic structure on these embedding vector spaces has the potential to allow models to preserve complex relational patterns, such as non-commutativity, which are intrinsically related to the vector space. In this paper, we use linear fractional transformations ( $\mathrm {LFT_{}}$ ) in the noncommutative Clifford Algebra of order 2, $Cl(\mathbb {R}^{2,0})$ , to embed relations. $\mathrm {LFT_{}}$ can be understood as transformations that generalize Möbius transformations to the framework of Clifford algebras, in a non-commutative settings. Thus, the $\mathrm {LFT_{}}$ relation embeddings in our proposed models can exhibit geometric transformations namely translation, rotation, reflection, homothety, inversion, and their composites. By examining the preservation of complex structures using eleven benchmark knowledge graph datasets, our models showed results which are superior or competitive to existing state-of-the-art models relative to learning relational, many-to-many structural, and hierarchical patterns. This literature does not only advance the performance of knowledge graph embeddings but also emphasizes the importance of robust mathematical foundations and particularly the potential of Clifford algebras in embedding models.

Accurate time series forecasting is critical in a variety of fields, including transportation, weather prediction, energy management, infrastructure monitoring, and finance. Forecasting highly skewed and heavy-tailed time series, particularly in multivariate environments, is still difficult. In these cases, accurately capturing the relationships between variables is critical for successful model design. This is especially true when dealing with extreme events like droughts or floods in streamflow forecasting, which can have severe consequences on public safety and social well-being. We present the Multivariate Segment-Expandable Encoder Decoder (MSEED), a novel framework designed to address the challenges of extreme-adaptive multivariate time series forecasting. MSEED features a hierarchical encoder-decoder architecture, a short-term-enhanced subnet, and a feature assembling layer that integrates spatial and temporal information across multivariate inputs. By capturing quantile distributions across segmented subsequences at multiple scales, the model is able to detect complex patterns, enhancing both the accuracy and robustness of forecasts. Additionally, MSEED incorporates a simple vanilla encoder-decoder model for strengthening rolling predictions. The framework has been tested on four challenging real-world datasets, focusing on two critical forecasting scenarios: long-term predictions (three days ahead) and rolling predictions (every four hours) to simulate real-time decision-making in water resource management. MSEED consistently outperforms state-of-the-art models, showing improvements in forecasting accuracy ranging from 18% to 74%.

With the significant growth of scientific literature, Author Name Disambiguation (AND) has become increasingly challenging and urgent for applications including literature retrieval systems, talent management systems, and academic web mining. However, traditional feature modeling has shown limitations when dealing with large-scale publications, such as under-performance in extracting deep semantic feature information, poor generalization of the heuristic rules constructed by experts, and over-reliance on the assumption of similar discriminability of different relational features. Hence, this paper proposes an effective AND framework to address the above issues. An extended MiniLM is designed for semantic feature extraction, in which sample selection preferences and training strategies are optimized. Features of inter-paper relations based on IDF metrics are constructed. An ensemble learning approach is introduced to model the discriminability differences and inter-feature relations of the above features. Based on the relational network of papers, an unsupervised feature fusion algorithm that synthesizes multi-order connectivity information with a lightweight convolution operation is designed to mitigate differences among features of authors to be disambiguated. Empirical experiments show that our method achieves competitive performance on the real-world dataset and prove its effectiveness in large-scale publications and complex namesake scenarios.

Sindhi word segmentation is a challenging task due to space omission and insertion issues. The Sindhi language itself adds to this complexity. It’s cursive and consists of characters with inherent joining and non-joining properties, independent of word boundaries. Existing Sindhi word segmentation methods rely on designing and combining hand-crafted features. However, these methods have limitations, such as difficulty handling out-of-vocabulary words, limited robustness for other languages, and inefficiency with large amounts of noisy or raw text. Neural network-based models, in contrast, can automatically capture word boundary information without requiring prior knowledge. In this paper, we propose a Subword-Guided Neural Word Segmenter (SGNWS) that addresses word segmentation as a sequence labeling task. The SGNWS model incorporates subword representation learning through a bidirectional long short-term memory encoder, position-aware self-attention, and a conditional random field. Our empirical results demonstrate that the SGNWS model achieves state-of-the-art performance in Sindhi word segmentation on six datasets.

Recent advancements in convolutional neural networks have improved computer vision applications, including satellite imagery analysis. However, the lack of large labeled datasets and the complexity of remote sensing tasks render supervised learning methods less effective. While ImageNet pre-trained models have been used to address this, the domain difference between natural and satellite images poses significant limitations. These facts motivate us to explore both supervised and self-supervised learning to capture in-domain visual representations from satellite images, address the domain difference with ImageNet, and reduce the need for labeled datasets and computational resources. Furthermore, our research endeavors to identify the effective characteristics that make a dataset a suitable candidate for representation learning in the satellite imagery domain. The importance of choosing the right pre-training dataset cannot be overstated; it directly influences model performance and generalization capabilities. Given the plethora of available datasets in this field, selecting an appropriate one is fraught with difficulty, as each dataset varies in terms of quality, resolution, and relevance to specific tasks. The obtained weights from proper datasets serve as the initial weights for segmentation and object detection models. In terms of self-supervised pre-training, the SimSiam algorithm, employing the ResNet50 backbone, was utilized. Our results underscore that selecting a dataset with high spatial resolution is crucial, as it significantly enhances feature learning and improves model performance in remote sensing applications. This study explores the impact of hierarchical pre-training on dense prediction tasks, initially utilizing public datasets such as ImageNet, followed by in-domain datasets. Our systematic approach yielded significant improvements in performance metrics, enhancing the mean Intersection over Union (mIOU) score and pixel accuracy by 4.06% and 9.62%, respectively, when compared to existing literature. Furthermore, relative to our baseline model, the proposed methodology achieved enhancements of 2.1% in mIOU and 0.88% in pixel accuracy for semantic segmentation tasks on the DeepGlobe Land Cover Classification dataset, surpassing the efficacy of conventional ImageNet pre-trained weights. Additionally, we modified the DeepLabv3 architecture by re-implementing it to facilitate the transfer of previously trained weights to its backbone. To enhance convergence speed, we set the overall output stride to 32. In object detection tasks, including those involving Oil Tank Storage and Airplanes, the mean Average Precision (mAP) metric exhibited enhancements of 2.65% and 1.47%, respectively. This finding indicates that selecting an appropriate dataset for pre-training visual representations can significantly enhance the efficacy of remote sensing image analysis. Consequently, our proposed method emerges as a promising and practical approach for advancing this field.

Learning high-quality representations of users, items, and tags from historical interactive data is crucial for personalized tag recommendation (PTR) systems. Currently, most PTR models are committed to learning representations from first-order interactions without considering the exploitation of high-order interactive relations, which can be beneficial for avoiding sub-optimal learning. Although several PTR models equipped with graph neural networks (GNN) have been proposed to capture higher-order semantic relevance from raw data, they all carry out representation learning in Euclidean space, which can still easily result in sub-optimal learning due to embedding distortion. In order to further improve the quality of representation learning for PTR, the paper proposes a novel PTR model based on a lightweight GNN framework with hyperbolic embedding, namely GHPTR. GHPTR explicitly injects higher-order relevance into entity representation through the message propagation and aggregation mechanism of GNN and leverages hyperbolic embedding to alleviate the embedding distortion problem. Experimental results on real-world datasets have demonstrated the superiority of our model over its Euclidean counterparts and state-of-the-art baselines.

In this article, we propose a method exploiting irregular and unaligned Sentinel-2 satellite image time series (SITS) for large-scale land cover pixel-based classification. We perform end-to-end learning by combining a time and space informed kernel interpolator with a sparse variational Gaussian processes (SVGP) classifier. The interpolator embeds irregular and unaligned SITS onto a fixed and reduced size latent representation. The spatial information is taken into account by using a spatial positional encoding. The obtained latent representation is given to the SVGP classifier and all the parameters are jointly optimized w.r.t. the classification task. We run experiments with irregular and unaligned Sentinel-2 SITS of the full year 2018 over an area of 200 000 $\text{km}^{2}$ (about two billion pixels) in the south of France (27 MGRS tiles). Such experimental condition exacerbates the irregular and unaligned issues of SITS. In terms of overall accuracy, with the learned latent representation instead of linearly interpolated SITS, the results of the SVGP classifier are improved by about 10 points. Moreover, with the learned latent representation, the SVGP classifier outperforms the main state-of-the-art methods from the literature at large scale (e.g., seven points for the multilayer perceptron) and is robust to the available timestamps used for training and testing.

The growth of the tourism industry has greatly boosted the Point-of-Interest (POI) recom- mendation tasks using Location-based Social Networks (LBSNs). The ever-evolving nature of user preferences poses a major problem. To address this, we propose a Long-Term Preference Mining (LTPM) approach that utilizes the Temporal Recency (TR) measure in the visits along with the location-aware recommendation based on Spatial Proximity (SP) to the user’s location. The temporal dynamics and changing preferences are exploited based on the modified Long Short-term Memory (LSTM) that utilizes the time decay. The spatial considerations are modeled in two aspects: geographical proximity based on enhanced representation learning using orthogonal mapping. Second, the Region-of-Interest (ROI) is based on spatial griding and metric learning to capture the spatial relationships between POIs to enhance the metric space representation. The final recommendations are based on a multi-head attention mechanism that allocates the weights to different features. The combination of three models, called, LTPM-TRSP approach captures the user-POI, POI-POI, and POI-time relationships by focusing on the informative representation of sequential and spatial data. The category-aware final recommendations based on comprehensive historical behavior and geographical context are quite efficacious. The experimentation on three real-world datasets, Gowalla, Foursquare, and Weeplaces, also suggests the potency compared to other state-of-the-art approaches.

In this article, a new self-supervised strategy for learning meaningful representations of complex optical satellite image time series (SITS) is presented. The methodology proposed, named Unet-BERT spAtio-temporal Representation eNcoder (U-BARN), exploits irregularly sampled SITS. The designed architecture allows learning rich and discriminative features from unlabeled data, enhancing the synergy between the spatio-spectral and the temporal dimensions. To train on unlabeled data, a time-series reconstruction pretext task inspired by the BERT strategy but adapted to SITS is proposed. A Sentinel-2 large-scale unlabeled dataset is used to pretrain U-BARN. During the pretraining, U-BARN processes annual time series composed of a maximum of 100 dates. To demonstrate its feature learning capability, representations of SITS encoded by U-BARN are then fed into a shallow classifier to generate semantic segmentation maps. Experimental results are conducted on a labeled crop dataset (PASTIS) as well as a dense land cover dataset (MultiSenGE). Two ways of exploiting U-BARN pretraining are considered: either U-BARN weights are frozen or fine-tuned. The obtained results demonstrate that representations of SITS given by the frozen U-BARN are more efficient for land cover and crop classification than those of a supervised-trained linear layer. Then, we observe that fine-tuning boosts U-BARN performances on MultiSenGE dataset. In addition, we observe on PASTIS, in scenarios with scarce reference data that the fine-tuning brings a significative performance gain compared to fully supervised approaches. We also investigate the influence of the percentage of elements masked during pretraining on the quality of the SITS representation. Eventually, semantic segmentation performances show that the fully supervised U-BARN architecture reaches better performances than the spatio-temporal baseline (U-TAE) on both downstream tasks: crop and dense land cover segmentation.

Abstract Drug–drug interactions (DDI) are a critical concern in healthcare due to their potential to cause adverse effects and compromise patient safety. Supervised machine learning models for DDI prediction need to be optimized to learn abstract, transferable features, and generalize to larger chemical spaces, primarily due to the scarcity of high-quality labeled DDI data. Inspired by recent advances in computer vision, we present SMR–DDI, a self-supervised framework that leverages contrastive learning to embed drugs into a scaffold-based feature space. Molecular scaffolds represent the core structural motifs that drive pharmacological activities, making them valuable for learning informative representations. Specifically, we pre-trained SMR–DDI on a large-scale unlabeled molecular dataset. We generated augmented views for each molecule via SMILES enumeration and optimized the embedding process through contrastive loss minimization between views. This enables the model to capture relevant and robust molecular features while reducing noise. We then transfer the learned representations for the downstream prediction of DDI. Experiments show that the new feature space has comparable expressivity to state-of-the-art molecular representations and achieved competitive DDI prediction results while training on less data. Additional investigations also revealed that pre-training on more extensive and diverse unlabeled molecular datasets improved the model’s capability to embed molecules more effectively. Our results highlight contrastive learning as a promising approach for DDI prediction that can identify potentially hazardous drug combinations using only structural information.

Bug reports are crucial in software maintenance, with concise summaries significantly enhancing the efficiency of bug triagers and ultimately contributing to the development of high-quality software products. Contemporary methods for automatic bug report summarization primarily utilize neural networks’ robust learning capabilities. However, these approaches often produce suboptimal summaries due to two primary limitations: 1) the difficulty in assimilating the domain-specific knowledge inherent in bug reports, and 2) the limitations of purely supervised learning in comprehending the comprehensive context of bug reports. To address the above two problems, in this paper, we propose a new approach for bug report summarization, namely KSCLP, which leverages large language models and domain-specific pre-training strategies, i.e., Knowledge-Specific and Contrastive Learning Pre-training. Specifically, the Knowledge-Specific strategy allows to pre-train KSCLP on project-specific bug reports corpus, by which the model can fully learn internal knowledge of bug reports, learning bug report-aware representation. As for the Contrastive Learning strategy, it performs a sequence-level pre-training for KSCLP, helping it capture the semantic information of bug reports on a global level. Upon completion of the pre-training phase, KSCLP undergoes further refinement through a Sequence-to-Sequence framework specifically tailored for bug report summarization. The efficacy of KSCLP is rigorously evaluated against five baseline models using a publicly available dataset. The empirical results demonstrate that KSCLP outperforms all baselines, achieving remarkable improvements by up to 23.73, 13.97, and 20.89 points in ROUGE-1, ROUGE-2, and ROUGE-L metrics, thereby setting new benchmarks in the field of bug report summarization.

Core to much of modern deep learning is the notion of representation learning, learning representations of things that are useful for performing some task(s) related to those things. Encoder-only language models, for example, learn representations of language useful for performing language-related tasks, often classification. While fruitful in many applications, inherent is the assumption that only one classification is to be made for a particular input. This poses challenges when multiple classifications are to be made about different portions of a single record, such as emotion recognition in conversation (ERC) where the objective is to classify the emotion in each utterance of a dialog. Existing methods for this task typically either involve redundant computation, non-trivial post-processing outside of the core language model backbone, or both. To address this, we generalize recent work for deriving player-specific embeddings from multi-player sequences of events in sport for domain-agnostic application while also enabling it to leverage inter-entity relationships. Seeing the efficacy of the method in regression and classification tasks, we explore how it can be used to cluster player representations, proposing a novel approach for distribution-aware deep-clustering in the absence of labels. We demonstrate how the proposed methods yield state-of-the-art performance on the disparate tasks of ERC in Natural Language Processing (NLP), long-tail partial-label-learning (LT-PLL) in Computer Vision (CV), and player form clustering in sports analytics.

This study presents a comprehensive framework for analyzing Parkinson’s disease (PD) progression and optimizing medication recommendations based on multimodal data integration and reinforcement learning (RL) techniques. The research framework integrates diverse biomarkers from structured clinical assessments, imaging data, and biospecimen information, creating various single, bimodal, and trimodal datasets. Deep learning-based embedding networks are employed to transform high-dimensional data into lower-dimensional representations, capturing essential patterns related to PD progression. The learned representations are evaluated using visualization methods and predictive modeling to differentiate clinically relevant outcomes. Subsequently, medication recommendation models are developed based on the learned representations, assisting clinicians in making informed decisions to manage PD progression effectively. Optimal medication policies are derived considering different modality settings, revealing insights into medication prescribing tendencies across modalities. The findings highlight the significance of integrating diverse data modalities in understanding and managing PD progression, showcasing the potential of RL-based medication recommendation systems in clinical decision-making. This research contributes to the advancement of personalized medicine strategies for PD patients, emphasizing the importance of tailored medication regimes based on multimodal data insights.