Search

In recent years, speech denoising has greatly benefited from the rapid development of neural networks. However, these models require substantial noisy-clean speech pairs for supervised training, which limits their widespread use. Although there have been attempts to train denoising networks with only noisy speech data, existing self-supervised methods often suffer from a lack of continuity, low noise reduction performance, or heavy dependence on noise modeling. In this work, we introduce an efficient self-supervised Dynamic Multi-Focusing Network (DMFNet), a noise-only trained speech denoising network that utilizes a multi-scale connected encoder-decoder architecture as its backbone. Specifically, we have designed an efficient Spectral Dynamic Focusing Unit (SDFU) that enables the network to dynamically adapt the shape of its convolutional kernels while learning features, thus effectively focusing on the spectral structure of the human voice. Additionally, we introduce a Complex Attention Module (CAM), designed with a cross-space structure specialized for feature interaction and extraction. Finally, to further enhance the recovery of fine spectral details, we propose the Complex Multi-Scale Feature Fusion Unit (CMFFU) and Complex Scope Fusion Unit (CSFU) to adaptively fuse the features from different stages in the encoding process. Extensive evaluations across multiple datasets demonstrate that the proposed DMFNet significantly outperforms other state-of-the-art methods.

Image denoising is a critical task in image processing aimed at removing noise artifacts. Typically, supervised deep learning often necessitates a large number of pairs of noisy and noise-free images for training. Noise2Noise techniques have demonstrated efficiency in noise removal without relying on a noise-free ground truth. This is achieved through a learning process that approaches input to target points, balancing results across all training inputs. While Noise2Noise can be adapted for single image denoising, it still faces challenges in single image and blind noise scenarios. To address this issue, our research introduces the concept of self-augmented noisy images for self-supervised Noise2Noise single image denoising. The proposed method leverages the behavior of the training process, which strives to balance the loss values appropriately for each training set. By utilizing the same noisy image for both input and validation to learn self-identification, it produces another set of noisy images that mimic the input noisy images. From the experimental results, measured using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) metrics, it is evident that the proposed self-augmented strategy enables Noise2Noise to remove noise in single image scenarios. Additionally, it achieves performance comparable to other unsupervised denoising methods without requiring additional augmentation manipulations.

We introduce Pyramidal Recursive learning (PyRv), a novel method for text representation learning. This approach constructs a pyramidal hierarchy by recursively building representations of phrases, starting from tokens (characters, subwords, or words). At each level, N representations are recursively combined, resulting in N-1 representations on the level above, abstracting the input text from characters or subwords to words, phrases, and potentially sentences. The proposed method employs two learning approaches: autoencoding and autoregression. The autoencoding head decodes encoded representation pairs, while the autoregressive head predicts neighboring representations on both the left and right. This method exhibits four key properties: hierarchical representation, representation compositionality, representation decodability, and self-supervised learning. To implement and validate the proposed method, we train the Pyramidal Recursive Neural Network (PyRvNN) model. Evaluation metrics include autoencoder decodability, plagiarism detection, memorization, and readability. The accuracy of autoencoder decodability serves as an indicator of the validity of the four key properties. Preliminary assessments demonstrate promising results, particularly in machine-paraphrased plagiarism, text readability, and a memorization experiment.

How a robot induces causality, abstracts information from states, and plans for tasks are major challenges in artificial intelligence. However, habitually used deep reinforcement learning is not a cure-all solution. This study proposes MoSN, a momentum-based Siamese network for a robotic visual task. Instead of trying to learn representations that abstract all the variation factors of the environment as in most prior studies, MoSN labels its observations as their causes so that it can learn representations of visual states in a self-supervised manner. These representations are then used to enhance perceptual and cognitive abilities and to perform our robotic visual task when a robot takes over a new environment. In particular, this study proposes a momentum-based Siamese network with augmentation methods and a hybrid explicit contrastive loss to improve the combinatorial generalization capability of our model. We validate MoSN in real-world environments equipped with lights and light switches and attempt to derive the cause-and-effect relationship between switch states and visual states. MoSN is compared with previous methods in both static and dynamic environments with both fixed and dynamic viewing angles. MoSN outperforms previous methods in most cases.

The analysis of 3D motion information is the key to solve various computer vision tasks. Scene flow estimation tackles the problem of obtaining the 3D motion field. In this paper, we review the recent scene flow estimation papers with a focus on learning-based methods. The problem formulation, challenges and applications are introduced. The existing datasets and performance metrics are presented. The reason behind learning-based methods replacing the traditional variational methods are discussed. CNN-based scene flow estimation methods are then categorized with respect to the level of supervision, data-availability and the number of steps involved in obtaining the results. The performance of different methods on the well known KITTI and FlyingThings3D datasets are tabulated. Their relative advantages and limitations are then analysed. Future trends and some open problems in the estimation of scene flow are discussed with special focus on the self-supervised methods that does not require labelled training data.

Many speech features and models, including Deep Neural Networks (DNN), are used for classification tasks between healthy and pathological speech with the Saarbruecken Voice Database (SVD). However, accuracy values of 80.71% for phrases or 82.8% for vowels /aiu/ are the highest reported for audio samples in SVD when the evaluation includes the wide amount of pathologies in the database, instead of a selection of some pathologies. This paper targets this top performance in the state-of-the-art Automatic Voice Disorder Detection (AVDD) systems. In the framework of a DNN-based AVDD system we study the capability of Self-Supervised (SS) representation learning for describing discriminative cues between healthy and pathological speech. The system processes the SS temporal sequence of features with a single feed-forward layer and Class-Token (CT) Transformer for obtaining the classification between healthy and pathological speech. Furthermore, there is evaluated a suitable data extension of the training set with out-of-domain data is also evaluated to deal with the low availability of data for using DNN-based models in voice pathology detection. Experimental results using audio samples corresponding to phrases in the SVD dataset, including all pathologies available, show classification accuracy values until 93.36%. This means that the proposed AVDD system achieved accuracy improvements of 4.1% without the training data extension, and 15.62% after the training data extension compared to the baseline system. Beyond the novelty of using SS representations for AVDD, the fact of obtaining accuracies over 90% in these conditions and using the whole set of pathologies in the SVD is a milestone for voice disorder-related research. Furthermore, the study on the amount of in-domain data in the training set related to the system performance show guidance for the data preparation stage. Lessons learned in this work suggest guidelines for taking advantage of DNN, to boost the performance in developing automatic systems for diagnosis, treatment, and monitoring of voice pathologies.

In recent years, the development of deep learning has been pushing image denoising to a new level. Among them, self-supervised denoising is increasingly popular because it does not require any prior knowledge. Most of the existing self-supervised methods are based on convolutional neural networks (CNN), which are restricted by the locality of the receptive field and would cause color shifts or textures loss. In this paper, we propose a novel Denoise Transformer for real-world image denoising, which is mainly constructed with Context-aware Denoise Transformer (CADT) units and Secondary Noise Extractor (SNE) block. CADT is designed as a dual-branch structure, where the global branch uses a window-based Transformer encoder to extract the global information, while the local branch focuses on the extraction of local features with small receptive field. By incorporating CADT as basic components, we build a hierarchical network to directly learn the noise distribution information through residual learning and obtain the first stage denoised output. Then, we design SNE in low computation for secondary global noise extraction. Finally the blind spots are collected from the Denoise Transformer output and reconstructed, forming the final denoised image. Extensive experiments on the real-world SIDD benchmark achieve 50.62/0.990 for PSNR/SSIM, which is competitive with the current state-of-the-art method and only 0.17/0.001 lower. Visual comparisons on public sRGB, Raw-RGB and greyscale datasets prove that our proposed Denoise Transformer has a competitive performance, especially on blurred textures and low-light images, without using additional knowledge, e.g., noise level or noise type, regarding the underlying unknown noise.

Brain magnetic resonance images (MRI) convey vital information for making diagnostic decisions and are widely used to detect brain tumors. This research proposes a self-supervised pre-training method based on feature representation learning through contrastive loss applied to unlabeled data. Self-supervised learning aims to understand vital features using the raw input, which is helpful since labeled data is scarce and expensive. For the contrastive loss-based pre-training, data augmentation is applied to the dataset, and positive and negative instance pairs are fed into a deep learning model for feature learning. Subsequently, the features are passed through a neural network model to maximize similarity and contrastive learning of the instances. This pre-trained model serves as an encoder for supervised training and then the classification of MRI images. Our results show that self-supervised pre-training with contrastive loss performs better than random or ImageNet initialization. We also show that contrastive learning performs better when the diversity of images in the pre-training dataset is more. We have taken three differently sized ResNet models as the base models. Further, experiments were also conducted to study the effect of changing the augmentation types for generating positive and negative samples for self-supervised training.

The key to an accurate understanding of terrain is to extract the informative features from the multi-modal data obtained from different devices. Sensors, such as RGB cameras, depth sensors, vibration sensors, and microphones, are used as the multi-modal data. Many studies have explored ways to use them, especially in the robotics field. Some papers have successfully introduced single-modal or multi-modal methods. However, in practice, robots can be faced with extreme conditions; microphones do not work well in crowded scenes, and an RGB camera cannot capture terrains well in the dark. In this paper, we present a novel framework using the multi-modal variational autoencoder and the Gaussian mixture model clustering algorithm on image data and audio data for terrain type clustering by forcing the features to be closer together in the feature space. Our method enables the terrain type clustering even if one of the modalities (either image or audio) is missing at the test-time. We evaluated the clustering accuracy with a conventional multi-modal terrain type clustering method and we conducted ablation studies to show the effectiveness of our approach.

While deep learning strategies achieve outstanding results in computer vision tasks, one issue remains: The current strategies rely heavily on a huge amount of labeled data. In many real-world problems, it is not feasible to create such an amount of labeled training data. Therefore, it is common to incorporate unlabeled data into the training process to reach equal results with fewer labels. Due to a lot of concurrent research, it is difficult to keep track of recent developments. In this survey, we provide an overview of often used ideas and methods in image classification with fewer labels. We compare 34 methods in detail based on their performance and their commonly used ideas rather than a fine-grained taxonomy. In our analysis, we identify three major trends that lead to future research opportunities. 1. State-of-the-art methods are scalable to real-world applications in theory but issues like class imbalance, robustness, or fuzzy labels are not considered. 2. The degree of supervision which is needed to achieve comparable results to the usage of all labels is decreasing and therefore methods need to be extended to settings with a variable number of classes. 3. All methods share some common ideas but we identify clusters of methods that do not share many ideas. We show that combining ideas from different clusters can lead to better performance.

Diabetic retinal image classification aims to conduct diabetic retinopathy automatically diagnosing, which has achieved considerable improvement by deep learning models. However, these methods all rely on sufficient network training by large scale annotated data, which is very labor-expensive in medical image labeling. Aiming to overcome these drawbacks, this paper focuses on embedding self-supervised framework into unsupervised deep learning architecture. Specifically, we propose a Self-supervised Fuzzy Clustering Network (SFCN) by a feature learning module, reconstruction module, and a fuzzy self-supervision module. The feature learning and reconstruction modules ensure the representative ability of the network, and fuzzy self-supervision module is in charge of further providing the training direction for the whole network. Furthermore, three losses of reconstruction, self-supervision, and fuzzy supervision jointly optimize the SFCN under an unsupervised manner. To evaluate the effectiveness of the proposed method, we implement the network on three widely used retinal image datasets, which results demonstrate the satisfied performance on unsupervised retinal image classification task.

Artificial recognition of tomato diseases is often time-consuming, laborious and subjective. For tomato disease images, it is difficult to find small discriminative features between different tomato diseases, which can bring challenges to fine-grained visual categorization of tomato leaf-based images. Therefore, we propose a novel model, which consists of 3 networks, including a Location network, a Feedback network, and a Classification network, named LFC-Net. At the same time, a self-supervision mechanism is proposed in the model, which can effectively detect informative regions of tomato image without the need for manual annotation such as bounding boxes/parts. Based on the consideration of the consistency between category of the image and informativeness of the image, we design a novel training paradigm. The Location network of the model first detects informative regions in the tomato image, and optimizes iterations under the guidance of the Feedback network. Then, the Classification network uses informative regions proposed by the Location network and the full image of the tomato for classification. Our model can be regarded as a multi-network collaboration, and networks can progress together. Compared with the pre-trained model on ImageNet, our model achieves the most advanced performance in the tomato dataset, with accuracy up to 99.7%. This work demonstrates that our model has a high accuracy and has the potential to be applied to other vegetable and fruit datasets, which can provide a reference for the prevention and control of tomato diseases.