Search

This article presents a novel convolutional neural network (CNN) architecture for segmenting significantly small and crowded objects in remote sensing imagery. Although such small objects are characteristic in the remote sensing domain, the previous works mostly follow the state-of-the-art CNN models designed for ground-based images and have yet to fully explore the method for segmenting the small objects. To this end, we propose a network with no downsampling layers by utilizing dilated convolutions. We find that naive use of dilated convolutions with “increasing” dilation rates fails to capture local relationships among neighboring features, resulting in grid-like noise in the prediction. To alleviate this problem, we propose a novel scheme of “increasing-decreasing” dilation rates. Specifically, we propose a network module with decreasing dilation rates and attach it to the dilated backbone to reconnect the neighboring pixels of the backbone features. In the experiments, we evaluated the proposed model on six remote sensing datasets, where the model showed remarkably high performance, especially for small objects.

This article presents multibranch network architecture for addressing the problem of large intraclass variation in building detection task. Previous methods solved the problem by learning single structured and shared feature space with regularization. However, we reveal that the feature sharing strategy is less advantageous at deeper layers. We have analyzed the channel-wise contribution of the deep features for recognizing individual buildings and find that the feature space is separated into several clusters, among which the discriminative features are not shared much. Based on the analysis, we propose a multibranch neural network that solves the problem by decomposing a building class into subclasses and learning specialized feature space for each subclass. The proposed model is demonstrated on two remote sensing building detection benchmarks, where the model outperforms the state-of-the-art segmentation models and the previous techniques for addressing the large intraclass variation.

Atomic Force Microscopy (AFM) has been a broadly used platform for high-resolution imaging and mechanical characterization of a wide range of samples. However, this technique can be time-consuming and heavily relies on constant human supervision and human insight for data acquisition and analysis. Recent advancement in artificial intelligence (AI) provides the potential for efficient data analysis for AFM applications. The fusion of AFM with AI for effective image analysis and classification still remains an ongoing research endeavor. In this study, we present a novel AFM image defect detection and classification framework, AFM_YOLO-ResNet, using advanced deep learning (DL) techniques. Central to our approach is a highly integrated DL model that consists of a YOLO image defect detection layer and a ResNet feature extraction and classification layer. The proposed AFM_YOLO-ResNet framework is trained with expert annotated AFM images, and prepared to assess future AFM images from similar samples. Performance of AFM_YOLO-ResNet was validated for AFM image defect detection and classification, and compared with three commonly used transfer learning and computer vision models (Googlenet, Darknet, and YOLOv8). The results with high training and validation accuracies demonstrated that the AFM_YOLO-ResNet framework greatly improves the AFM imaging analysis efficiency.

This study presents a fully automated image informatics framework. The framework is combined with a deep learning (DL) approach to automatically predict visual acuity outcomes for people undergoing surgery for idiopathic full-thickness macular holes using 3D spectral-domain optical coherence tomography (SD-OCT) images. To overcome the impact of high variation in real-world image quality on the robustness of DL models, comprehensive imaging data pre-processing, quality assurance, and anomaly detection procedures were utilised. We then implemented, trained, and tested nine state-of-the-art DL predictive models through our designed loss function with multiple 2D input channels on the imaging dataset. Finally, we quantitatively compared the models using four evaluation metrics. Overall, the predictive model achieved a MAE of 6.47 ETDRS letters score, demonstrating high predictability. This confirms that our fully automated approach with input from seven central SD-OCT images from each patient can robustly predict visual acuity measurements. Further research will focus on adapting 3D DL-based predictive models and the uncertainty of 2D and 3D DL-based predictive models.

Automation improves the quality of fruits through quick and accurate detection of pest and disease infections, thus contributing to the country’s economic growth and productivity. Although humans can identify the fruit damage caused by pests and diseases, the methods used are inconsistent, time-consuming, and variable. The surface features of fruits typically observed by consumers who seek their health benefits affect their market value. The issue of pest and disease infections further deteriorates fruits’ quality, becoming a mounting stressor on farmers since they reduce the potential revenue from fruit production, processing, and export. This article reviews various studies on detecting and classifying damages in fruits. Specifically, we review articles where state-of-the-art approaches under segmentation, image processing, machine learning, and deep learning have proved effective in developing automated systems that address hurdles associated with manual methods of assessing damage using visual experiences. This survey reviews thirty-two journal and conference papers from the past thirteen years that were found electronically through Google Scholar, Scopus, IEEE, ScienceDirect, and standard online searches. This survey further presents a detailed discussion of previous research done in the past while emphasizing their strengths and limitations as well as outlining potential future research topics. It also reveals that much as the use of automated detection and classification of fruit damage has yielded promising results in the horticulture industry, more research is still needed with systems required to fully automate the detection and classification processes, especially those that are mobile phone-based towards addressing occlusion challenges.

This study explores the effectiveness of the ConvNeXt model, an advanced computer vision architecture, in the task of image captioning. We integrated ConvNeXt with a Long Short-Term Memory network that includes a visual attention module, focusing on assessing its performance across different scenarios. Experiments were conducted using various ConvNeXt versions for feature extraction, different learning rates during the training phase were tested, and the impact of including or excluding teacher-forcing was analyzed. The MS COCO 2014 dataset was employed, with top-5 accuracy and BLEU metrics used to evaluate performance. The implementation of ConvNeXt in image captioning systems reveals notable performance enhancements. In terms of BLEU-4 scores, ConvNeXt outperformed existing benchmarks by 43.04% for models using soft-attention and by 39.04% for those with hard-attention mechanisms. Furthermore, ConvNeXt surpassed models based on vision transformers and data-efficient image transformers by 4.57% and 0.93%, respectively, in BLEU-4 scores. When compared with systems using encoders such as ResNet-101, ResNet-152, VGG-16, ResNeXt-101, and MobileNet V3, ConvNeXt achieved higher top-5 accuracy improvements of 6.44%, 6.46%, 6.47%, 6.39%, and 6.68%, and reduced loss by 18.46%, 18.44%, 18.46%, 18.24%, and 18.72%, respectively.

iquaflow is a framework that provides a set of tools to assess image quality. The user can add custom metrics that can be easily integrated and a set of unsupervised methods is offered by default. Furthermore, iquaflow measures quality by using the performance of AI models trained on the images as a proxy. This also helps to easily make studies of performance degradation of several modifications of the original dataset, for instance, with images reconstructed after different levels of lossy compression; satellite images would be a use case example, since they are commonly compressed before downloading to the ground. In this situation, the optimization problem involves finding images that, while being compressed to their smallest possible file size, still maintain sufficient quality to meet the required performance of the deep learning algorithms. Thus, a study with iquaflow is suitable for such case. All this development is wrapped in Mlflow: an interactive tool used to visualize and summarize the results. This document describes different use cases and provides links to their respective repositories. To ease the creation of new studies, we include a cookiecutter repository. The source code, issue tracker and aforementioned repositories are all hosted on GitHub.

Computational histopathology is focused on the automatic analysis of rich phenotypic information contained in gigabyte whole slide images, aiming at providing cancer patients with more accurate diagnosis, prognosis, and treatment recommendations. Nowadays deep learning is the mainstream methodological choice in computational histopathology. Transformer, as the latest technological advance in deep learning, learns feature representations and global dependencies based on self-attention mechanisms, which is increasingly gaining prevalence in this field. This article presents a comprehensive review of state-of-the-art vision transformers that have been explored in histopathological image analysis for classification, segmentation, and survival risk regression applications. We first overview preliminary concepts and components built into vision transformers. Various recent applications including whole slide image classification, histological tissue component segmentation, and survival outcome prediction with tailored transformer architectures are then discussed. We finally discuss key challenges revolving around the use of vision transformers and envisioned future perspectives. We hope that this review could provide an elaborate guideline for readers to explore vision transformers in computational histopathology, such that more advanced techniques assisting in the precise diagnosis and treatment of cancer patients could be developed. [ABSTRACT FROM AUTHOR]

There are mainly two types of wafer map failure pattern recognition, i.e., traditional classification based and deep learning based approaches. Traditional classification usually needs feature engineering, and deep learning requires lower human intervention and feature engineering. The joint requirement is noise filtering and identical input dimension size. Feature engineering and dimension resizing are artificial work and vary from study to study. In our study, we proposed an image hashing based wafer map resizing and pattern information retrieval framework. Traditional classification methods and CNN are employed for the wafer map failure pattern recognition. The experiments show that the image hashing based framework is the more appropriate method when compared to feature extraction and image resizing methods. We also found that proper input data manipulation can result in acceptable performance whether the used method is traditional classification algorithms or CNN. [ABSTRACT FROM AUTHOR]

Quantum machine learning (QML) is an emerging technology that only recently has begun to take root in the research fields of earth observation (EO) and remote sensing (RS), and whose state-of-the-art (SOTA) is roughly divided into one group oriented to fully quantum solutions, and in another oriented to hybrid solutions. Very few works applied QML to EO tasks, and none of them explored a methodology that can give guidelines on the hyperparameter tuning of the quantum part for land cover classification (LCC). As a first step in the direction of quantum advantage for RS data classification, this letter opens new research lines, allowing us to demonstrate that there are more convenient solutions to simply increasing the number of qubits in the quantum part. To pave the first steps for researchers interested in the above, the structure of a new hybrid quantum neural network (QNN) for EO data and LCC is proposed with a strategy to choose the number of qubits to find the most efficient combination in terms of both system complexity and results accuracy. We sampled and tried a number of configurations, and using the suggested method, we came up with the most efficient solution (in terms of the selected metrics). Better performance is achieved with less model complexity when tested and compared with SOTA and standard techniques for identifying volcanic eruptions chosen as a case study. Additionally, the method makes the model more resilient to dataset imbalance, a significant problem when training classical models. Lastly, the code is freely available so that interested researchers can reproduce and extend the results. [ABSTRACT FROM AUTHOR]

Synthetic aperture radar (SAR) image understanding is crucial in remote sensing applications, but it is hindered by its intrinsic noise contamination, called speckle. Sophisticated statistical models, such as the $\mathcal {G}^{0}$ family of distributions, have been employed to SAR data and many of the current advancements in processing this imagery have been accomplished by extracting information from these models. In this letter, we propose improvements to parameter estimation in $\mathcal {G}^{0}$ distributions using the Method of Log-Cumulants (LCum). First, using Bayesian modeling, we construct the regularly produced reliable heterogeneity estimates under both $\mathcal {G}^{0}_{A}$ and $\mathcal {G}^{0}_{I}$ models. Second, we make use of an approximation of the Trigamma function to compute the estimated heterogeneity in constant time, making it considerably faster than the existing method for this task. Finally, we show how we can use this method to achieve fast and reliable SAR image understanding based on heterogeneity information. [ABSTRACT FROM AUTHOR]

Scene classification is a fundamental task in the remote sensing (RS) field, assigning semantic labels to RS images. Multispectral (MS) images play an essential role in scene classification as they contain richer spectral information than red, green, blue (RGB) images. However, MS images are not always available due to the higher cost and complexity of MS sensors compared to RGB sensors. To improve scene classification performance using only RGB images, in this letter, we propose a novel MS-to-RGB knowledge distillation (MS2RGB-KD) framework that transfers MS knowledge from a teacher model to a student model. Specifically, our MS2RGB-KD drives a student model that requires only an RGB image as input to mimic the feature representations of different modalities extracted by the teacher model. Moreover, we introduce novel loss functions that encourage the student model to preserve intramodal and intermodal relationships of the feature representations in the teacher model. Experiments on the EuroSAT dataset demonstrate the effectiveness of MS2RGB-KD compared with other KD baselines. [ABSTRACT FROM AUTHOR]

Comprehensive hardware assurance and failure analysis methods require exhaustive validation of the design layout through post-silicon imaging. Overlooked segmentation errors in such images cause significant inflation of resource requirements, both in terms of human resources and process time frame, for fault isolation and debugging in the hardware assurance process. Further, due to their lack of contextual awareness, the existing segmentation measures lack the ability to detect and suppress segmentation errors. In this paper, we introduce the first reference-less context-aware segmentation quality evaluation metric for scanning electron microscopy images, called Structural Equivalency and Connection Uniformity for Reverse Engineering (SECURE), to alleviate this issue by capturing an implicit understanding of design layout from weakly labeled unpaired images. The proposed metric can be seamlessly integrated into the design layout validation workflow for in-line rejection or correction of corrupted segmented images preventing the corrupted data from compromising the reliability of the hardware assurance process while simultaneously supporting process automation. Exhaustive qualitative and quantitative validation is provided to support the efficacy of the metric.

The foundation model (FM) has garnered significant attention for its remarkable transfer performance in downstream tasks. Typically, it undergoes task-agnostic pretraining on a large dataset and can be efficiently adapted to various downstream applications through fine-tuning. While FMs have been extensively explored in language and other domains, their potential in remote sensing has also begun to attract scholarly interest. However, comprehensive investigations and performance comparisons of these models on remote sensing tasks are currently lacking. In this survey, we provide essential background knowledge by introducing key technologies and recent developments in FMs. Subsequently, we explore essential downstream applications in remote sensing, covering classification, localization, and understanding. Our analysis encompasses over 30 FMs in both natural and remote sensing fields, and we conduct extensive experiments on more than 10 datasets, evaluating global feature representation, local feature representation, and target localization. Through quantitative assessments, we highlight the distinctions among various FMs and confirm that pretrained large-scale natural FMs can also deliver outstanding performance in remote sensing tasks. After that, we systematically presented a brain-inspired framework for remote sensing foundation models (RSFMs). We delve into the brain-inspired characteristics in this framework, including structure, perception, learning, and cognition. To conclude, we summarize 12 open problems in RSFMs, providing potential research directions. Our survey offers valuable insights into the burgeoning field of RSFMs and aims to foster further advancements in this exciting area.

Automatic recognition of illegal substations is of great importance, since most of the leakage electricity in Turkey is due to the use of these substations in agricultural fields. One of the most effective ways to detect illegal substations is to employ remote sensing images and machine learning techniques together. Because, thanks to remote sensing images, it is possible to analyze illegal substations on huge agricultural lands in a short time. In this study, illegal substations on the agricultural fields in the southeast Anatolian region, which is one of the regions where leakage electricity are most common, have been detected with the aid of Landsat-8 OLI images and machine learning algorithm. The proposed study has been carried out in several stages, respectively. In the first stage, the locations of 42 substations and 21 non-substation objects on the pilot area have been recorded with the help of GPS and these coordinates have been later transferred to the Landsat-8 OLI image dated on 14 June 2019. In the second stage, an image analysis has been performed by calculating the spectral band parameters from the Landsat-8 OLI images. In the next stage, relationships among illegal substations and non-substation objects have been set by utilizing the statistical metrics of obtained spectral band parameters. In the last stage, by utilizing LSTM (Long Short-Term Memory) method, which is a recurrent neural network model that has gained popularity in both remote sensing and various scientific disciplines in recent years and the Logit-Boost method, which is one of the popular boosting machine learning algorithms, automatic recognition of substations has been performed with an average accuracy of 88.89% for Logit-Boost method and 84.21% for LSTM method. It is notable from this study that the Logit-Boost Algorithm yields more proficient results than the LSTM model.

Linking a digital image or video to its originating device, or checking the content integrity still represent challenging forensic tasks. Even though several technologies based on metadata, file format, and sensor noise have been developed to address these problems, current methods are frequently made obsolete by new customized acquisition pipelines implemented by manufacturers. Therefore, to assess the performance of the available tools and push the research activity, researchers continuously need new datasets containing contents captured with recent technologies. In this paper, we present a new image and video dataset for forensic analysis. Data, acquired by the most recent acquisition devices, were collected under strictly controlled procedures designed to limit the bias induced by differences in the acquisition process between different devices. The dataset includes over 9000 media contents captured by 46 smartphones of 11 major brands. For each device, we collected at least 100 unique natural images, 30 unique natural videos, 30 flat images, and 4 flat videos. Great care has been taken in collecting data that can be used for multiple forensic tasks; moreover, images and videos have been carefully organized so that FloreView could be used by the community immediately and effortlessly. Finally, two case studies related to image source identification and video brand identification have been performed, using state-of-the-art methods, to show how the proposed dataset can be effectively used for forensic tasks.

Blockchain (BC) and Computer Vision (CV) are the two emerging fields with the potential to transform various sectors. BC can offer decentralized and secure data storage, while CV allows machines to learn and understand visual data. The integration of the two technologies holds massive promise for developing innovative applications that can provide solutions to the challenges in various sectors such as supply chain management, healthcare, smart cities, and defense. This review explores a comprehensive analysis of the integration of BC and CV by examining their combination and potential applications. It also provides a detailed analysis of the fundamental concepts of both technologies, highlighting their strengths and limitations. This paper also explores current research efforts that make use of the benefits offered by this combination. The BC can be used as an added layer of security in CV systems and also ensure data integrity, enabling decentralized image and video analytics. The challenges and open issues associated with this integration are also identified, and appropriate potential future directions are also proposed.

Most of the world’s oil reserves and natural gas are stored within carbonate rock’s pores and fractures. Pores and fractures are quite popular for predicting the amount of petroleum under an adequate trap condition. Hence, their petrophysical properties, such as shape and size, are paramount for accurately predicting the reservoir’s state and condition. Current modelling techniques are mostly based on manual and expert judgement which are time-consuming and cost-intensive. In this study, we devised a robust and scalable image processing framework that uses the combination of pixel-based clustering approach with a polar coordinate feature transformation technique to intelligently identify the pores of carbonate rock samples. We reported that such a method can be effective in detecting pores of different shapes and sizes in an automated fashion. We rigorously tested the proposed method on the computed tomography-scanned micro-images of a carbonate rock sample, and the results demonstrate improved identification accuracy of the proposed method than the current deep learning counterparts. Another key advantage compared to deep learning methods, the proposed method does not require extensive training on data, which saves time and effort without being computationally too expensive.

In geosciences, micropaleontology studies the evolution of microfossils (e.g., foraminifera) throughout geological records and utilizes such information to reconstruct past environmental and climatic conditions. This field depends primarily on the visual recognition of various features in microfossils, which makes it ideal for applying computer vision technology, specifically deep convolutional neural networks (CNNs), to automate and optimize different microfossil identification and classification. In addition, the unlabeled, low-resolution micropaleontological dataset is often available in a large volume compared to another geosciences dataset. While the application of deep learning in micropaleontology is rapidly growing, these efforts have been severely hampered by (i) the limited availability of high-quality and high-resolution labeled fossil images and (ii) significant effort in manually labeling various fossils by subject matter experts. Furthermore, previous works primarily exploited CNN with transfer learning to obtain high-accuracy prediction, which may reduce the explainability and reproducibility of the model. To overcome this issue, we propose a novel deep learning workflow that couples hierarchical vision transformers with style-based generative adversarial network algorithms to efficiently acquire and synthetically generate realistic high-resolution labeled datasets of micropaleontology in a large volume. Our study demonstrates that the proposed workflow could generate high-resolution images with a high signal-to-noise ratio, achieving 39.1 dB, and realistic synthetic images with a Fréchet inception distance similarity score of 14.88. In addition, our proposed workflow could provide a considerable volume of self-labeled datasets that can be used for model benchmarking and various downstream visual tasks, including fossil classification and segmentation. We further performed, for the first time, a few-shot semantic segmentation of different foraminifera chambers on both the generated and synthetic images with high accuracy. This novel meta-learning approach is only possible when a high-resolution and high-volume labeled dataset is available. Therefore, our proposed deep learning-based workflow is promising and shows a potential to advance and optimize micropaleontological research and other visual-dependent geological analysis.

In the paper, a novel hybrid method for an automatic quality assessment of binary images is proposed that may be useful, e.g., for computationally limited embedded systems or Optical Character Recognition applications. Since the quality of binary images used as the input for further image analysis strongly influences the obtained results, a reliable evaluation of their quality is a crucial element for the validation of such systems. Assuming the availability of several video frames, an objective quality assessment of individual video frames may also be helpful for the choice of a binarized image leading to the “best” final results. Nevertheless, most of the image quality assessment methods concern the analysis of grayscale or color images, utilizing the available image datasets containing numerous images subject to various distortions together with the corresponding subjective quality scores. Therefore, a reliable quality assessment of binary images is troublesome due to a small number of datasets and methods dedicated to binary images. The approach presented in the paper is based on the combination of one of such metrics, known as Border Distance, with some other methods, utilizing the nonlinear model based on the weighted sum and product of individual metrics. The experimental verification of the proposed hybrid metric conducted for the publicly available Bilevel Image Similarity Ground Truth Archive dataset leads to a significantly higher correlation with subjective quality scores in comparison to the other methods.

Ear biting is a welfare challenge in commercial pig farming. Pigs sustain injuries at the bite site paving the way for bacterial infections. Early detection and management of this behavior are important to enhance animal health and welfare, increase productivity, and minimize inputs from medication. Pig management using physical observation is impractical because of the scale of modern pig production systems. The same applies to the manual analysis of videos captured from pigsty. Therefore, a method of automated detection is desirable. In this study, we introduce an automatic detection pipeline based on deep learning for the quantification of ear biting outbreaks. Two state-of-the-art detection networks, YOLOv4 and YOLOv7, were trained to localize the regions of ear biting. The detected regions were tracked over multiple video frames using DeepSORT and Centroid tracking algorithms. Tracking provided the association between detected instances in video frames, enabling the computation of the frequency and duration of occurrence. The frequency and duration of ear biting were expressed as the cumulative performance of each group of pigs. The pipeline was evaluated using two datasets from experimental and commercial farms with diverse management and monitoring settings. The detection networks achieved comparable average precision values of 98% & 97.5% and 85.6% & 80.9% on the respective datasets. The tracking algorithms produced 14% and 34% False-Alarm rates, respectively. The results show that automated detection and tracking of ear biting is possible. Subsequently, we applied our method to videos in which pigs were managed in a manner that was expected to affect the frequency of ear biting to different degrees. This method can be used as the basis of an early warning system for the detection of ear-biting in commercial farms.

In this paper, we propose a novel dual gradient-based desnowing algorithm that can accurately remove snow from a scene by characterizing snow particles. To localize snow in an image, we present a gradient-based snow activation map that can be estimated using snow classification. To recognize various patterns in the shapes and trajectories of snow particles, we introduce a gradient-based snow edge map. Using these two gradients, we estimate an accurate snow attention mask that is subsequently used for snow removal. In addition, we propose a translucency-aware context restoration network to handle various degrees of snow transparency and thus, prevent our method from losing the image context information during desnowing. Experimental results demonstrate that the proposed method considerably outperforms other state-of-the-art desnowing algorithms quantitatively and qualitatively.

The automatic and accurate analysis of medical images (e.g., segmentation,detection, classification) are prerequisites for modern disease diagnosis and prognosis. Computer-aided diagnosis (CAD) systems empower accurate and effective detection of various diseases and timely treatment decisions. The past decade witnessed a spur in deep learning (DL)-based CADs showing outstanding performance across many health care applications. Medical imaging is hindered by multiple sources of uncertainty ranging fromnteasurement (aleatoric) errors, physiological variability, and limited medical knowledge (epistemic errors). However, uncertainty quantification (UQ) in most existing DL methods is insufficiently investigated, particularly in medical image analysis. Therefore, to address this gap, in this article, we propose a simple yet novel hierarchical attentive multilevel feature fusion model with an uncertainty-aware module for medical image classification coined Hercules. This approach is tested on several real medical image classification challenges. The proposed Hercules model consists of two main feature fusion blocks, where the former concentrates on attention-based fusion with uncertainty quantification module and the latter uses the raw features. Hercules was evaluated across three medical imaging datasets, i.e., retinal OCT, lung CT, and chest X-ray. Hercules produced the best classification accuracy in retinal OCT (94.21%), lung CT (99.59%), and chest X-ray (96.50%) datasets, respectively, against other state-of-the-art medical image classification methods. [ABSTRACT FROM AUTHOR]

Remote sensing scene classification is an important yet challenging task. In recent years, the excellent feature representation ability of convolutional neural networks (CNNs) has led to substantial improvements in scene classification accuracy. However, handling resolution variations of remote sensing images is still challenging because CNNs are not inherently capable of modeling multiresolution input images. In this letter, we propose a novel scene classification method with scale and resolution adaptation ability by leveraging the recent advances in implicit neural representations (INRs). Unlike previous CNN-based methods that make predictions based on rasterized image inputs, the proposed method converts the images as continuous functions with INRs optimization and then performs classification within the function space. When the image is represented as a function, the image resolution can be decoupled from the pixel values so that the resolution does not have much impact on the classification performance. Our method also shows great potential for multiresolution remote sensing scene classification. Using only a simple multilayer perceptron (MLP) classifier in the proposed function space, our method achieves classification accuracy comparable to deep CNNs but exhibits better adaptability to image scale and resolution changes. [ABSTRACT FROM AUTHOR]

This paper aims at proposing an unsupervised hierarchical nonparametric Bayesian framework for modeling axial data (i.e., observations are axes of direction) that can be partitioned into multiple groups, where each observation within a group is sampled from a mixture of Watson distributions with an infinite number of components that are allowed to be shared across different groups. First, we propose a hierarchical nonparametric Bayesian model for modeling grouped axial data based on the hierarchical Pitman-Yor process mixture model of Watson distributions. Then, we demonstrate that by setting the discount parameters of the proposed model to 0, another hierarchical nonparametric Bayesian model based on hierarchical Dirichlet process can be derived for modeling axial data. To learn the proposed models, we systematically develop a closed-form optimization algorithm based on the collapsed variational Bayes (CVB) inference. Furthermore, to ensure the convergence of the proposed learning algorithm, an annealing mechanism is introduced to the framework of CVB inference, leading to an averaged collapsed variational Bayes inference strategy. The merits of the proposed models for modeling grouped axial data are demonstrated through experiments on both synthetic data and real-world applications involving gene expression data clustering and depth image analysis. [ABSTRACT FROM AUTHOR]

The general tensor-based methods can recover missing values of multidimensional images by exploiting the low-rankness on the pixel level. However, especially when considerable pixels of an image are missing, the low-rankness is not reliable on the pixel level, resulting in some details losing in their results, which hinders the performance of subsequent image applications (e.g., image recognition and segmentation). In this article, we suggest a novel multiscale feature (MSF) tensorization by exploiting the MSFs of multidimensional images, which not only helps to recover the missing values on a higher level, that is, the feature level but also benefits subsequent image applications. By exploiting the low-rankness of the resulting MSF tensor constructed by the new tensorization, we propose the convex and nonconvex MSF tensor train rank minimization (MSF-TT) to conjointly recover the MSF tensor and the corresponding original tensor in a unified framework. We develop the alternating directional method of multipliers (ADMMs) to solve the convex MSF-TT and the proximal alternating minimization (PAM) to solve the nonconvex MSF-TT. Moreover, we establish the theoretical guarantee of convergence for the PAM algorithm. Numerical examples of real-world multidimensional images show that the proposed MSF-TT outperforms other compared approaches in image recovery and the recovered MSF tensor can benefit the subsequent image recognition. [ABSTRACT FROM AUTHOR]

Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas – fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community. [ABSTRACT FROM AUTHOR]

Self-supervised representation learning has been extremely successful in medical image analysis, as it requires no human annotations to provide transferable representations for downstream tasks. Recent self-supervised learning methods are dominated by noise-contrastive estimation (NCE, also known as contrastive learning), which aims to learn invariant visual representations by contrasting one homogeneous image pair with a large number of heterogeneous image pairs in each training step. Nonetheless, NCE-based approaches still suffer from one major problem that is one homogeneous pair is not enough to extract robust and invariant semantic information. Inspired by the archetypical triplet loss, we propose GraVIS, which is specifically optimized for learning self-supervised features from dermatology images, to group homogeneous dermatology images while separating heterogeneous ones. In addition, a hardness-aware attention is introduced and incorporated to address the importance of homogeneous image views with similar appearance instead of those dissimilar homogeneous ones. GraVIS significantly outperforms its transfer learning and self-supervised learning counterparts in both lesion segmentation and disease classification tasks, sometimes by 5 percents under extremely limited supervision. More importantly, when equipped with the pre-trained weights provided by GraVIS, a single model could achieve better results than winners that heavily rely on ensemble strategies in the well-known ISIC 2017 challenge. Code is available at https://bit.ly/3xiFyjx. [ABSTRACT FROM AUTHOR]

Industrial ultrasonic tomography (UT) possesses unique benefits in nonintrusive detection of cross-sectional medium and has received broad attention in decades. Aiming at the complex bubble imaging, the conventional independent-transducer-based UT has low spatial resolution due to wide projection path and low projection number. Based on ultrasound phase array (UPA) transducers, this article proposes a sequential dynamic aperture focusing strategy for transmissive ultrasonic phase array tomography (UPAT). The emitting aperture focuses the ultrasound beam on a close distance to generate a high intensity and wide-angle acoustic field that covers multiple receivers in single excitation and further induces high amount of multiangle narrow projections, through which the measurement speed and imaging performance are substantially improved. Beyond numerical analysis on image accuracy and noise robustness, an experimental UPAT system with independent T/R channels is developed to evaluate the proposed method, where the improvement on imaging accuracy and speed are testified through qualitatively and quantitatively comparison with the state-of-art UT and the beam scanning UPAT techniques. The UPAT-focus strategy is proved to be advanced and feasible in imaging complex medium distribution. [ABSTRACT FROM AUTHOR]

Due to low cost and low complexity, passive ultrahigh-frequency radio-frequency identification (UHF RFID) is gaining popularity in indoor localization. According to current RFID channel models, multipath propagation and noise are two significant factors that affect the accuracy of measured RF signals. To minimize the impacts of measurement errors on localization, scene analysis algorithms are proposed and got lots of attention in recent years. These algorithms make use of reference tags as similarity markers in neighboring locations to improve localization accuracy. However, densely deployed tags induce a new challenge, antenna interference, leading to the poor signal similarity between nearby tags. Nonetheless, there are no refined channel models for explaining the effect of spacing distances between tags on RF signal fluctuations, thus the choice of spacing distances becomes an outstanding issue for scene analysis localization algorithms. In this article, we propose Timing, the Tag Interference ModelING, for analyzing the relationship between signal fluctuations and spacing distances. Unlike inductive coupling in the near field, as the tags are all located at the far field of interacted antenna, we introduce microwave transmission line theory for channel model analysis. The experimental results verify that Timing is well suited to authentic situations. Additionally, we examine the performance of the ${k}$ -nearest neighbor (${k}$ -NN) algorithm under various spacing distances. The findings demonstrate that the spacing distance identified by Timing as a stable point achieves better accuracy with scene analysis algorithms. Therefore, Timing could be used as a critical foundation for the choice of spacing distances in future tag deployment applications. [ABSTRACT FROM AUTHOR]

Root-knot nematodes (Meloidogyne spp.; RKN) are major plant-parasitic nematodes that cause significant loss to agricultural production. An accurate assessment of the RKN population density at a field level is crucial for decisions about the application of control measures to minimise yield losses. Traditionally, RKN populations are identified and counted by nematologists using a microscope. This method is a specialised, time-consuming process and prone to errors. In our study, we investigated three semi-automated methods to detect and count RKN eggs using image analysis: contour arc (CA), skeleton structure (SS), and extreme point (EP). These methods were used to automate the length measurement of RKN eggs, and the results were compared with traditional methods of quantification. The EP method produced the highest correlation with the manual length measurement of RKN eggs. Further, these methods were used to detect and count RKN eggs to quantify low and highly cluttered images. We estimated the optimal range of the ratio of each method to detect and count RKN eggs. Overall, the EP method computed using mid-width of RKN eggs revealed better detection and counting of RKN eggs as compared to the SS and CA methods. A counting correlation up to $\text{R}^{2} =0.905$ was obtained. This study found that the difference between mid-width and the average width of RKN eggs and soil particles could be used to discriminate 70-80 % of soil particles. Our research thus contributes a new feature that can be used to discriminate or classify objects in object detection techniques.

Ripening is a very important process that contributes to cheese quality, as its characteristics are determined by the biochemical changes that occur during this period. Therefore, monitoring ripening time is a fundamental task to market a quality product in a timely manner. However, it is difficult to accurately determine the degree of cheese ripeness. Although some scientific methods have also been proposed in the literature, the conventional methods adopted in dairy industries are typically based on visual and weight control. This study proposes a novel approach aimed at automatically monitoring the cheese ripening based on the analysis of cheese images acquired by a photo camera. Both computer vision and machine learning techniques have been used to deal with this task. The study is based on a dataset of 195 images (specifically collected from an Italian dairy industry), which represent Pecorino cheese forms at four degrees of ripeness. All stages but the one labeled as “day 18”, which has 45 images, consist of 50 images. These images have been handled with image processing techniques and then classified according to the degree of ripening, i.e., 18, 22, 24, and 30 days. A 5-fold cross-validation strategy was used to empirically evaluate the performance of the models. During this phase, each training fold was augmented online. This strategy allowed to use 624 images for training, leaving 39 original images per fold for testing. Experimental results have demonstrated the validity of the approach, showing good performance for most of the trained models.

Generalized image understanding of natural outdoor images is fundamentally a bottom-up approach guided by the data and information received from the digital input images. The importance of intermediate image representation that aids generalized image understanding, inspired the proposed a unified approach for providing a new meaningful and objective representation of a given outdoor scene. The proposed unified framework provides a novel set of string and semantic codes, based on inferential statistical data obtained from segments characterization and blocks characterization of natural outdoor images. The framework acts as an analytical tool that can categorize the input images into four types based on the inferential statistical representation - 1) Object-based image, 2) Heterogeneous color based texture image, 3) Homogeneous color based texture image, 4) Pure toned image. The proposed string and semantic code characterization is also rotation and scale variant. Such a representation is simple, unsupervised, computationally inexpensive and also places minimal demand on the storage. Later for a higher-level image analysis and understanding the proposed unified approach can provide focused interaction between the above intermediate representation and domain scene knowledge for object detection, classification, or recognition.

Objective: Respiratory disease is a rapidly growing global health issue that impacts the quality of living of tens of millions of people around the world. Neutrophil elastase (NE) represents a key inflammatory biomarker and has previously been demonstrated to have the capability of predicting exacerbation risk related to respiratory diseases. This paper utilises a low-cost Point of Care (PoC) approach using Lateral Flow Assays (LFAs) to provide quantitative measurement of active NE in a patient’s sputum. Methods and procedures: The main aim of this study is to develop a quantitative platform using a Complementary Metal-Oxide-Semiconductor (CMOS) to image the LFAs and with an adaptable image analysis algorithm to measure a target biomarker concentration. This result could be used to monitor a patient’s health and quality of living. In the paper, NE is used as the target biomarker to determine if the patient is suffering from a high risk of exacerbations. Results: The results presented in the paper indicate the CMOS reader approach is promising for rapid and low-cost PoC devices, with the current system able to provide quantitative trends of NE concentrations as low as 100 ng/ml and is comparable to a research-based laboratory lateral flow reader. Conclusion: The image analysis algorithm used in the CMOS reader can estimate the minimum NE concentration of 250 ng/ml to indicate the high-risk category for exacerbations from respiratory illnesses with the same accuracy as expensive a research-based laboratory reader but by using low-cost components and onboard image analysis. Clinical impact: The image analysis algorithm is evaluated to analyse LFAs with NE biomarker to determine the patient in a high-risk category for exacerbations. The device communicates the analysis result to medical professionals for daily historical logging for daily health monitoring without regular hospital appointments. The low-cost approach of the proposed system and image analysis approach can be adapted to analyse different biomarkers for other health concerns including multiplex LFAs without additional hardware in the reader design.

In this paper, we propose a framework for evaluation of co-salient object detection algorithms along with two novel CoSOD methods. The processing pipeline of this framework is based on the CoEGNet algorithm, where the saliency detection part can be easily replaced by any saliency detector to be evaluated. By leveraging the proposed framework, we developed two new algorithms: one based on U2Net and the other based on the label decoupling framework (LDF). They are called in this paper CoU2Net and CoLDF, respectively. The proposed solutions were tested on three datasets: CoCA, CoSal2015, and CoSOD3k, and compared with some of the best algorithms in co-salient object detection: GICD and CoEGNet. The advantages and disadvantages of the proposed methods are highlighted and discussed. As a generalization of the aforementioned methods, we also propose a framework called Sal.Co. It is a modification of the CoEGNet method and it works on a saliency mask obtained from a saliency detector and attempts to indicate salient objects coexisting in a group of images. Both CoLDF and CoU2Net achieved better results than CoEGNet on the CoCA dataset. On the CoSal2015 and CoSOD3k datasets, they performed similarly to state-of-the-art methods, while maintaining a highly customizable structure. The source code can be found at: https://github.com/jakubkorczakowski/cosal_sal_testing.

Traditional target detection algorithms have difficulty to adapt complex environmental changes and have limited applicable scenarios. However, the deep-learning-based target detection model can automatically learn with strong generalization capability. In this article, we choose a single-stage deep-learning-based target detection model for research based on the model’s real-time processing requirements and to improve the accuracy and the robustness of target detection in remote sensing images. In addition, we improve the YOLOv4 network and present a new approach. First, we propose a classification setting of the nonmaximum suppression threshold to increase the accuracy without affecting the speed. Second, we study the anchor frame allocation problem in YOLOv4 and propose two allocation schemes. The proposed anchor frame scheme also improves the detection performance, and experimental results on the DOTA dataset validate their effectiveness.

Detection of humans accurately in aerial images is critical for various applications such as surveillance, detecting and tracking athletes on sports fields, and search and rescue operations (SAR). The goal of SAR is to assist, detect, and rescue people who have had accidents in the mountains or other hazardous environments. By using drones in SAR applications, it is desirable to minimize the cost and time spent on SAR operations. In this paper, we present a convolutional neural network-based model for the detection of humans in aerial images of mountain landscapes acquired by unmanned aerial vehicles (UAVs) used in search and rescue operations. Detection of humans in aerial images remains a complex task due to various challenges such as pose and scale variations of humans, low visibility, camouflaged environment, adverse weather conditions, motion blur, and high-resolution aerial images. Due to imaging from high altitudes, in most high-resolution aerial images captured by UAVs, only 0.1 to 0.2 percentage of the image represents humans. To solve the problem of low coverage of the object of interest in high-resolution aerial images, we propose to implement a deep learning-based object detection model. In this paper, we propose a novel method for the detection of humans in aerial images based on the EfficientDET architecture and ensemble learning. The method has been validated on the HERIDAL image dataset. By implementing the proposed methodologies, we achieved an mAP of 95.11%. To the best of our knowledge, this is the highest accuracy result for human detection on the HERIDAL dataset.

We present an automated wound localizer from 2D wound and ulcer images using a deep neural network as the first step towards building an automatic and complete wound diagnostic system. The wound localizer is developed using the YOLOv3 model, which is then turned into an iOS mobile application. The developed localizer can detect the wound and its surrounding tissues and isolate the localized wounded region from images, which would help future processing such as wound segmentation and classification due to the removal of unnecessary regions from wound images. For Mobile App development with video processing, a lighter version of YOLOv3 named tiny-YOLOv3 has been used. The model is trained and tested on our image dataset, collaborating with AZH Wound and Vascular Center, Milwaukee, Wisconsin. The YOLOv3 model is compared with the SSD model in our developed dataset (AZH wound database, available at https://github.com/uwm-bigdata/wound_localization), showing that YOLOv3 gives an mAP value of 93.9%, which is much better than the SSD model (86.4%). The robustness and reliability of these models are also tested on a publicly available dataset named Medetec, which shows very good performance. Finally, a bigger dataset (BMAZHM Wound Database) is created, and the developed model is retrained with the new dataset to achieve a very high mAP of 97.3%.

Automatic longitudinal assessment of the disease progression of coronavirus disease 2019 (COVID-19) is invaluable to ensure timely treatment for severe or critical patients. An artificial intelligence system that combines chest computed tomography (CT) and laboratory examinations may provide a more accurate diagnosis. To explore an artificial intelligence solution to longitudinally assess the condition of COVID-19 using CT imaging and laboratory findings, from January 27, 2020, to April 3, 2020, multiple follow-up examinations of COVID-19 inpatients were retrospectively collected. CT imaging features were automatically extracted using a deep learning method and combined with laboratory tests. The progression sequences were generated with two follow-ups, each of which contained 60 imaging and 24 laboratory features. Pearson’s correlation was conducted to rank the importance of each univariate feature, and multivariate logistic regression was adopted for feature selection. The selected features were used to train a 2-layer long short-term memory network (LSTM) with pulse oxygen saturation (SpO2) as an indicator of disease progression in three classes: alleviated, stable, and aggravated. The performance of models trained on various feature subsets was compared with five-fold cross validation.559 patients with 1734 examinations were collected, and 1450 progression sequences were generated. Of the 559 patients, 262 (46.9%) were male. The mean age of the patients was 60 ± 14 years. The mean hospitalization duration was 31 ± 12 days. Based on the ranking of importance, 26 features from the imaging and laboratory tests were selected, achieving the best accuracy of 0.85 for progression assessment. The comparisons demonstrated that CT features outperformed laboratory features. The best sensitivities for alleviated and aggravated obtained with CT features alone were 0.83 and 0.85, respectively, while laboratory features improved the assessment precision by about 3%. Longitudinal assessment using deep learning with combined features from CT imaging and laboratory tests better predicts the progression of COVID-19 than either of them.

Recently, the rapid advancements in Deep Learning and Computer Vision technologies have introduced a new and exciting era in the field of skin disease analysis. However, there are certain challenges in the roadmap towards developing such technologies for real-life applications that must be investigated. This study considers one of the key challenges in data acquisition and computation, viz. data scarcity. Data scarcity is a central problem in acquiring medical images and applying machine learning techniques to train Convolutional Neural Networks for disease diagnosis. The main objective of this study is to explore the possible methods to deal with the data scarcity problem and to improve diagnosis with small datasets. The challenges in data acquisition for a few lamentably neglected skin conditions such as rosacea are an excellent instance to explore the possibilities of improving computer-aided skin disease diagnosis. With data scarcity in mind, the possible techniques explored and discussed include Generative Adversarial Networks, Meta-Learning, Few-Shot classification, and 3D face modelling. Furthermore, the existing studies are discussed based on skin conditions considered, data volume and implementation choices. Some future research directions are recommended.

Optical Coherence Tomography (OCT) is a rapidly growing imaging modality in biomedical optics. OCT can perform high-resolution, cross-sectional imaging of the microstructure of biological tissues by measuring the coherent spectrum from the backscattered light. OCT systems with broad spectral bandwidths are often constructed using free-space optics to avoid dispersion by fibre optic components. This paper presents a fibre-based OCT system at a centre wavelength of 1300 nm with an axial resolution of 3.8 µm in air, surpassing any previously reported values to the best of our knowledge. Despite the challenges in transporting a broadband spectrum using fibre-optics, the system investigation was motivated by the ever-increasing demand for commercialization of high-resolution OCT systems and simplification of construction. We also evaluate and demonstrate the direct measurement method for axial resolution using an air wedge. Imaging of biomedical and other samples is demonstrated using a high numerical aperture sample lens and compared with images from a commercial OCT system. We discuss the effect of the improved structural visibility by achieving image voxels closer to an isometric shape with a high NA sample lens.

In a mobile millimeter wave (mmWave) communication system, blockages cause disconnections or serious degradation of communications. Several techniques have been proposed to control radio links between multiple base stations based on blockage prediction using camera images to avoid these problems. However, blockage prediction requires continuously determining the position of user equipment (UE) with decimeter precision, which is difficult when sensors and resources on the UE side are not available, and there are many moving objects around the UE. To resolve this problem, we propose a UE tracking method that uses a received signal strength indicator (RSSI) and RGB-D camera images from the base station. The proposed method consists of a combination of visual tracking and reidentification of the UE using radio information and camera images. We use the synchronization of the RSSI variation with the occlusion on the image by movements of the UE and objects for reidentification. We evaluated the proposed method experimentally in an outdoor environment by simulating a communication area formed by mmWave band base stations. The proposed method achieved an 11.5% improvement in tracking accuracy compared with conventional visual tracking.

Conventional deformable registration methods aim at solving an optimization model carefully designed on image pairs and their computational costs are exceptionally high. In contrast, recent deep learning-based approaches can provide fast deformation estimation. These heuristic network architectures are fully data-driven and thus lack explicit geometric constraints which are indispensable to generate plausible deformations, e.g., topology-preserving. Moreover, these learning-based approaches typically pose hyper-parameter learning as a black-box problem and require considerable computational and human effort to perform many training runs. To tackle the aforementioned problems, we propose a new learning-based framework to optimize a diffeomorphic model via multi-scale propagation. Specifically, we introduce a generic optimization model to formulate diffeomorphic registration and develop a series of learnable architectures to obtain propagative updating in the coarse-to-fine feature space. Further, we propose a new bilevel self-tuned training strategy, allowing efficient search of task-specific hyper-parameters. This training strategy increases the flexibility to various types of data while reduces computational and human burdens. We conduct two groups of image registration experiments on 3D volume datasets including image-to-atlas registration on brain MRI data and image-to-image registration on liver CT data. Extensive results demonstrate the state-of-the-art performance of the proposed method with diffeomorphic guarantee and extreme efficiency. We also apply our framework to challenging multi-modal image registration, and investigate how our registration to support the down-streaming tasks for medical image analysis including multi-modal fusion and image segmentation. [ABSTRACT FROM AUTHOR]

Medical image segmentation is fundamental and essential for the analysis of medical images. Although prevalent success has been achieved by convolutional neural networks (CNN), challenges are encountered in the domain of medical image analysis by two aspects: 1) lack of discriminative features to handle similar textures of distinct structures and 2) lack of selective features for potential blurred boundaries in medical images. In this paper, we extend the concept of contrastive learning (CL) to the segmentation task to learn more discriminative representation. Specifically, we propose a novel patch-dragsaw contrastive regularization (PDCR) to perform patch-level tugging and repulsing. In addition, a new structure, namely uncertainty-aware feature re- weighting block (UAFR), is designed to address the potential high uncertainty regions in the feature maps and serves as a better feature re- weighting. Our proposed method achieves state-of-the-art results across 8 public datasets from 6 domains. Besides, the method also demonstrates robustness in the limited-data scenario. The code is publicly available at https://github.com/lzh19961031/PDCR_UAFR-MIShttps://github.com/lzh19961031/PDCR_UAFR-MIS. [ABSTRACT FROM AUTHOR]

In recent years, there were many suggestions regarding modifications of the well-known U-Net architecture in order to improve its performance. The central motivation of this work is to provide a fair comparison of U-Net and its five extensions using identical conditions to disentangle the influence of model architecture, model training, and parameter settings on the performance of a trained model. For this purpose each of these six segmentation architectures is trained on the same nine data sets. The data sets are selected to cover various imaging modalities (X-rays, computed tomography, magnetic resonance imaging), single- and multi-class segmentation problems, and single- and multi-modal inputs. During the training, it is ensured that the data preprocessing, data set split into training, validation, and testing subsets, optimizer, learning rate change strategy, architecture depth, loss function, supervision and inference are exactly the same for all the architectures compared. Performance is evaluated in terms of Dice coefficient, surface Dice coefficient, average surface distance, Hausdorff distance, training, and prediction time. The main contribution of this experimental study is demonstrating that the architecture variants do not improve the quality of inference related to the basic U-Net architecture while resource demand rises. [ABSTRACT FROM AUTHOR]

Representation learning is the critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the dataset and the lack of labels. In this paper, we propose a stochastic routing normalization and neighborhood embedding framework with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy. In order to align the low-level and mid-level of pCLE and histology domain, we firstly build the domain-specific normalization module with stochastic activation strategy considering both depth-wise and feature-wise criterion. For high-level features, the latent centers are learned from the histology domain as the template for feature matching. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image classification with limited training samples demonstrates that the proposed method can outperform previous works on domain alignment. [ABSTRACT FROM AUTHOR]

The performance of deep networks for medical image analysis is often constrained by limited medical data, which is privacy-sensitive. Federated learning (FL) alleviates the constraint by allowing different institutions to collaboratively train a federated model without sharing data. However, the federated model is often suboptimal with respect to the characteristics of each client's local data. Instead of training a single global model, we propose Customized FL (CusFL), for which each client iteratively trains a client-specific/private model based on a federated global model aggregated from all private models trained in the immediate previous iteration. Two overarching strategies employed by CusFL lead to its superior performance: 1) the federated model is mainly for feature alignment and thus only consists of feature extraction layers; 2) the federated feature extractor is used to guide the training of each private model. In that way, CusFL allows each client to selectively learn useful knowledge from the federated model to improve its personalized model. We evaluated CusFL on multi-source medical image datasets for the identification of clinically significant prostate cancer and the classification of skin lesions. [ABSTRACT FROM AUTHOR]

In medical image analysis (MIA) and computer-assisted surgery (CAS), aligning two multiple point sets (PSs) together is an essential but also a challenging problem. For example, rigidly aligning multiple point sets into one common coordinate frame is a prerequisite for statistical shape modelling (SSM). Accurately aligning the pre-operative space with the intra-operative space in CAS is very crucial to successful interventions. In this article, we formally formulate the multiple generalized point set registration problem (MGPSR) in a probabilistic manner, where both the positional and the normal vectors are used. The six-dimensional vectors consisting of both positional and normal vectors are called as generalized points. In the formulated model, all the generalized PSs to be registered are considered to be the realizations of underlying unknown hybrid mixture models (HMMs). By assuming the independence of the positional and orientational vectors (i.e., the normal vectors), the probability density function (PDF) of an observed generalized point is computed as the product of Gaussian and Fisher distributions. Furthermore, to consider the anisotropic noise in surgical navigation, the positional error is assumed to obey a multi-variate Gaussian distribution. Finally, registering PSs is formulated as a maximum likelihood (ML) problem, and solved under the expectation maximization (EM) technique. By using more enriched information (i.e., the normal vectors), our algorithm is more robust to outliers. By treating all PSs equally, our algorithm does not bias towards any PS. To validate the proposed approach, extensive experiments have been conducted on surface points extracted from CT images of (i) a human femur bone model; (ii) a human pelvis bone model. Results demonstrate our algorithm’s high accuracy, robustness to noise and outliers. Note to Practitioners—This paper was motivated by solving the problem of registering two or more PSs. Most existing registration approaches use only the positional information associated with each point, and thus lacks robustness to noise and outliers. Three significant improvements are brought by our proposed approach. First, the normal vectors that can be extracted from the point sets are utilized in the registration. Second, the positional error distribution is assumed to be anisotropic and inhomogeneous. Third, all the PSs to be registered are treated equally that means no PS is considered as the model one. The registration problem is cast into a maximum likelihood (ML) problem and solved under the expectation maximization (EM) framework. We have demonstrated through extensive experiments that the proposed registration approach achieves significantly improved accuracy, robustness to noise and outliers. The algorithm is particularly suitable for biomedical applications involving the registration procedures, such as image-guided surgery. [ABSTRACT FROM AUTHOR]

Tensor data with rich structural information become increasingly important in process modeling, monitoring, and diagnosis in manufacturing medical and other applications. Here structural information is referred to the information of tensor components such as sparsity, smoothness, low-rank, and piecewise constancy. To reveal useful information from tensor data, we propose to decompose the tensor into the summation of multiple components based on their different structural information. In this article, we provide a new definition of structural information in tensor data. We then propose an additive tensor decomposition (ATD) framework to extract useful information from tensor data. This framework specifies a high dimensional optimization problem to obtain the components with distinct structural information. An alternating direction method of multipliers (ADMM) algorithm is proposed to solve it, which is highly parallelable and thus suitable for the proposed optimization problem. Two simulation examples and a real case study in medical image analysis illustrate the versatility and effectiveness of the ATD framework. Note to Practitioners—This article was motivated by a real case in medical imaging: extracting aortic valve calcification (AVC) regions from the tensor data obtained from computed tomography (CT) image series of the aortic region. The main objective is to decompose image series into multiple components corresponding to tissues, calcium deposition, and error. Similar needs are pervasive in other medical image analysis applications as well as the image-based modeling, monitoring, and diagnosis of industrial processes and systems. Existing methods fail to incorporate a detailed description of the properties of image series that reflect the physical understanding of the system in both the spatial and temporal domains. In this article, we provide a systematic description of the properties of image series and use them to develop a decomposition framework. It is applicable to various applications and can generate more accurate and interpretable results. [ABSTRACT FROM AUTHOR]

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges – the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction. [ABSTRACT FROM AUTHOR]