Search

Segmentation of major brain vessels is very important for the diagnosis of cerebrovascular disorders and subsequent surgical planning. Vessel segmentation is an important pre-processing step for a wide range of algorithms for the automatic diagnosis or treatment of several vascular pathologies and as such, it is valuable to have a well-performing vascular segmentation pipeline. In this article, we propose an end-to-end multiscale residual dual attention deep neural network for resilient major brain vessel segmentation. In the proposed network, the encoder and decoder blocks of the U-Net are replaced with the multi-level atrous residual blocks to enhance the learning capability by increasing the receptive field to extract the various semantic coarse- and fine- grained features. Dual attention block is incorporated in the bottleneck to perform effective multiscale information fusion to obtain detailed structure of blood vessels. The methods were evaluated on the publicly available TubeTK data set. The proposed method outperforms the state-of-the-art techniques with dice of 0.79 on the whole-brain prediction. The statistical and visual assessments indicate that proposed network is robust to outliers and maintains higher consistency in vessel continuity than the traditional U-Net and its variations.

Objective: Large-scale microscopy-based experiments often result in images with rich but sparse information content. An experienced microscopist can visually identify regions of interest (ROIs), but this becomes a cumbersome task with large datasets. Here we present SimSearch, a framework for quick and easy user-guided training of a deep neural model aimed at fast detection of ROIs in large-scale microscopy experiments. Methods: The user manually selects a small number of patches representing different classes of ROIs. This is followed by feature extraction using a pre-trained deep-learning model, and interactive patch selection pruning, resulting in a smaller set of clean (user approved) and larger set of noisy (unapproved) training patches of ROIs and background. The pre-trained deep-learning model is thereafter first trained on the large set of noisy patches, followed by refined training using the clean patches. Results: The framework is evaluated on fluorescence microscopy images from a large-scale drug screening experiment, brightfield images of immunohistochemistry-stained patient tissue samples, and malaria-infected human blood smears, as well as transmission electron microscopy images of cell sections. Compared to state-of-the-art and manual/visual assessment, the results show similar performance with maximal flexibility and minimal a priori information and user interaction. Conclusions: SimSearch quickly adapts to different data sets, which demonstrates the potential to speed up many microscopy-based experiments based on a small amount of user interaction. Significance: SimSearch can help biologists quickly extract informative regions and perform analyses on large datasets helping increase the throughput in a microscopy experiment.

Objective: Large-scale microscopy-based experiments often result in images with rich but sparse information content. An experienced microscopist can visually identify regions of interest (ROIs), but this becomes a cumbersome task with large datasets. Here we present SimSearch, a framework for quick and easy user-guided training of a deep neural model aimed at fast detection of ROIs in large-scale microscopy experiments. Methods: The user manually selects a small number of patches representing different classes of ROIs. This is followed by feature extraction using a pre-trained deep-learning model, and interactive patch selection pruning, resulting in a smaller set of clean (user approved) and larger set of noisy (unapproved) training patches of ROIs and background. The pre-trained deep-learning model is thereafter first trained on the large set of noisy patches, followed by refined training using the clean patches. Results: The framework is evaluated on fluorescence microscopy images from a large-scale drug screening experiment, brightfield images of immunohistochemistry-stained patient tissue samples, and malaria-infected human blood smears, as well as transmission electron microscopy images of cell sections. Compared to state-of-the-art and manual/visual assessment, the results show similar performance with maximal flexibility and minimal a priori information and user interaction. Conclusions: SimSearch quickly adapts to different data sets, which demonstrates the potential to speed up many microscopy-based experiments based on a small amount of user interaction. Significance: SimSearch can help biologists quickly extract informative regions and perform analyses on large datasets helping increase the throughput in a microscopy experiment.

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

Objective: Large-scale microscopy-based experiments often result in images with rich but sparse information content. An experienced microscopist can visually identify regions of interest (ROIs), but this becomes a cumbersome task with large datasets. Here we present SimSearch, a framework for quick and easy user-guided training of a deep neural model aimed at fast detection of ROIs in large-scale microscopy experiments. Methods: The user manually selects a small number of patches representing different classes of ROIs. This is followed by feature extraction using a pre-trained deep-learning model, and interactive patch selection pruning, resulting in a smaller set of clean (user approved) and larger set of noisy (unapproved) training patches of ROIs and background. The pre-trained deep-learning model is thereafter first trained on the large set of noisy patches, followed by refined training using the clean patches. Results: The framework is evaluated on fluorescence microscopy images from a large-scale drug screening experiment, brightfield images of immunohistochemistry-stained patient tissue samples, and malaria-infected human blood smears, as well as transmission electron microscopy images of cell sections. Compared to state-of-the-art and manual/visual assessment, the results show similar performance with maximal flexibility and minimal a priori information and user interaction. Conclusions: SimSearch quickly adapts to different data sets, which demonstrates the potential to speed up many microscopy-based experiments based on a small amount of user interaction. Significance: SimSearch can help biologists quickly extract informative regions and perform analyses on large datasets helping increase the throughput in a microscopy experiment.

UK Biobank (UKB) has recruited more than 500000 volunteers from the United Kingdom, collecting health-related information on genetics, lifestyle, blood biochemistry, and more. Ongoing medical imaging of 100 000 participants with 70 000 follow-up sessions will yield up to 170 000 MRI scans, enabling image analysis of body composition, organs, and muscle. This study presents an experimental inference engine for automated analysis of UKB neck-to-knee body 1.5-T MRI scans. This retrospective cross-validation study includes data from 38 916 participants (52% female; mean age, 64 years) to capture baseline characteristics, such as age, height, weight, and sex, as well as measurements of body composition, organ volumes, and abstract properties, such as grip strength, pulse rate, and type 2 diabetes status. Prediction intervals for each end point were generated based on uncertainty quantification. On a subsequent release of UKB data, the proposed method predicted 12 body composition metrics with a 3% median error and yielded mostly well-calibrated individual prediction intervals. The processing of MRI scans from 1000 participants required 10 minutes. The underlying method used convolutional neural networks for image-based mean-variance regression on two-dimensional representations of the MRI data. An implementation was made publicly available for fast and fully automated estimation of 72 different measurements from future releases of UKB image data. (C) RSNA, 2022., Erratum in:Radiology: Artificial Intelligence 2022;4(4)https://doi.org/10.1148/ryai.229001

Objective: Large-scale microscopy-based experiments often result in images with rich but sparse information content. An experienced microscopist can visually identify regions of interest (ROIs), but this becomes a cumbersome task with large datasets. Here we present SimSearch, a framework for quick and easy user-guided training of a deep neural model aimed at fast detection of ROIs in large-scale microscopy experiments. Methods: The user manually selects a small number of patches representing different classes of ROIs. This is followed by feature extraction using a pre-trained deep-learning model, and interactive patch selection pruning, resulting in a smaller set of clean (user approved) and larger set of noisy (unapproved) training patches of ROIs and background. The pre-trained deep-learning model is thereafter first trained on the large set of noisy patches, followed by refined training using the clean patches. Results: The framework is evaluated on fluorescence microscopy images from a large-scale drug screening experiment, brightfield images of immunohistochemistry-stained patient tissue samples, and malaria-infected human blood smears, as well as transmission electron microscopy images of cell sections. Compared to state-of-the-art and manual/visual assessment, the results show similar performance with maximal flexibility and minimal a priori information and user interaction. Conclusions: SimSearch quickly adapts to different data sets, which demonstrates the potential to speed up many microscopy-based experiments based on a small amount of user interaction. Significance: SimSearch can help biologists quickly extract informative regions and perform analyses on large datasets helping increase the throughput in a microscopy experiment.

Microscopy imaging is a powerful technique when studying biology at a cellular and sub-cellular level. When combined with digital image analysis it creates an invaluable tool for investigating complex biological processes and phenomena. However, imaging at the cell and sub-cellular level tends to generate large amounts of data which can be difficult to analyse, navigate and store. Despite these difficulties, large data volumes mean more information content which is beneficial for computational methods like machine learning, especially deep learning. The union of microscopy imaging and deep learning thus provides numerous opportunities for advancing our scientific understanding and uncovering interesting and useful biological insights. The work in this thesis explores various means for optimising information extraction from microscopy data utilising image analysis with deep learning. The focus is on three different imaging modalities: bright-field; fluorescence; and transmission electron microscopy. Within these modalities different learning-based image analysis and processing techniques are explored, ranging from image classification and detection to image restoration and translation. The main contributions are: (i) a computational method for diagnosing oral and cervical cancer based on smear samples and bright-field microscopy; (ii) a hierarchical analysis of whole-slide tissue images from fluorescence microscopy and introducing a confidence based measure for pixel classifications; (iii) an image restoration model for motion-degraded images from transmission electron microscopy with an evaluation of model overfitting on underlying textures; and (iv) an image-to-image translation (virtual staining) of cell images from bright-field to fluorescence microscopy, optimised for biological feature relevance. A common theme underlying all the investigations in this thesis is that the evaluation of the methods used is in relation to the biological question at hand.

Microscopy imaging is a powerful technique when studying biology at a cellular and sub-cellular level. When combined with digital image analysis it creates an invaluable tool for investigating complex biological processes and phenomena. However, imaging at the cell and sub-cellular level tends to generate large amounts of data which can be difficult to analyse, navigate and store. Despite these difficulties, large data volumes mean more information content which is beneficial for computational methods like machine learning, especially deep learning. The union of microscopy imaging and deep learning thus provides numerous opportunities for advancing our scientific understanding and uncovering interesting and useful biological insights. The work in this thesis explores various means for optimising information extraction from microscopy data utilising image analysis with deep learning. The focus is on three different imaging modalities: bright-field; fluorescence; and transmission electron microscopy. Within these modalities different learning-based image analysis and processing techniques are explored, ranging from image classification and detection to image restoration and translation. The main contributions are: (i) a computational method for diagnosing oral and cervical cancer based on smear samples and bright-field microscopy; (ii) a hierarchical analysis of whole-slide tissue images from fluorescence microscopy and introducing a confidence based measure for pixel classifications; (iii) an image restoration model for motion-degraded images from transmission electron microscopy with an evaluation of model overfitting on underlying textures; and (iv) an image-to-image translation (virtual staining) of cell images from bright-field to fluorescence microscopy, optimised for biological feature relevance. A common theme underlying all the investigations in this thesis is that the evaluation of the methods used is in relation to the biological question at hand.

Much biomedical and medical research relies on the collection of ever-larger amounts of image data (both 2D images and 3D volumes, as well as time-series) and increasingly from multiple sources. Image registration, the process of finding correspondences between images based on the affinity of features of interest, is often required as a vital step towards the final analysis, which may consist of a comparison of images, measurement of movement, or fusion of complementary information. The contributions in this work are centered around reliable image registration methods for both 2D and 3D images with the aim of wide applicability: similarity and distance measures between images for image registration, algorithms for efficient computation of these, and other commonly used measures for both local and global optimization frameworks, and representations for multimodal image registration where the appearance and structures present in the images may vary dramatically. The main contributions are: (i) distance measures for affine symmetric intensity image registration, combining intensity and spatial information based on the notion of alpha-cuts from fuzzy set theory; (ii) the extension of the affine registration method to more flexible deformable transformation models, leading to the framework Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE); (iii) two efficient algorithms for computing the proposed distances and their spatial gradients and thereby enabling local gradient-based optimization; (iv) a contrastive representation learning method, Contrastive Multimodal Image Representation for Registration (CoMIR), utilizing deep learning techniques to obtain common representations that can be registered using methods designed for monomodal scenarios; (v) efficient algorithms for global optimization of mutual information and similarities of normalized gradient fields; (vi) a comparative study exploring the applicability of modern image-to-image tran

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

Arterial cerebral vessel assessment is critical for thediagnosis of patients with cerebrovascular disease e.g., hypertension, Intracranial aneurysms, and dementia. Magnetic resonance angiography is a primary imaging technique for diagnosing cerebrovascular diseases. There are many Convolutional neuralnetworks (CNN) based methods for cerebral vessel segmentation but lack to identify the target vessels and understand the arterial tree structure for diagnosis and endovascular surgical planning.In the present study, we generated annotations for major vesselsegmentation and analyzed fully automatic segmentation of major vessels using state-of-the-art U-Net based deep learning models. Computer-aided major cerebral vessel segmentation incorporatedinto clinical practice may help speed up the diagnosis of time-critical vessel anomalies and help find important bio-markers for neurological dysfunction. We validated and compared U-Net based models for volumetric segmentation and predictionof cerebral arteries and it could be done in real-time withoutany image pre-processing.

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

Microscopy imaging is a powerful technique when studying biology at a cellular and sub-cellular level. When combined with digital image analysis it creates an invaluable tool for investigating complex biological processes and phenomena. However, imaging at the cell and sub-cellular level tends to generate large amounts of data which can be difficult to analyse, navigate and store. Despite these difficulties, large data volumes mean more information content which is beneficial for computational methods like machine learning, especially deep learning. The union of microscopy imaging and deep learning thus provides numerous opportunities for advancing our scientific understanding and uncovering interesting and useful biological insights. The work in this thesis explores various means for optimising information extraction from microscopy data utilising image analysis with deep learning. The focus is on three different imaging modalities: bright-field; fluorescence; and transmission electron microscopy. Within these modalities different learning-based image analysis and processing techniques are explored, ranging from image classification and detection to image restoration and translation. The main contributions are: (i) a computational method for diagnosing oral and cervical cancer based on smear samples and bright-field microscopy; (ii) a hierarchical analysis of whole-slide tissue images from fluorescence microscopy and introducing a confidence based measure for pixel classifications; (iii) an image restoration model for motion-degraded images from transmission electron microscopy with an evaluation of model overfitting on underlying textures; and (iv) an image-to-image translation (virtual staining) of cell images from bright-field to fluorescence microscopy, optimised for biological feature relevance. A common theme underlying all the investigations in this thesis is that the evaluation of the methods used is in relation to the biological question at hand.

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

Much biomedical and medical research relies on the collection of ever-larger amounts of image data (both 2D images and 3D volumes, as well as time-series) and increasingly from multiple sources. Image registration, the process of finding correspondences between images based on the affinity of features of interest, is often required as a vital step towards the final analysis, which may consist of a comparison of images, measurement of movement, or fusion of complementary information. The contributions in this work are centered around reliable image registration methods for both 2D and 3D images with the aim of wide applicability: similarity and distance measures between images for image registration, algorithms for efficient computation of these, and other commonly used measures for both local and global optimization frameworks, and representations for multimodal image registration where the appearance and structures present in the images may vary dramatically. The main contributions are: (i) distance measures for affine symmetric intensity image registration, combining intensity and spatial information based on the notion of alpha-cuts from fuzzy set theory; (ii) the extension of the affine registration method to more flexible deformable transformation models, leading to the framework Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE); (iii) two efficient algorithms for computing the proposed distances and their spatial gradients and thereby enabling local gradient-based optimization; (iv) a contrastive representation learning method, Contrastive Multimodal Image Representation for Registration (CoMIR), utilizing deep learning techniques to obtain common representations that can be registered using methods designed for monomodal scenarios; (v) efficient algorithms for global optimization of mutual information and similarities of normalized gradient fields; (vi) a comparative study exploring the applicability of modern image-to-image tran

Microscopy imaging is a powerful technique when studying biology at a cellular and sub-cellular level. When combined with digital image analysis it creates an invaluable tool for investigating complex biological processes and phenomena. However, imaging at the cell and sub-cellular level tends to generate large amounts of data which can be difficult to analyse, navigate and store. Despite these difficulties, large data volumes mean more information content which is beneficial for computational methods like machine learning, especially deep learning. The union of microscopy imaging and deep learning thus provides numerous opportunities for advancing our scientific understanding and uncovering interesting and useful biological insights. The work in this thesis explores various means for optimising information extraction from microscopy data utilising image analysis with deep learning. The focus is on three different imaging modalities: bright-field; fluorescence; and transmission electron microscopy. Within these modalities different learning-based image analysis and processing techniques are explored, ranging from image classification and detection to image restoration and translation. The main contributions are: (i) a computational method for diagnosing oral and cervical cancer based on smear samples and bright-field microscopy; (ii) a hierarchical analysis of whole-slide tissue images from fluorescence microscopy and introducing a confidence based measure for pixel classifications; (iii) an image restoration model for motion-degraded images from transmission electron microscopy with an evaluation of model overfitting on underlying textures; and (iv) an image-to-image translation (virtual staining) of cell images from bright-field to fluorescence microscopy, optimised for biological feature relevance. A common theme underlying all the investigations in this thesis is that the evaluation of the methods used is in relation to the biological question at hand.

Objective: Large-scale microscopy-based experiments often result in images with rich but sparse information content. An experienced microscopist can visually identify regions of interest (ROIs), but this becomes a cumbersome task with large datasets. Here we present SimSearch, a framework for quick and easy user-guided training of a deep neural model aimed at fast detection of ROIs in large-scale microscopy experiments. Methods: The user manually selects a small number of patches representing different classes of ROIs. This is followed by feature extraction using a pre-trained deep-learning model, and interactive patch selection pruning, resulting in a smaller set of clean (user approved) and larger set of noisy (unapproved) training patches of ROIs and background. The pre-trained deep-learning model is thereafter first trained on the large set of noisy patches, followed by refined training using the clean patches. Results: The framework is evaluated on fluorescence microscopy images from a large-scale drug screening experiment, brightfield images of immunohistochemistry-stained patient tissue samples, and malaria-infected human blood smears, as well as transmission electron microscopy images of cell sections. Compared to state-of-the-art and manual/visual assessment, the results show similar performance with maximal flexibility and minimal a priori information and user interaction. Conclusions: SimSearch quickly adapts to different data sets, which demonstrates the potential to speed up many microscopy-based experiments based on a small amount of user interaction. Significance: SimSearch can help biologists quickly extract informative regions and perform analyses on large datasets helping increase the throughput in a microscopy experiment.

Multimodal image alignment involves finding spatial correspondences between volumes varying in appearance and structure. Automated alignment methods are often based on local optimization that can be highly sensitive to initialization. We propose a novel efficient algorithm for computing similarity of normalized gradient fields (NGF) in the frequency domain, which we globally optimize to achieve rigid multimodal 3D image alignment. We validate the method experimentally on a dataset comprised of 20 brain volumes acquired in four modalities (T1w, Flair, CT, [18F] FDG PET), synthetically displaced with known transformations. The proposed method exhibits excellent performance on all six possible modality combinations and outperforms the four considered reference methods by a large margin. An important advantage of the method is its speed; global rigid alignment of 3.4 Mvoxel volumes requires approximately 40 seconds of computation, and the proposed algorithm outperforms a direct algorithm for the same task by more than three orders of magnitude. Open-source code is provided.

Much biomedical and medical research relies on the collection of ever-larger amounts of image data (both 2D images and 3D volumes, as well as time-series) and increasingly from multiple sources. Image registration, the process of finding correspondences between images based on the affinity of features of interest, is often required as a vital step towards the final analysis, which may consist of a comparison of images, measurement of movement, or fusion of complementary information. The contributions in this work are centered around reliable image registration methods for both 2D and 3D images with the aim of wide applicability: similarity and distance measures between images for image registration, algorithms for efficient computation of these, and other commonly used measures for both local and global optimization frameworks, and representations for multimodal image registration where the appearance and structures present in the images may vary dramatically. The main contributions are: (i) distance measures for affine symmetric intensity image registration, combining intensity and spatial information based on the notion of alpha-cuts from fuzzy set theory; (ii) the extension of the affine registration method to more flexible deformable transformation models, leading to the framework Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE); (iii) two efficient algorithms for computing the proposed distances and their spatial gradients and thereby enabling local gradient-based optimization; (iv) a contrastive representation learning method, Contrastive Multimodal Image Representation for Registration (CoMIR), utilizing deep learning techniques to obtain common representations that can be registered using methods designed for monomodal scenarios; (v) efficient algorithms for global optimization of mutual information and similarities of normalized gradient fields; (vi) a comparative study exploring the applicability of modern image-to-image tran

Much biomedical and medical research relies on the collection of ever-larger amounts of image data (both 2D images and 3D volumes, as well as time-series) and increasingly from multiple sources. Image registration, the process of finding correspondences between images based on the affinity of features of interest, is often required as a vital step towards the final analysis, which may consist of a comparison of images, measurement of movement, or fusion of complementary information. The contributions in this work are centered around reliable image registration methods for both 2D and 3D images with the aim of wide applicability: similarity and distance measures between images for image registration, algorithms for efficient computation of these, and other commonly used measures for both local and global optimization frameworks, and representations for multimodal image registration where the appearance and structures present in the images may vary dramatically. The main contributions are: (i) distance measures for affine symmetric intensity image registration, combining intensity and spatial information based on the notion of alpha-cuts from fuzzy set theory; (ii) the extension of the affine registration method to more flexible deformable transformation models, leading to the framework Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE); (iii) two efficient algorithms for computing the proposed distances and their spatial gradients and thereby enabling local gradient-based optimization; (iv) a contrastive representation learning method, Contrastive Multimodal Image Representation for Registration (CoMIR), utilizing deep learning techniques to obtain common representations that can be registered using methods designed for monomodal scenarios; (v) efficient algorithms for global optimization of mutual information and similarities of normalized gradient fields; (vi) a comparative study exploring the applicability of modern image-to-image tran

Microscopy imaging is a powerful technique when studying biology at a cellular and sub-cellular level. When combined with digital image analysis it creates an invaluable tool for investigating complex biological processes and phenomena. However, imaging at the cell and sub-cellular level tends to generate large amounts of data which can be difficult to analyse, navigate and store. Despite these difficulties, large data volumes mean more information content which is beneficial for computational methods like machine learning, especially deep learning. The union of microscopy imaging and deep learning thus provides numerous opportunities for advancing our scientific understanding and uncovering interesting and useful biological insights. The work in this thesis explores various means for optimising information extraction from microscopy data utilising image analysis with deep learning. The focus is on three different imaging modalities: bright-field; fluorescence; and transmission electron microscopy. Within these modalities different learning-based image analysis and processing techniques are explored, ranging from image classification and detection to image restoration and translation. The main contributions are: (i) a computational method for diagnosing oral and cervical cancer based on smear samples and bright-field microscopy; (ii) a hierarchical analysis of whole-slide tissue images from fluorescence microscopy and introducing a confidence based measure for pixel classifications; (iii) an image restoration model for motion-degraded images from transmission electron microscopy with an evaluation of model overfitting on underlying textures; and (iv) an image-to-image translation (virtual staining) of cell images from bright-field to fluorescence microscopy, optimised for biological feature relevance. A common theme underlying all the investigations in this thesis is that the evaluation of the methods used is in relation to the biological question at hand.

Much biomedical and medical research relies on the collection of ever-larger amounts of image data (both 2D images and 3D volumes, as well as time-series) and increasingly from multiple sources. Image registration, the process of finding correspondences between images based on the affinity of features of interest, is often required as a vital step towards the final analysis, which may consist of a comparison of images, measurement of movement, or fusion of complementary information. The contributions in this work are centered around reliable image registration methods for both 2D and 3D images with the aim of wide applicability: similarity and distance measures between images for image registration, algorithms for efficient computation of these, and other commonly used measures for both local and global optimization frameworks, and representations for multimodal image registration where the appearance and structures present in the images may vary dramatically. The main contributions are: (i) distance measures for affine symmetric intensity image registration, combining intensity and spatial information based on the notion of alpha-cuts from fuzzy set theory; (ii) the extension of the affine registration method to more flexible deformable transformation models, leading to the framework Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE); (iii) two efficient algorithms for computing the proposed distances and their spatial gradients and thereby enabling local gradient-based optimization; (iv) a contrastive representation learning method, Contrastive Multimodal Image Representation for Registration (CoMIR), utilizing deep learning techniques to obtain common representations that can be registered using methods designed for monomodal scenarios; (v) efficient algorithms for global optimization of mutual information and similarities of normalized gradient fields; (vi) a comparative study exploring the applicability of modern image-to-image tran

Multimodal image alignment is the process of finding spatial correspondences between images formed by different imaging techniques or under different conditions, to facilitate heterogeneous data fusion and correlative analysis. The information-theoretic concept of mutual information (MI) is widely used as a similarity measure to guide multimodal alignment processes, where most works have focused on local maximization of MI, which typically works well only for small displacements. This points to a need for global maximization of MI, which has previously been computationally infeasible due to the high run-time complexity of existing algorithms. We propose an efficient algorithm for computing MI for all discrete displacements (formalized as the cross-mutual information function (CMIF)), which is based on cross-correlation computed in the frequency domain. We show that the algorithm is equivalent to a direct method while superior in terms of run-time. Furthermore, we propose a method for multimodal image alignment for transformation models with few degrees of freedom (e.g., rigid) based on the proposed CMIF-algorithm. We evaluate the efficacy of the proposed method on three distinct benchmark datasets, containing remote sensing images, cytological images, and histological images, and we observe excellent success-rates (in recovering known rigid transformations), overall outperforming alternative methods, including local optimization of MI, as well as several recent deep learning-based approaches. We also evaluate the run-times of a GPU implementation of the proposed algorithm and observe speed-ups from 100 to more than 10,000 times for realistic image sizes compared to a GPU implementation of a direct method. Code is shared as open-source at github.com/MIDA-group/globalign.

Multiplexed and spatially resolved single-cell analyses that intend to study tissue heterogeneity and cell organization invariably face as a first step the challenge of cell classification. Accuracy and reproducibility are important for the downstream process of counting cells, quantifying cell–cell interactions, and extracting information on disease-specific localized cell niches. Novel staining techniques make it possible to visualize and quantify large numbers of cell-specific molecular markers in parallel. However, due to variations in sample handling and artifacts from staining and scanning, cells of the same type may present different marker profiles both within and across samples. We address multiplexed immunofluorescence data from tissue microarrays of low-grade gliomas and present a methodology using two different machine learning architectures and features insensitive to illumination to perform cell classification. The fully automated cell classification provides a measure of confidence for the decision and requires a comparably small annotated data set for training, which can be created using freely available tools. Using the proposed method, we reached an accuracy of 83.1% on cell classification without the need for standardization of samples. Using our confidence measure, cells with low-confidence classifications could be excluded, pushing the classification accuracy to 94.5%. Next, we used the cell classification results to search for cell niches with an unsupervised learning approach based on graph neural networks. We show that the approach can re-detect specialized tissue niches in previously published data, and that our proposed cell classification leads to niche definitions that may be relevant for sub-groups of glioma, if applied to larger data sets.

Along with rich health-related metadata, an ongoing imaging study has acquired MRI of over 40,000 male and female UK Biobank participants aged 44-82 since 2014. Phenotypes derived from these images, such as measurements of body composition, can reveal new links between genetics, cardiovascular disease, and metabolic conditions. In this retrospective study, six measurements of body composition were automatically estimated by ResNet50 neural networks for image-based regression from neck-to-knee body MRI. Despite the potential for high speed and accuracy, these networks produce no output segmentations that could indicate the reliability of individual measurements. The presented experiments therefore examine mean-variance regression and ensembling for predictive uncertainty estimation, which can quantify individual measurement errors and thereby help to identify potential outliers, anomalies, and other failure cases automatically. In 10-fold cross-validation on data of about 8,500 subjects, mean-variance regression and ensembling showed complementary benefits, reducing the mean absolute error across all predictions by 12%. Both improved the calibration of uncertainties and their ability to identify high prediction errors. With intra-class correlation coefficients (ICC) above 0.97, all targets except the liver fat content yielded relative measurement errors below 5%. Testing on another 1,000 subjects showed consistent performance, and the method was finally deployed for inference to 30,000 subjects with missing reference values. The results indicate that deep regression ensembles could ultimately provide automated, uncertainty-aware measurements of body composition for more than 120,000 UK Biobank neck-to-knee body MRI that are to be acquired within the coming years.

Shear wave elasticity imaging (SWEI) is a non-invasive imaging modality that provides tissue elasticity information by measuring the travelling speed of an induced shear-wave. It is commercially available on clinical ultrasound scanners and popularly used in the diagnosis and staging of liver disease and breast cancer. In conventional SWEI methods, a sequence of acoustic radiation force (ARF) pushes are used for inducing a shear-wave, which is tracked using high frame-rate multi-angle plane wave imaging (MA-PWI) to estimate the shear-wave speed (SWS). Conventionally, these plane waves are beamformed using a constant speed-of-sound (SoS), assuming an a-priori known and homogeneous tissue medium. However, soft tissues are inhomogeneous, with intrinsic SoS variations. In this work, we study the SoS effects and inhomogeneities on SWS estimation, using simulation and phantoms experiments with porcine muscle as an abbarator, and show how these aberrations can be corrected using local speed-of-sound adaptive beamforming. For shear-wave tracking, we compare standard beamform with spatially constant SoS values to software beamforming with locally varying SoS maps. We show that, given SoS aberrations, traditional beamforming using a constant SoS, regardless of the utilized SoS value, introduces a substantial bias in the resulting SWS estimations. Average SWS estimation disparity for the same material was observed over 4.3 times worse when a constant SoS value is used compared to that when a known SoS map is used for beamforming. Such biases are shown to be corrected by using a local SoS map in beamforming, indicating the importance of and the need for local SoS reconstruction techniques.

Prostate cancer is a common cancer type in men, yet some of its traits are still under-explored. One reason for this is high molecular and morphological heterogeneity. The purpose of this study was to develop a method to gain new insights into the connection between morphological changes and underlying molecular patterns. We used artificial intelligence (AI) to analyze the morphology of seven hematoxylin and eosin (H & E)-stained prostatectomy slides from a patient with multi-focal prostate cancer. We also paired the slides with spatially resolved expression for thousands of genes obtained by a novel spatial transcriptomics (ST) technique. As both spaces are highly dimensional, we focused on dimensionality reduction before seeking associations between them. Consequently, we extracted morphological features from H & E images using an ensemble of pre-trained convolutional neural networks and proposed a workflow for dimensionality reduction. To summarize the ST data into genetic profiles, we used a previously proposed factor analysis. We found that the regions were automatically defined, outlined by unsupervised clustering, associated with independent manual annotations, in some cases, finding further relevant subdivisions. The morphological patterns were also correlated with molecular profiles and could predict the spatial variation of individual genes. This novel approach enables flexible unsupervised studies relating morphological and genetic heterogeneity using AI to be carried out.

Image-based multiplexed in situ RNA detectionmakes it possible to map the spatial gene expression of hundreds to thousands of genes in parallel, and thus discern at the sametime a large numbers of different cell types to better understand tissue development, heterogeneity, and disease. Fluorescent signals are detected over multiple fluorescent channels and imaging rounds and decoded in order to identify RNA molecules in their morphological context. Here we present a graph-based decoding approach that models the decoding process as a network flow problem jointly optimizing observation likelihoods and distances of signal detections, thus achieving robustness with respect to noise and spatial jitter of the fluorescent signals. We evaluated our method on synthetic data generated at different experimental conditions, and on real data of in situ RNA sequencing, comparng results with respect to alternative and gold standard imagede coding pipelines.

The management of complex mandible fractures, i.e severely comminuted or fractures of edentulous/atrophic mandibles, can be challenging. This is due to the three-dimensional loss of bone, which limits the possibility for accurate anatomic reduction. Virtual surgery planning (VSP) can provide improved accuracy and shorter operating times, but is often not employed for trauma cases because of time constraints and complex user interfaces limited to two-dimensional interaction with three-dimensional data. In this study, we evaluate the accuracy, precision, and time efficiency of the Haptic Assisted Surgery Planning system (HASP), an in-house VSP system that supports stereo graphics, six degrees-of-freedom input and haptics, to improve the surgical planning. Three operators performed planning in HASP on Computed Tomography (CT) and Come Beam Computed Tomography (CBCT) images of a plastic skull model and on twelve retrospective cases with complex mandible fractures. The result shows an accuracy and reproducibility of less than 2mm when using HASP, with an average planning time of 15 minutes, including time for segmentation in the software BoneSplit. This study presents an in-house haptic assisted planning tool for cranio-maxillofacial surgery with high usability that can be used for preoperative planning and evaluation of complex mandible fractures.

Purpose Given the high level of expertise required for navigation and interpretation of ultrasound images, computational simulations can facilitate the training of such skills in virtual reality. With ray-tracing based simulations, realistic ultrasound images can be generated. However, due to computational constraints for interactivity, image quality typically needs to be compromised. Methods We propose herein to bypass any rendering and simulation process at interactive time, by conducting such simulations during a non-time-critical offline stage and then learning image translation from cross-sectional model slices to such simulated frames. We use a generative adversarial framework with a dedicated generator architecture and input feeding scheme, which both substantially improve image quality without increase in network parameters. Integral attenuation maps derived from cross-sectional model slices, texture-friendly strided convolutions, providing stochastic noise and input maps to intermediate layers in order to preserve locality are all shown herein to greatly facilitate such translation task. Results Given several quality metrics, the proposed method with only tissue maps as input is shown to provide comparable or superior results to a state-of-the-art that uses additional images of low-quality ultrasound renderings. An extensive ablation study shows the need and benefits from the individual contributions utilized in this work, based on qualitative examples and quantitative ultrasound similarity metrics. To that end, a local histogram statistics based error metric is proposed and demonstrated for visualization of local dissimilarities between ultrasound images. Conclusion A deep-learning based direct transformation from interactive tissue slices to likeness of high quality renderings allow to obviate any complex rendering process in real-time, which could enable extremely realistic ultrasound simulations on consumer-hardware by moving the time-intensive processe

Knowledge discovery for understanding mechanisms of disease requires the integration of multiple sources of data collected at various magnifications and by different imaging techniques. Using spatial information, we can build maps of tissue and cells in which it is possible to extract, e.g., measurements of cell morphology, protein expression, and gene expression. These measurements reveal knowledge about cells such as their identity, origin, density, structural organization, activity, and interactions with other cells and cell communities. Knowledge that can be correlated with survival and drug effectiveness. This thesis presents multidisciplinary projects that include a variety of methods for image and data analysis applied to images coming from fluorescence- and brightfield microscopy. In brightfield images, the number of proteins that can be observed in the same tissue section is limited. To overcome this, we identified protein expression coming from consecutive tissue sections and fused images using registration to quantify protein co-expression. Here, the main challenge was to build a framework handling very large images with a combination of rigid and non-rigid image registration. Using multiplex fluorescence microscopy techniques, many different molecular markers can be used in parallel, and here we approached the challenge to decipher cell classes based on marker combinations. We used ensembles of machine learning models to perform cell classification, both increasing performance over a single model and to get a measure of confidence of the predictions. We also used resulting cell classes and locations as input to a graph neural network to learn cell neighborhoods that may be correlated with disease. Finally, the work leading to this thesis included the creation of an interactive visualization tool, TissUUmaps. Whole slide tissue images are often enormous and can be associated with large numbers of data points, creating challenges which call for advanced

Microscopy imaging experiments generate vast amounts of data, and there is a high demand for smart acquisition and analysis methods. This is especially true for transmission electron microscopy (TEM) where terabytes of data are produced if imaging a full sample at high resolution, and analysis can take several hours. One way to tackle this issue is to collect a continuous stream of low resolution images whilst moving the sample under the microscope, and thereafter use this data to find the parts of the sample deemed most valuable for high-resolution imaging. However, such image streams are degraded by both motion blur and noise. Building on deep learning based approaches developed for deblurring videos of natural scenes we explore the opportunities and limitations of deblurring and denoising images captured from a fast image stream collected by a TEM microscope. We start from existing neural network architectures and make adjustments of convolution blocks and loss functions to better fit TEM data. We present deblurring results on two real datasets of images of kidney tissue and a calibration grid. Both datasets consist of low quality images from a fast image stream captured by moving the sample under the microscope, and the corresponding high quality images of the same region, captured after stopping the movement at each position to let all motion settle. We also explore the generalizability and overfitting on real and synthetically generated data. The quality of the restored images, evaluated both quantitatively and visually, show that using deep learning for image restoration of TEM live image streams has great potential but also comes with some limitations.

Multiplexed and spatially resolved single-cell analyses that intend to study tissue heterogeneity and cell organization invariably face as a first step the challenge of cell classification. Accuracy and reproducibility are important for the downstream process of counting cells, quantifying cell–cell interactions, and extracting information on disease-specific localized cell niches. Novel staining techniques make it possible to visualize and quantify large numbers of cell-specific molecular markers in parallel. However, due to variations in sample handling and artifacts from staining and scanning, cells of the same type may present different marker profiles both within and across samples. We address multiplexed immunofluorescence data from tissue microarrays of low-grade gliomas and present a methodology using two different machine learning architectures and features insensitive to illumination to perform cell classification. The fully automated cell classification provides a measure of confidence for the decision and requires a comparably small annotated data set for training, which can be created using freely available tools. Using the proposed method, we reached an accuracy of 83.1% on cell classification without the need for standardization of samples. Using our confidence measure, cells with low-confidence classifications could be excluded, pushing the classification accuracy to 94.5%. Next, we used the cell classification results to search for cell niches with an unsupervised learning approach based on graph neural networks. We show that the approach can re-detect specialized tissue niches in previously published data, and that our proposed cell classification leads to niche definitions that may be relevant for sub-groups of glioma, if applied to larger data sets.

Fluorescence microscopy, which visualizes cellular components with fluorescent stains, is an invaluable method in image cytometry. From these images various cellular features can be extracted. Together these features form phenotypes that can be used to determine effective drug therapies, such as those based on nanomedicines. Unfortunately, fluorescence microscopy is time-consuming, expensive, labour intensive, and toxic to the cells. Bright-field images lack these downsides but also lack the clear contrast of the cellular components and hence are difficult to use for downstream analysis. Generating the fluorescence images directly from bright-field images using virtual staining (also known as “label-free prediction” and “in-silico labeling”) can get the best of both worlds, but can be very challenging to do for poorly visible cellular structures in the bright-field images. To tackle this problem deep learning models were explored to learn the mapping between bright-field and fluorescence images for adipocyte cell images. The models were tailored for each imaging channel, paying particular attention to the various challenges in each case, and those with the highest fidelity in extracted cell-level features were selected. The solutions included utilizing privileged information for the nuclear channel, and using image gradient information and adversarial training for the lipids channel. The former resulted in better morphological and count features and the latter resulted in more faithfully captured defects in the lipids, which are key features required for downstream analysis of these channels.

Fluorescence microscopy, which visualizes cellular components with fluorescent stains, is an invaluable method in image cytometry. From these images various cellular features can be extracted. Together these features form phenotypes that can be used to determine effective drug therapies, such as those based on nanomedicines. Unfortunately, fluorescence microscopy is time-consuming, expensive, labour intensive, and toxic to the cells. Bright-field images lack these downsides but also lack the clear contrast of the cellular components and hence are difficult to use for downstream analysis. Generating the fluorescence images directly from bright-field images using virtual staining (also known as “label-free prediction” and “in-silico labeling”) can get the best of both worlds, but can be very challenging to do for poorly visible cellular structures in the bright-field images. To tackle this problem deep learning models were explored to learn the mapping between bright-field and fluorescence images for adipocyte cell images. The models were tailored for each imaging channel, paying particular attention to the various challenges in each case, and those with the highest fidelity in extracted cell-level features were selected. The solutions included utilizing privileged information for the nuclear channel, and using image gradient information and adversarial training for the lipids channel. The former resulted in better morphological and count features and the latter resulted in more faithfully captured defects in the lipids, which are key features required for downstream analysis of these channels.

With the increasing amount of image data collected from biomedical experiments there is an urgent need for smarter and more effective analysis methods. Many scientific questions require analysis of image subregions related to some specific biology. Finding such regions of interest (ROIs) at low resolution and limiting the data subjected to final quantification at high resolution can reduce computational requirements and save time. In this paper we propose a three-step pipeline: First, bounding boxes for ROIs are located at low resolution. Next, ROIs are subjected to semantic segmentation into sub-regions at mid-resolution. We also estimate the confidence of the segmented sub-regions. Finally, quantitative measurements are extracted at high resolution. We use deep learning for the first two steps in the pipeline and conformal prediction for confidence assessment. We show that limiting final quantitative analysis to sub regions with high confidence reduces noise and increases separability of observed biological effects.

Combining the information of different imaging modalities offers complimentary information about the properties of the imaged specimen. Often these modalities need to be captured by different machines, which requires that the resulting images need to be matched and registered in order to map the corresponding signals to each other. This can be a very challenging task due to the varying appearance of the specimen in different sensors. We have recently developed a method which uses contrastive learning to find representations of both modalities, such that the images of different modalities are mapped into the same representational space. The learnt representations (referred to as CoMIRs) are abstract and very similar with respect to a selected similarity measure. There are requirements which these representations need to fulfil for downstream tasks such as registration - e.g rotational equivariance or intensity similarity. We present a hyperparameter free modification of the contrastive loss, which is based on InfoNCE, to produce equivariant, dense-like image representations. These representations are similar enough to be considered in a common space, in which monomodal methods for registration can be exploited.

Prostate cancer is a common cancer type in men, yet some of its traits are still under-explored. One reason for this is high molecular and morphological heterogeneity. The purpose of this study was to develop a method to gain new insights into the connection between morphological changes and underlying molecular patterns. We used artificial intelligence (AI) to analyze the morphology of seven hematoxylin and eosin (H & E)-stained prostatectomy slides from a patient with multi-focal prostate cancer. We also paired the slides with spatially resolved expression for thousands of genes obtained by a novel spatial transcriptomics (ST) technique. As both spaces are highly dimensional, we focused on dimensionality reduction before seeking associations between them. Consequently, we extracted morphological features from H & E images using an ensemble of pre-trained convolutional neural networks and proposed a workflow for dimensionality reduction. To summarize the ST data into genetic profiles, we used a previously proposed factor analysis. We found that the regions were automatically defined, outlined by unsupervised clustering, associated with independent manual annotations, in some cases, finding further relevant subdivisions. The morphological patterns were also correlated with molecular profiles and could predict the spatial variation of individual genes. This novel approach enables flexible unsupervised studies relating morphological and genetic heterogeneity using AI to be carried out.

This chapter introduces the subject of computer-assisted analysis of multimodality image data and goes on to outline the areas covered in the remaining chapters of this section of the book.

In this chapter, we give a general view on computer assisted image processing in a multimodal context. We discuss the main challenges of the field, some popular approaches in addressing these challenges, and ongoing community efforts to facilitate communication, collaboration, and knowledge exchange in this highly multidisciplinary, and rapidly developing field.