Search

Fluorescence staining techniques, such as Cell Painting, together with fluorescence microscopy have proven invaluable for visualizing and quantifying the effects that drugs and other perturbations have on cultured cells. However, fluorescence microscopy is expensive, time-consuming, labor-intensive, and the stains applied can be cytotoxic, interfering with the activity under study. The simplest form of microscopy, brightfield microscopy, lacks these downsides, but the images produced have low contrast and the cellular compartments are difficult to discern. Nevertheless, by harnessing deep learning, these brightfield images may still be sufficient for various predictive purposes. In this study, we compared the predictive performance of models trained on fluorescence images to those trained on brightfield images for predicting the mechanism of action (MoA) of different drugs. We also extracted CellProfiler features from the fluorescence images and used them to benchmark the performance. Overall, we found comparable and largely correlated predictive performance for the two imaging modalities. This is promising for future studies of MoAs in time-lapse experiments for which using fluorescence images is problematic. Explorations based on explainable AI techniques also provided valuable insights regarding compounds that were better predicted by one modality over the other., De två första författarna delar förstaförfattarskapetDe två sista författarna delar sistaförfattarskapet

Microscopy imaging is a powerful technique when studying biology at a cellular and sub-cellular level. When combined with digital image analysis it creates an invaluable tool for investigating complex biological processes and phenomena. However, imaging at the cell and sub-cellular level tends to generate large amounts of data which can be difficult to analyse, navigate and store. Despite these difficulties, large data volumes mean more information content which is beneficial for computational methods like machine learning, especially deep learning. The union of microscopy imaging and deep learning thus provides numerous opportunities for advancing our scientific understanding and uncovering interesting and useful biological insights. The work in this thesis explores various means for optimising information extraction from microscopy data utilising image analysis with deep learning. The focus is on three different imaging modalities: bright-field; fluorescence; and transmission electron microscopy. Within these modalities different learning-based image analysis and processing techniques are explored, ranging from image classification and detection to image restoration and translation. The main contributions are: (i) a computational method for diagnosing oral and cervical cancer based on smear samples and bright-field microscopy; (ii) a hierarchical analysis of whole-slide tissue images from fluorescence microscopy and introducing a confidence based measure for pixel classifications; (iii) an image restoration model for motion-degraded images from transmission electron microscopy with an evaluation of model overfitting on underlying textures; and (iv) an image-to-image translation (virtual staining) of cell images from bright-field to fluorescence microscopy, optimised for biological feature relevance. A common theme underlying all the investigations in this thesis is that the evaluation of the methods used is in relation to the biological question at hand.

Fluorescence staining techniques, such as Cell Painting, together with fluorescence microscopy have proven invaluable for visualizing and quantifying the effects that drugs and other perturbations have on cultured cells. However, fluorescence microscopy is expensive, time-consuming, and labor-intensive, and the stains applied can be cytotoxic, interfering with the activity under study. The simplest form of microscopy, brightfield microscopy, lacks these downsides, but the images produced have low contrast and the cellular compartments are difficult to discern. Nevertheless, by harnessing deep learning, these brightfield images may still be sufficient for various predictive purposes. In this study, we compared the predictive performance of models trained on fluorescence images to those trained on brightfield images for predicting the mechanism of action (MoA) of different drugs. We also extracted CellProfiler features from the fluorescence images and used them to benchmark the performance. Overall, we found comparable and correlated predictive performance for the two imaging modalities. This is promising for future studies of MoAs in time-lapse experiments.

With the increasing amount of image data collected from biomedical experiments there is an urgent need for smarter and more effective analysis methods. Many scientific questions require analysis of image subregions related to some specific biology. Finding such regions of interest (ROIs) at low resolution and limiting the data subjected to final quantification at high resolution can reduce computational requirements and save time. In this paper we propose a three-step pipeline: First, bounding boxes for ROIs are located at low resolution. Next, ROIs are subjected to semantic segmentation into sub-regions at mid-resolution. We also estimate the confidence of the segmented sub-regions. Finally, quantitative measurements are extracted at high resolution. We use deep learning for the first two steps in the pipeline and conformal prediction for confidence assessment. We show that limiting final quantitative analysis to sub regions with high confidence reduces noise and increases separability of observed biological effects.

Fluorescence microscopy, which visualizes cellular components with fluorescent stains, is an invaluable method in image cytometry. From these images various cellular features can be extracted. Together these features form phenotypes that can be used to determine effective drug therapies, such as those based on nanomedicines. Unfortunately, fluorescence microscopy is time-consuming, expensive, labour intensive, and toxic to the cells. Bright-field images lack these downsides but also lack the clear contrast of the cellular components and hence are difficult to use for downstream analysis. Generating the fluorescence images directly from bright-field images using virtual staining (also known as “label-free prediction” and “in-silico labeling”) can get the best of both worlds, but can be very challenging to do for poorly visible cellular structures in the bright-field images. To tackle this problem deep learning models were explored to learn the mapping between bright-field and fluorescence images for adipocyte cell images. The models were tailored for each imaging channel, paying particular attention to the various challenges in each case, and those with the highest fidelity in extracted cell-level features were selected. The solutions included utilizing privileged information for the nuclear channel, and using image gradient information and adversarial training for the lipids channel. The former resulted in better morphological and count features and the latter resulted in more faithfully captured defects in the lipids, which are key features required for downstream analysis of these channels.

Background: Early prediction of time-lapse microscopy experiments enables intelligent data management and decision-making. Aim: Using time-lapse data of HepG2 cells exposed to lipid nanoparticles loaded with mRNA for expression of GFP, the authors hypothesized that it is possible to predict in advance whether a cell will express GFP. Methods: The first modeling approach used a convolutional neural network extracting per-cell features at early time points. These features were then combined and explored using either a long short-term memory network (approach 2) or time series feature extraction and gradient boosting machines (approach 3). Results: Accounting for the temporal dynamics significantly improved performance. Conclusion: The results highlight the benefit of accounting for temporal dynamics when studying drug delivery using high-content imaging.

Microscopy imaging experiments generate vast amounts of data, and there is a high demand for smart acquisition and analysis methods. This is especially true for transmission electron microscopy (TEM) where terabytes of data are produced if imaging a full sample at high resolution, and analysis can take several hours. One way to tackle this issue is to collect a continuous stream of low resolution images whilst moving the sample under the microscope, and thereafter use this data to find the parts of the sample deemed most valuable for high-resolution imaging. However, such image streams are degraded by both motion blur and noise. Building on deep learning based approaches developed for deblurring videos of natural scenes we explore the opportunities and limitations of deblurring and denoising images captured from a fast image stream collected by a TEM microscope. We start from existing neural network architectures and make adjustments of convolution blocks and loss functions to better fit TEM data. We present deblurring results on two real datasets of images of kidney tissue and a calibration grid. Both datasets consist of low quality images from a fast image stream captured by moving the sample under the microscope, and the corresponding high quality images of the same region, captured after stopping the movement at each position to let all motion settle. We also explore the generalizability and overfitting on real and synthetically generated data. The quality of the restored images, evaluated both quantitatively and visually, show that using deep learning for image restoration of TEM live image streams has great potential but also comes with some limitations.

eSSENCE

Background: The incidence of oral cancer is increasing and it is effecting younger individuals. PAP smear-based screening, visual, and automated, have been used for decades, to successfully decrease the incidence of cervical cancer. Can similar methods be used for oral cancer screening? We have carried out a pilot study using neural networks for classifying cells, both from cervical cancer and oral cancer patients. The results which were reported from a technical point of view at the 2017 IEEE International Conference on Computer Vision Workshop (ICCVW), were particularly interesting for the oral cancer cases, and we are currently collecting and analyzing samples from more patients. Methods: Samples were collected with a brush in the oral cavity and smeared on glass slides, stained, and prepared, according to standard PAP procedures. Images from the slides were digitized with a 0.35 micron pixel size, using focus stacks with 15 levels 0.4 micron apart. Between 245 and 2,123 cell nuclei were manually selected for analysis for each of 14 datasets, usually 2 datasets for each of the 6 cases, in total around 15,000 cells. A small region was cropped around each nucleus, and the best 2 adjacent focus layers in each direction were automatically found, thus creating images of 100x100x5 pixels. Nuclei were chosen with an aim to select well preserved free-lying cells, with no effort to specifically select diagnostic cells. We therefore had no ground truth on the cellular level, only on the patient level. Subsets of these images were used for training 2 sets of neural networks, created according to the ResNet and VGG architectures described in literature, to distinguish between cells from healthy persons, and those with precancerous lesions. The datasets were augmented through mirroring and 90 degrees rotations. The resulting networks were used to classify subsets of cells from different persons, than those in the training sets. This was repeated for a total of 5 folds. Result