Search

The field of systems biology has developed under the acknowledgement that quantitative approaches are necessary to fully understand the complexities that shape cells' structure and behaviour. The development of automated microscopy protocols has led to the accumulation of high-throughput data sets but the high resolution of modern imaging hardware compounded with time series generates an analysis bottleneck. I develop machine learning tools to track Saccharomyces cerevisiae cells in time lapses that are resilient experimental and technical noise. Then I coordinate cell segmentation and tracking software to develop software that automates analyses and makes them reproducible. I then study the protein aggregation dynamics of metabolic enzymes that form reversible aggregates via phase separation. Using single-cell information provides a deeper understanding of the impact of these proteins on yeast cells' growth and division. Heterogeneous responses, distinguishable by using mutant strains with varying aggregation phenotypes, provide several novel insights. This work shows that the development of computational tools can make the acquisition of biological insight faster and increases reproducibility while providing a toolset to tackle the challenge of dealing with high-content microscopy data - one without existing solutions able to scale computationally. it also furthers biological understanding of a conserved phenomenon such as reversible protein aggregation, in a way only possible by studying behaviour and physiology of thousands of cells at the same time. my thesis aims to pave the way for more comprehensive analyses using aggregated high-quality data.

3D printing has become a substantial part of the engineering toolbox. The technology offers vast opportunities across fields because it can make almost any geometry out of almost any material. However, many applications remain at the research-stage because printers are vulnerable to errors. Currently, expert operators must recognise errors and repeat the print with manually updated settings. Existing error detection solutions often only work for a single machine, material, geometry, and setup, whilst primarily focussing on the detection of a single error modality. Furthermore, little work exists on how to correct these errors once manifested. This thesis presents a number of deep learning enabled computer vision based approaches to detect and correct a wide range of errors both in real time and for the subsequent print. The developed models outperform existing approaches at error detection in performance and generality, whilst further advancing the state-of-the-art by autonomously controlling and updating printing parameters to correct errors. However, the training of these modern deep learning architectures requires significant quantities of high quality training data. Therefore, a unique data collection and labelling engine is developed to generate process monitoring data from a fleet of 3D printers. With this tool, three large datasets are created and used to trained deep learning models for different applications: (i) generalisable real-time error detection and multi-parameter correction; (ii) real-time quantitative prediction of flow rate and few-shot correction; and (iii) recognition and correction of long-term thermal deformation. These models are then used for parameter discovery to autonomously learn how to print unseen and novel materials. Additionally, ideas from explainable AI are introduced to create visualisations which shed light on how the deep learning models make their decisions.

Birds fly with apparent ease, remaining in control during manoeuvres despite large variations in wind conditions. They do this reliably, repeatably and efficiently. This thesis looks at the flight control of two species of bird, the lesser black-backed gull (Larus fuscus) and the red kite (Milvus milvus). Rotational stereo videography, a 3D multi-point tracking technique, is employed in the field to study the flight control of wild members of these species in free flight. The motion of the bird's wings and tail was related to the measured flight path to estimate flight control parameters, giving an indication of the control strategies being implemented by the birds. By examining the equivalent control laws for fixed wing aircraft this work highlights the features of the birds' flight control strategy which might enhance the performance of a gliding bird when compared to a traditional fixed wing aircraft with discreet trailing edge control surfaces. Looking firstly at longitudinal control of flight path angle in gliding flight, it was found that the gulls used fore and aft movement, adjusting the effective sweep of their wings to control their longitudinal flight path angle when in steady glide. This result is interesting as it highlights the decoupling of pitch angle from longitudinal flight path control as is typically found in rigid body fixed wing vehicles. Secondly when making gliding turns the gulls kept their tails furled and used bank angle to adjust their turning radius much in the same way that rigid body fixed wing aircraft do. It was found that the equations of motion for rigid body fixed wing turning mechanics can be applied to model the gulls turning performance as in this case the tail played little roll in controlling turning flight. Conversely a very different result was seen in the turning performance of the red kites. These birds make active use of a widely spread and comparatively large set of tail feather to enhance their control in turning flight. In a straight glide the tail was seen to twist up to ± 30º to help the bird to maintain its desired ground track and in steeper turns the tail was pro-versely deflected into the direction of the turn to enhance the effective amount of pro-turn force and to increase turn rate and reduce turn radius for a given bank angle. Turns at a lower bank angle made more use of this tail twist than turns at higher bank angles where a small change in wing angle has a larger effect. The study of the birds inspired three novel control strategies: wing sweep for pitch control, wing twist for direct lift control, and wing rise/flap for variable lateral-directional stability. These three control schemes are implemented on a representative small un[1]manned aerial vehicle focusing on control about the principal axes using an articulating main wing, with freedom to rotate about the wing root. Wind tunnel testing and computational modelling using a vortex lattice method were used to study the flight dynamics and control potential of these strategies. Wing sweep for longitudinal flight path control was found to be highly effective, particularly at higher angles of attack. This effective weight shift changes the moment arm between the centre of pressure and the centre of mass and can generate large pitching moments without making large changes to the angle of attack, as such dynamic manoeuvres such as end over end tumbles are achievable. Wing twist for roll control was found to be no more effective than properly sized ailerons, with the upgoing wing being pushed close to its critical angle and in extreme cases stalling and inducing control reversal. Symmetric wing twist for direct lift control has some transient benefits but effectively changes the longitudinal trim position of the fuselage as the wing adopts a new setting angle, its effect was of limited benefit. Finally, being able to dynamically vary the wings dihedral/anhedral angle in flight can profoundly affect the lateral direction stability of the vehicle. High dihedral angles stabilise the spiral mode and excite the Dutch roll mode and might be beneficial in environments where the vehicle would be subject to strong and variable lateral gusts. To conclude, birds fly quite differently from three-axis, rigid body fixed wing vehicles and some elements of directly controlling the main wing may be beneficial for small unmanned aerial vehicles that demand high levels of agility performance in unsteady flow fields. Having articulated wings and tails extends the flight envelope of 'fixed wing' vehicles, traditional design rules become less applicable and new performance metrics and control concepts are required to fully exploit the benefits over a fixed wing design.

Computer vision systems are implemented in many applications and require high tests performance, especially in applications linked to safety. A key component in developing any system is the training dataset, which is the set of images used by the system to learn how to detect the desired objects. When images are collected for training datasets it is often seen that little to no thought is put into how the conditions under which the images are taken can affect the appearance of features, and therefore affect training and test performance. This can result in many training datasets causing sub-optimal test performance because they include images taken under undesirable conditions. Images can be affected by many factors, for example the environmental conditions related to lighting. Datasets using synthetic images can be affected by simulation factors, for example the rendering engine used. As little research exists into the effects of different factors, for both real-world and synthetic datasets, there is little understanding into how factors affect test performance, how they may interact, and how the effects and interactions can be identified through a robust process. The thesis explores and demonstrates the importance of considering the effects on test performance, whilst establishing a process which can be used to identify the effects and interactions between various factors. This is taken further by investigating how the information can be used to make/identify an optimised training dataset. The thesis shows this through three hypothetical application based case studies which use the convolutional neural networks. Case study 1 concentrates on detecting construction machines to investigate the initial process used for identifying the effects and interactions of environmental factors in a real-world dataset, and simulation factors in a synthetic dataset. Case study 2 focuses on general litter items to investigate any commonalities between different applications. Furthermore, the synthetic dataset uses more factors to establish how the process needs to adapt as the number of factors being investigated increases. Case study 3 uses LEGO® to create objects of varying complexity, which enables investigations into how the effect of factors change as the complexity of the objects in a scene also changes. Each of the case studies provides numerous insights into how the factors influence the test performance and defines an overarching process on how to gather images and investigate the effects and interactions for use in other applications.

Multi-light imaging captures a scene from a fixed viewpoint through multiple photographs, each of which are illuminated from a different direction. Every image reveals information about the surface, with the intensity reflected from each point being measured for all lighting directions. The images captured are known as multi-light image collections (MLICs), for which a variety of techniques have been developed over recent decades to acquire information from the images. These techniques include shape from shading, photometric stereo and reflectance transformation imaging (RTI). Pixel coordinates from one image in a MLIC will correspond to exactly the same position on the surface across all images in the MLIC since the camera does not move. We assess the relevant literature to the methods presented in this thesis in chapter 1 and describe different types of reflections and surface types, as well as explaining the multi-light imaging process. In chapter 2 we present a novel automated RTI method which requires no calibration equipment (i.e. shiny reference spheres or 3D printed structures as other methods require) and automatically computes the lighting direction and compensates for non-uniform illumination. Then in chapter 3 we describe our novel MLIC method termed Remote Extraction of Latent Fingerprints (RELF) which segments each multi-light imaging photograph into superpixels (small groups of pixels) and uses a neural network classifier to determine whether or not the superpixel contains fingerprint. The RELF algorithm then mosaics these superpixels which are classified as fingerprint together in order to obtain a complete latent print image, entirely contactlessly. In chapter 4 we detail our work with the Metropolitan Police Service (MPS) UK, who described to us with their needs and requirements which helped us to create a prototype RELF imaging device which is now being tested by MPS officers who are validating the quality of the latent prints extracted using our technique. In chapter 5 we then further developed our multi-light imaging latent fingerprint technique to extract latent prints from curved surfaces and automatically correct for surface curvature distortions. We have a patent pending for this method.

In this thesis we investigate the use of deep learning for cross-modality and multi-modal image-based profiling applications. In particular, we explore the utility of the brightfield image modality with deep generative models, and also propose new methods to integrate metadata labels freely obtained in high-content screening into deep learning architectures. The use of automated microscopy in high-content phenotypic screening of cells treated with compounds or genetic perturbations produces a large amount of high-dimensional data. One example which we focus on in this thesis is Cell Painting, a standardized pipeline used to capture rich cell morphology. Typically, fluorescent stained images are the focus of image-based profiling, where the aim is to extract meaningful features from the images which can be used to represent the biology of the cells, and compare the effects of the treatments. Image-based profiling is central to screening in drug discovery, and is used to guide the selection of drug candidates to take to clinical trials. With an abundance of large databases of high-dimensional images, deep learning for image-based profiling has risen as its own sub-field, and there are now entire drug discovery pipelines selecting drugs to take to clinical trials built upon deep learning foundations. This is possible due to advances in deep learning in computer vision. In this thesis we explore and adapt recent and powerful deep learning approaches including the generative adversarial network, self-supervised vision transformers, and the diffusion denoising probabilistic model. A major challenge in this space is to use state-of-the-art frameworks from computer vision in a way which is sensitive to the challenges of drug discovery. Image-based profiling, and particularly deep learning in image-based profiling, is a new and maturing field. In this thesis we tackle the unaddressed challenge of incorporating the cheaper, easier to obtain but classically less informative brightfield modality, as well as using underutilised but freely available metadata alongside the images to guide the training of deep neural networks. As we are working in interdisciplinary science, we do this while making models which are visually interpretable to the many people working in drug discovery who are not familiar with, or potentially sceptical of deep learning. This thesis presents the first study to predict all five fluorescent Cell Painting channels from brightfield images. We explore the potential, benefits and limitations of this new approach. Next, we introduce a weakly-self supervised learning framework to learn feature representations which are guided by informative metadata. Finally, we present the first study to use a diffusion model with high-content microscopy images. We generate entire plates of synthetic Cell Painting images of exceptional image quality to make predictions about the information these models are capable of capturing, and investigate if this can also be guided by labels.

This study aimed to automate detecting animals in aerial images and improve detection by combining computer vision techniques with statistical modelling of the surveyed area. Knowing the number of animals in an area is important for wildlife management and existing methods require trained observers in larger planes or photography from smaller aircraft requiring manual counting. Automating detection would allow large areas to be surveyed more frequently with lower human input. This thesis shows that animals can be automatically detected from aerial images using the YOLO object detection network. Studies ruled out classical computer vision techniques due to excess false positives. The YOLO method detects 61% of the animals compared to 79% detection by humans, however it also detects 11.6 False Positives Per Image (FPPI). Modelling the distribution of multiple species required a multinomial model. CReSS based GAMs were extended to the multinomial case and simulation studies were carried out to compare CReSS to other multinomial approaches, showing CReSS was preferred for: high noise, low sample sizes or animal densities close to exclusion zones. Confidence intervals from the statistical model were concatenated with the YOLO model. This reduced the FPPI from 11.6 to 6.6, showing that combining prior knowledge from a statistical model improves performance of animal detection. Manual checking time per image was reduced by 97%, from 5 minutes to 11 seconds. Using the automated detections to guide manual checks spotted additional animals increasing the recall to 0.81, greater than the recall estimated for human performance of 0.79. The methods described have reduced the estimated manual checking time for the 40,000 photographs covering the 7,500km2 survey area in Namibia from 9 months to 3 weeks, meaning this method could be used frequently to give timely and reliable results.

Research in the biological and wildlife sciences is increasingly reliant on video data for measuring animal behaviour, however large-scale analysis is often limited by the time and resources it takes to process video archives. Computer vision holds serious potential to unlock these datasets to analyse behaviour at an unprecedented level of scale, depth and reliability, however thus far a framework for processing and analysing behaviour from large-scale video datasets is lacking. This thesis attempts to solve this problem by developing the theory and methods for capturing long-term sociality of animal populations from longitudinal video archives, laying the foundations for an emerging field; computational ethology of animals in the wild. It makes several key contributions by a) establishing the first unified longitudinal video dataset of wild chimpanzee stone tool use across a 30 year period, and building a framework for collaborative research using cloud-technology b) developing a set of computational tools to allow for processing of large volumes of video data for automated individual identification and behaviour recognition c) applying these automated methods to validate use for social network analysis and d) measuring the social dynamics and behaviour of a group of wild chimpanzees living in the forest of Bossou, Guinea, West Africa. In Chapter 1 I introduce the theoretical and historical context for the thesis, and outline the novel methodological framework for using computer vision to measure animal social behaviour in video. In Chapter 2 I introduce the methodology for processing and managing a longitudinal video archive, and future directions for a new framework for collaborative research workflows in the wildlife sciences using cloud technology. In Chapter 3 I lay the foundations of this framework for analysing behaviour and unlocking video datasets, using deep learning and face recognition. In Chapter 4 I evaluate the robustness of the method for modelling long-term sociality and social networks at Bossou and test whether life history variables predict individual-level sociality patterns. In Chapter 5 I introduce the final component to this framework for measuring long-term animal behaviour, through audiovisual behavioural recognition of chimpanzee nut-cracking. In my final chapter (6) I discuss the main contributions, limitations and future directions for research. Overall this thesis integrates a diverse range of interdisciplinary methods and concepts from primatology, ethology, engineering, and computer vision, to build the foundations for further exploration of cognition, ecology and evolution in wild animals using automated methods.

Heterogeneous catalyst materials play a key role in modern society, as many processes in the chemical and energy industry rely on them. Optimising their performance is directly connected to a large potential of reductions in energy consumption, and thus to a more sustainable future. A fundamental part in the optimisation process is represented by materials characterisation. This is often done using in situ (Scanning) Transmission Electron Microscopy ((S)TEM) imaging in order to obtain a full understanding of the catalyst performance in different environments and temperatures at high resolution. However, the analysis of corresponding dynamical datasets is often time-consuming and requires manual intervention alongside tailored post-processing routines. At the same time, the emergence of direct electron detectors allowing for the acquisition of datasets at kiloHertz frame rates, as well as novel imaging techniques raised data generation rates significantly and created a need for new, reliable and automated data processing techniques. This work introduces 'nNPipe' as Deep Learning based method for the automated analysis of morphologically diverse heterogeneous catalyst systems. The method is based on two Covolutional Neural Networks (CNNs) that were exclusively trained on computationally generated HRTEM image simulations and allow for rapid and precise analysis of raw 2048 x 2048 experimental HRTEM images. The performance of 'nNPipe' is demonstrated in a realistic automated imaging scenario where statistically significant material properties are inferred accurately. Moreover, time-efficient and reproducible retraining methods based small experimental datasets are described for both, further performance improvements and adaption to new imaging scenarios. In this context, a potentially new pathway for the generation of suitable training datasets obtained by thousands of mostly non-expert annotations is highlighted. Finally, the analytical capacities of the 'nNPipe' method are showcased on time-resolved datasets in two advanced applications scenarios: i) Live image analysis during sample acquisition and ii) Analysis of the particle coalescence of an in situ heated Pd/C catalyst.

Dense prediction is the task of predicting a label for each pixel in the image. Given 3D data (point clouds or RGB-D images) as input, dense prediction can also be extended to 3D space and assign each 3D point/location a label. According to the label type, dense prediction can be mainly categorized as depth estimation, motion prediction, segmentation, and other related tasks. There are four major challenges for learning dense predictions: i) how to significantly improve the accuracy and resolve the ambiguous regions, ii) high memory and computational costs, iii) the dependency on a large amount of labeled data for training, and iv) the poor cross-domain generalization to novel datasets. This integrated thesis focuses on dense prediction tasks, from correspondence estimation (stereo matching and optical flow) to 2D/3D semantic segmentation. Seven robust deep neural network models are proposed to achieve state-of-the-art accuracy, to realize effective training with just synthetic data or unlabeled real data, and to boost the cross-domain generalization to various unseen datasets. For the first task, traditional 3D geometry constraints are embedded into end-to-end trainable stereo matching networks to achieve state-of-the-art accuracy on two stereo matching benchmarks (by publication date). Based on this work, a domain-invariant stereo matching network is proposed. It is trained on the synthetic data but outperforms many models fine-tuned on real data. For the second task, a Separable Flow network is developed for optical flow estimation, which ranks the first on two standard optical flow benchmarks (by the time of publication). It's also one of the best methods for predicting optical flow on various unseen datasets. Moreover, research is also conducted on unsupervised pre-training and domain adaptation for semantic image segmentation. Finally, the 2D image segmentation knowledge is further leveraged for tackling 3D segmentation. The proposed 3D segmentation networks achieve the leading position on large-scale point-cloud segmentation benchmarks (at the time of publication).

Light-responsive particle-stabilised (Pickering) emulsions can, in principle, be selectively emulsified or demulsified on-demand through the remote application of light. However, despite their wide-ranging potential in applications such as drug delivery and biphasic catalysis, their rational design is extremely challenging and there are very few examples to date. In this thesis, we investigate a model system based on silica particles functionalised with azobenzene-derived photoswitches to understand the key factors that determine the characteristics of light-responsive Pickering emulsions. We address their design on three distinct length scales: the photoswitch at the molecular level, the photoswitchable particle at the sub-micron scale, and light-responsive emulsion at the tens of micron scale. Key to the creation of light-responsive Pickering emulsions is the design of photo-switchable molecules that can impart light-responsive behaviour. These molecules are attached to the particle-stabilisers and upon switching change the hydrophobicity of the particle, resulting in a change in the emulsion's stability. Two groups of photoswitches are investigated in this work, which belong to the azobenzene and arylazopyrazole families. When irradiated with either UV or blue light, these molecules isomerise from a more hydrophobic 'trans' state to a less hydrophobic 'cis' state. Both groups are derivatised with hydrophobic or hydrophilic terminal modifications and by the addition of a carbon chain spacer. The arylazopyrazoles are also inherently more hydrophilic. Investigation of the optical properties of both families show successful photoswitching and, for arylazopyrazoles, an exceptionally high photostationary state (>90 %) and half-lives as long as 24 days. The photoswitchable molecules are then appended to fumed silica particle at three different grafting concentrations. The effect of the grafting density and type of photoswitch on the hydrophobicity of the particle is first analysed by surface energy analysis, before and after 'trans-cis' photoisomerisation. It is found that the length of the carbon spacer is the most important factor in controlling particle hydrophobicity and the azobenzene-modified particles show a greater difference in hydrophobicity when isomerised compared to the arylazopyrazoles. Emulsions are then produced using oils of different polarity and their stability and morphology assessed by optical microscopy. A computer-vision application is produced to help extract droplet size information from microscopy images using the circle Hough-transform. This method is benchmarked against the most commonly used alternative, a region growing technique, and it is found to have higher accuracy, recall and precision. The light-responsive behaviour of the emulsions is also assessed and, for the first time, a reversible transition between emulsified water-in-oil droplets and demulsified water and oil phases is observed with the application of either UV or blue light, which can be repeatedly cycled. Using the observed trends and data from the surface energy analysis of the particles, a set of design rules are presented which will help facilitate the rational design and, therefore, more widespread application of light-responsive Pickering emulsions.

Giving machines the ability to precisely perceive and understand the 3D visual world is the fundamental step to allow them to interact competently within our physical world. However, the research on large-scale 3D scene understanding and perception is still in its infancy, due to the complex geometrical structure of 3D shapes and limited high-quality data resources. Among various 3D representations, point clouds have attracted increasing attention due to its flexibility, compactness, and the characteristic of being close to raw sensory data. Despite this, the semantic understanding of large-scale 3D point clouds remains a challenge due to its orderless, unstructured, and non-uniform proper-ties. To this end, this thesis makes three core contributions from a high-quality urban-scale dataset, and then fully-supervised semantic understanding to weakly supervised label efficient learning of large-scale 3D point clouds. The main contributions of this thesis are three-fold. In Chapter 3, we start by building an urban-scale photogrammetric point cloud dataset with nearly three billion richly annotated points. We also identify a number of open and unique challenges faced by urban-scale 3D scene understanding and conduct comprehensive experimental analysis to address these challenges. This work points out the limitations of existing algorithms and provides several insightful conclusions for understanding large-scale 3D point clouds in urban environments. With high-quality large-scale 3D datasets, we further investigate the research problem of efficient semantic segmentation of large-scale 3D point clouds in Chapter 4. We begin by analyzing the strengths and weaknesses of existing downsampling strategies and find that random sampling is a suitable component for efficient learning on large-scale point clouds. Additionally, we propose a local feature aggregation module to increase the receptive field hierarchically and preserve important features. We then build an efficient, lightweight neural architecture called RandLA-Net that can directly infer per-point semantics for large-scale point clouds with millions of points that span hundreds of meters. In Chapter 5, we take a step further to investigate the label-efficient learning of large-scale 3D point clouds. That is, achieving high-quality semantic segmentation with limited annotations. We first examine key issues in weakly supervised learning of 3D point clouds, including different weak supervision schemes and the critical point of weak annotations. Through a pilot study, we find that dense 3D annotations are actually redundant and unnecessary. Motivated by this, we propose a novel weak supervision framework that implicitly augments the total amount of available supervision signals by leveraging the semantic homogeneity between neighboring points. This is achieved using a point neighborhood query to allow the sparse training signals to be back-propagated to a wider context. All of the algorithms and datasets presented in this thesis have been open-source on GitHub to facilitate future research. The RandLA-Net algorithm was recognized as one of the most influential papers at the CVPR 2020 conference and has been integrated into code libraries such as Open3D and TorchPoints3D. The SensatUrban dataset was used as the platform for two Urban3D challenges, contributing to the advancement of semantic understanding of city-scale point clouds. Overall, this thesis presents a high-quality dataset and two novel data-driven algorithms, aiming to achieve efficient, scalable, and effective learning-based semantic understanding of large-scale 3D point clouds, eventually improving the real-time 3D perception capacity of intelligent machines in practice.

The aspiration of computer vision is to enable computers to make sense of a world, one that exists in a vast sea of colorful pixels reflecting what humans see every day. Central to this goal perhaps, is the discovery of object representations from these pixels: clusters of pixels representing the things from big to small that constitute our physical world and the interactions with which form our daily life. Visual object segmentation is a subject that deals with this task. It concerns the prediction of pixel-wise masks that delineate objects of interest from visual data. Different types of evidence have been leveraged for segmenting out objects (according to the application), e.g., contours, saliency, motion, extreme points, and pre-defined semantic categories, among others. In this thesis, we investigate the exploitation of temporal correspondences and natural language descriptions for visual object segmentation, motivated by their omnipresence and efficacy, and contribute new methods that improve the state of the art for the relevant applications. In the first part of the thesis, we consider the modeling of temporal dependencies for object segmentation in videos, where the continuous movement of an object offers natural correspondences in time. Conventional methods based on sequential modeling are ineffective at exploiting such cues, and they still perform on a par with static segmentation models. In this thesis, we address this issue by introducing a technique that learns dense correspondences between pixels of frames that can be arbitrarily far apart in time. This enables the learning of long-term dependencies without conditioning on intermediate frames, which significantly improves the (deep) representation consistency of the object over time and leads to accurate object segmentation in videos. In the second part, we explore the use of natural language for object segmentation in images, where an object that corresponds to a natural language description is segmented out. With the aim of aligning linguistic meanings and visual clues in a common feature space, previous methods adopt the paradigm of fusing visual and linguistic features after they have been independently extracted from a vision encoder and a language encoder, respectively. In this thesis, we introduce a new paradigm that relocates cross-modal feature alignment to an earlier phase, i.e., to be conducted jointly with image encoding. The main idea is to supplement linguistic features to multiple intermediate layers of a vision Transformer network, to have linguistic information embedded jointly with visual information by the Transformer layers during the forward pass. With this approach, we tap into the correlation modeling power of the Transformer model at an early stage, and can harvest segmentation with a lightweight mask predictor. In the last part, we investigate the joint utilization of temporal and linguistic cues for visual object segmentation in the task of video object segmentation from referring expressions. We show how appearance, language, and motion features can be simultaneously aligned in a single network that is end-to-end trainable by exploiting the hierarchical structure of convolutional neural networks. Compared with previous methods, the proposed method has a much-enhanced capability in representing multi-modal inputs at different semantic and spatial granularity levels while enjoying a much simpler design.

RGB-D cameras are a type of imaging device that includes a standard colour camera and a depth sensor. The depth sensor records depth images, where each pixel measures the distance of the target from the camera. Using data from the RGB-D camera, we can recon- struct a three-dimensional scene. Since the introduction of the Microsoft Kinect, multiple affordable RGB-D cameras have reached the market and are even integrated into laptops and mobile phones. In this thesis, healthcare applications of RGB-D cameras are investigated, focussing on two areas. The first is Structured Light Plethysmography (SLP), a method for measuring lung function. The non-contact nature of SLP is ideal for respiratory measurement, as it avoids potential contamination without requiring single-use equipment. The current SLP device is large and expensive, while the data it produces is low-resolution and does not adapt to the patient. A new SLP processing pipeline based on an RGB-D camera is proposed, which will produce automatically aligned, high-resolution data in a portable device. Algorithm development to enhance the data processing capabilities of this new pipeline is presented. A method for interpreting data from the high-resolution grid, via a control point-based shape modelling technique, is used to show differences in the respiratory pat- tern before and after exercise. A system is described for compensating for the motion of the subject, so the resulting data only shows motion due to breathing. This method is demonstrated by compensating for the pedalling motion of a subject on an exercise bike. The second application area is the Neonatal Intensive Care Unit (NICU), which cares for critically ill new-born babies. A proof-of-concept study is designed and conducted to investigate the potential use of an RGB-D camera in the NICU. In this thesis, the collec- tion of 24 hours of RGB-D video footage is presented, and it is shown how the infant's respiratory rate can be measured from the depth data.

Recognising and understanding the complex visual world is the ultimate goal of intelligent vision systems. Computer vision and artificial intelligence have been a long lasting research hotspot with increasing major discoveries and breakthroughs. Human action understanding is one of the crucial topics due to its potential value in both academia and industry. Various steep challenges remain due to semantically implicit and ambiguous definitions of video events and their inherent signal complexities from streamed videos ill-affected by target occlusion and variation of illumination conditions. Classic strategies and techniques for addressing these critical challenges of human action understanding have been investigated in this research. An innovative machine learning-augmented analytical framework for visual behaviour understanding has been proposed. The corresponding operational pipeline first integrates the discrete wavelet transform technique into the dense trajectory model to gain more defining human action features. Then the end-to-end multimodality neural networks are deployed for automatic feature learning and action classification. Performance enhancement has been achieved through the innovation of an efficient two-stream aggregation network by adopting optical flow-guided features and spatial-temporal fusion blocks in a cascaded spatial and temporal space. This research has also addressed the context-biased problem causing long aggravation to the deep-learning community when dealing with generalisation issues. A long-short-term motion encoding scheme is presented to interpret human actions based on their semantic meanings embedded in pose skeletons, which has greatly alleviated the open-set action recognition problem by introducing the Euclidean and Additive Angular Margin Loss. To facilitate the real-world implementation of the devised human action understanding models and techniques, the state-of-the-art and future trends of edge computing have also been explored. Corresponding experiments have demonstrated the viability and effectiveness of open format-based transferrable model generation for rapid and mass deployment in live.

Humans can easily understand their surroundings, analyze the situation, and even figure out the intentions of others, all in a few seconds. However, computers see the surroundings as an array of numbers and we need to write algorithms for them to make a sense of it. It is a huge challenge but significant strides have been made toward the scene understanding problem over the last decade, thanks to an increase in computing infrastructure and datasets. However, several challenges remain in making computers more 'intelligent', some of which are tackled in this thesis. This thesis develops different techniques to create more reliable systems and proposes innovations in loss functions by analyzing the system's prediction distributions. First, we try to address the well-known problem of mode collapse in Generative Adversarial Networks (GANs) and propose MAD-GAN, an intuitive generalization of the GANs. MAD-GAN is a multi-agent GAN architecture incorporating multiple generators and one discriminator. To enforce that different generators capture diverse high probability modes, the objective function of the discriminator is designed such that along with identifying the real and fake samples, it is also required to identify the generator that generated the sample if it is fake. Intuitively, to succeed in this task, the discriminator must learn to push different generators toward different identifiable modes. Extensive experiments on synthetic and real datasets demonstrate the effectiveness of the MAD-GAN approach. We then consider the task of semantic segmentation using weak supervision with bounding box annotations. Bounding boxes are noisy labels for the foreground objects. To focus on foreground objects, our approach predicts a per-class attention map that saliently guides the per-pixel cross-entropy loss and refines the segmentation boundaries. This avoids propagating erroneous gradients due to the incorrect foreground labels. Additionally, our approach learns pixel embeddings to simultaneously optimize for high intra-class feature affinity while increasing discrimination between features across different classes. This helps in capturing global context via long-range pairwise interactions. Qualitative and quantitative results along with ablation studies show the benefit of different loss terms on the overall performance. The widespread usage of neural networks is particularly dependent on their reliability and trustworthiness. We therefore also consider the important task of calibrating deep neural networks - where a model's confidence should be the same as its correctness. Ideally, we want networks to be accurate, calibrated, and confident. We provide a thorough analysis of the factors causing miscalibration and use the obtained insights to justify the empirically excellent performance of focal loss. We show that, as opposed to the standard cross-entropy loss, focal loss allows us to learn models that are already very well-calibrated. When combined with temperature scaling, whilst preserving accuracy, focal loss yields state-of-the-art calibrated models. To facilitate its use in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. Extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, justify that this approach achieves state-of-the-art calibration without compromising accuracy in almost all cases. One-shot video object segmentation is an important problem in computer vision with many interesting real-world applications. To tackle this, we finally introduce a similarity learning approach, that can learn to perform dense label transfer from one image to the other. More specifically, the objective is to learn a similarity metric for dense pixel-wise correspondence between two images. This learned model can then be used in a label transfer framework to propagate object annotation from the reference frame to all the subsequent frames in a video. Unlike previous methods, our similarity learning approach works fairly well across various domains, even when no domain adaptation is involved. We demonstrate the effectiveness of our method on two standard datasets with favourable results. This approach not only provides good object segmentation but is also time-efficient. Therefore, using it we also achieved second place in the first DAVIS challenge for interactive video object segmentation, in both quality and speed tracks.

Videos are an appealing source of data to train computer vision models. There exist almost infinite supplies of videos online, but exhaustive manual annotation is infeasible. The goal of this thesis is to learn strong video representations efficiently via self-supervised learning: a method that learns from the data rather than human annotations. The thesis is structured around three themes: (1) self-supervised learning for short-term videos, (2) efficient video representation learning, and (3) self- supervised learning for long-term videos. For short-term videos lasting only a few seconds, we show that predicting the video in the future is a strong learning signal at a large scale. We further show that strong video representations can be learned by taking two complementary modalities, namely RGB and optical flow, and using them to teach each other. For efficient video representation learning, we show that large-scale pre-trained vision-language models can be effectively adapted via a prompt tuning technique. We also show that dropping image patches can accelerate the finetuning of classification tasks and pre-training of video-language models. For long-term videos that last longer than a few minutes, we show that temporal alignment networks can be trained from the weak visual-textual correspondence within instructional videos. The resulting networks can automatically clean up the natural videos for effective vision-language training. In addition, we show that movie description models can be trained by leveraging the pre-trained vision- language models.

Although deep neural networks (DNNs) attain excellent performance on the specific tasks they are trained for, this often seems to be obtained using easier-to-learn proxies for the truly relevant concepts. The problem with proxies is that they cannot be relied on in new situations - the proxy departs from the true concept. And DNNs will very likely be deployed in new situations because the world is always changing, and training data are never exhaustive. Unfortunately, existing approaches are limited in their ability to diagnose the bad proxies being relied on by a DNN. We also cannot accurately characterise a model's generalisation performance on different kinds of data beyond its training task. These limitations additionally prevent us from developing systems that do not rely on bad proxies. To improve this situation, this thesis introduces two new test generation procedures for image classification DNNs. These improve on existing approaches by identifying more kinds of inputs for which a particular DNN gives incorrect outputs. This is achieved by exploiting generative machine learning to solve the test oracle problem in new ways. The first new procedure trains a generative network to directly output test cases that identify failures. The second dynamically perturbs the activation values of a pretrained generative network as it generates new examples - the perturbations adjust the features of the generated data so that they also induce failures in the DNN being evaluated. Besides the primary contribution of these algorithms, this thesis also presents an empirical finding: standard adversarial training that aims to increase model robustness surprisingly decreases DNNs' ability to generalise correctly to changes in high-level features such as object position, orientation, shape or colour.

This thesis aims to take the first step in developing a quantitative analysis workflow for characterising histological changes in pre-implantation renal biopsies. Solving this is highly beneficial as there is a significant disparity between organ supply and demand worldwide. It can also help to alleviate the workload caused by a shortage of pathologists. This work would allow quantitative measurement of histological features related to chronic and acute injuries in pre-implantation biopsies. These features can then be linked to scores given by pathologists, post-transplant clinical endpoints, and proteomic biomarkers to help researchers better understand the mechanisms of organ injury and repair. First, we build the main feature extraction workflow based on convolutional neural networks (CNN) to automatically segment different tissue compartments in renal biopsies. We design handcrafted features and qualitatively show that some of these features can capture visual changes related to chronic and acute injuries. Then, we find that many tissue compartments in the primary dataset have artefacts not present in the training data. We discuss Bayesian Neural Networks (BNNs) to quantify uncertainties in segmentation and propose a way to improve histological estimates. Next, we propose using tissue compartment-based features to improve classification and interpretability in a multi-instance learning setting. In addition to learned attention, tissue compartments are weighted by segmentation quality to reduce the impact of instances that bear little resemblance to the training data. Finally, we combine techniques developed in this project to detect acute injuries from biopsies. We propose an algorithmic score consisting of models trained on pathologists' assessments and donors' clinical states. The score is significantly associated with Delayed Graft Function (DGF) in the test data. These findings suggest that our workflow can facilitate research in kidney transplant where pathologist time is limited. It also shows histology has great potential in predicting post-transplant complications.

We live in a physical 3D world. Being able to perceive the world in 3D allows us to confidently navigate and perform sophisticated actions in it. Developing 3D perception systems is not only key to many AR and robotics applications, but also fundamental to the goal of visual understanding. However, most existing learning-based image understanding models process images simply as compositions of 2D patterns, ignoring the fact that images arise from a physical 3D world. Recent advances in differentiable rendering have opened new possibilities for learning physically-grounded disentangled 3D representations from raw images, marrying geometry with learning at last. Learning 3D representations typically requires 3D supervision. This has previously come in the form of 3D labels, shape models, multi-views, or 2D geometric annotations, such as keypoints, masks and camera viewpoints. However, all of these are prohibitively expensive to collect on a large scale, which has become a major obstacle for learning 3D representations of general objects in the wild. In this thesis, we explore the possibility of learning deformable 3D objects simply from raw casually recorded 2D image data "in the wild", such as online photos and YouTube videos, without relying on heavy manual supervision. At the core of our approach is a Photo-Geometric Autoencoding framework that decomposes images into a set of physically-grounded photometric and geometric factors, including 3D shape, pose, shiny surface material, environment lighting and camera parameters, with the objective of recomposing them to reconstruct the input images through a differentiable renderer. In order to resolve the ambiguity in this highly ill-posed task, we propose to inject generic inductive biases to exploit the symmetries and regularities of the world. The resulting model is able to learn disentangled 3D priors of various deformable object categories, allowing for single-image inference at test time.

During the early years of facial expression research, works have mostly employed macro expressions which are easily identifiable. In contrast, in recent years utilizing facial micro expression has gained more acknowledgement in facial analysis due to stronger genuineness of its attributes. Subsequently, emotion analysis through facial micro expression has higher acceptability especially in psychology, autism, pain assessment, security, criminal investigations, and similar circumstances that demand critical decision making. Owing to its cross-discipline application, today micro expression analysis using facial images remains an active research field. Due to extreme minuteness of these expressions, they are often missed during observations however, studies show with the introduction of computer vison and machine/deep learning algorithms they have a higher chance of being identified. Therefore, the focus of this thesis is to conduct thorough investigations and design novel approaches for micro expression analysis employing suitable methods. Most of the existing literature has overlooked the phase information while describing image patterns specifically to achieve micro expression recognition. Consequently, this thesis investigates the effectiveness of employing phase information for micro expression analysis. Furthermore, interpolation and video magnification are also introduced in later experiments to aid the extraction method. Additionally, the literature also highlighted the absence of adequate work examining the impact of resolutions and image quality for micro expression analysis. Therefore, the aim of this thesis is to explore micro expression to design a pipeline capable of boosting the expression recognition performance. Moreover, this thesis establishes threefold contributions to address the research gap: firstly, a pipeline that exploits interpolation, phase and temporal information in a non-cross database environment is utilized. Secondly, influence of video magnification is examined to improve expression recognition within this pipeline. Third, a novel pipeline to employ low quality micro expression images is developed by reconstructing such images using deep learning and generative adversarial networks. In this thesis, to verify the suitability of combining phase, temporal, and magnification methods for micro expression, experiments are conducted on seven spontaneous micro expression databases. Results obtained clearly indicate the approach is as competitive as any other existing traditional methods. Furthermore, the experimental results obtained after introducing deep learning and generative adversarial networks into the second novel pipeline clearly highlight the significance of image reconstruction in achieving recognition boost even when the quality of input is compromised. Therefore, this thesis establishes significant progress towards the development of techniques for micro expression recognition that can be collaborated with medical/security and similar fields to assist in identifying vital cues.

This dissertation develops a new modelling system based on deep learning for the study and design of street network configurations. It aims to fill an existing gap in the use of deep learning in this field, through hypothesising that convolutional neural network (CNN) and generative adversarial network (GAN) models can make a revolutionary breakthrough (like in many geometry and image-intensive fields) if a modelling system is designed to suit the nature of street network configurations and the processes for designing them. This new modelling system consists of three components. First, a thirteen-channel urban fabric image generation module, UrbanFramer, that provides a new way to represent the physical urban fabric as image patches. By integrating datasets of high-resolution 3D built form, road network data and topography of the landform, this module generates image patches as inputs for urban fabric classification and street network generation. Secondly, an urban fabric classification module, UrbanClassifier, which takes the image patches from UrbanFramer and classifies them into morphology types using a CNN. The novelty is characterised by a unique approach of assimilating urban fabric features across multiple spatial scales and viewpoints. Furthermore, a transfer learning process in this module provides a semi-automatic way to expand and incorporate specialist planning knowledge when labelling the image patches for deep learning-based street network generation. Thirdly, a street network design module, StreetGEN, which combines human and machine intelligence to support city-specific, contextual street network configuration design. The core of the machine intelligence consists of two GAN models, PlanStreet and TopoStreet, which learn from existing street network configuration samples and provide near-real-time design suggestions that incorporate, respectively, human guidance and local topographic conditions. Users are able to progressively and iteratively input their domain knowledge and creativity in street network design. StreetGEN turns the existing one-way, learning-based street network image generation into iterative human-computer interaction. The above modules have been put through systematic tests using purpose-built datasets created by UrbanFramer in two groups of case study areas: Amsterdam, Barcelona, Berlin and Prague in the development of UrbanClassifier, PlanStreet, and Siena, Perugia, Rome and Florence in the development of TopoStreet. The case study tests produce four key findings regarding the capabilities and potential of deep learning. First, equipped with multiple spatial scales and viewpoints, UrbanClassifier has very significantly improved the predictive performance, outperforming the hitherto state-of-art benchmark CNN model for image classification on all three urban fabric classification tasks by 0.14, 0.22 and 0.24 in terms of F1-Score. Secondly, through transfer learning, UrbanClassifier has performed urban fabric classification for new cities with much reduced needs of manual labelling for training. When transferring to a new city, UrbanClassifier only requires 25% of labelled data from a new city to attain what is achieved by it directly trained on full data. In other words, with UrbanClassifier, manual labelling required for classifying new cities can be significantly reduced. The expanded dataset, in turn, offers the potential to build an ever-growing database of real-world street network configurations for deep learning. Thirdly, by introducing a modest amount of human guidance on road junction locations and pattern types, PlanStreet has shown to be far more capable of reproducing the ground truth street networks than in an automatic-only prediction mode. Taking less than 40% of the ground truth junction information as input guidance, PlanStreet is able to predict street configurations at the same level of precision as that of existing learning-based models with 100% of the ground truth junction information. Fourthly, by introducing local topography, TopoStreet has outperformed benchmark models for cities in hilly areas, by sharply narrowing the prediction performance differences between flat and hilly areas, from 17.02% to 3.56% in Absolute Percentage Error for Length-Weighted Metric Choice. In summary, the design and use of this modelling system have proven that CNN and GAN models can make a real contribution to the geometrical modelling and design of street networks. This fills an existing gap in this field. Furthermore, the modelling system has designed and tested a human-machine interaction process that enables professionals and laypersons to test, near real-time, alternative designs that follow and respect local planning and topography contexts. This opens up practical applications of deep learning in the field of street network configuration design.

Humans navigate in their environment by learning a mental model of the world through passive observation and active interaction. Their world model allows them to anticipate what might happen next and act accordingly with respect to an underlying objective. Such world models hold strong promises for planning in complex environments like in autonomous driving. A human driver, or a self-driving system, perceives their surroundings with their eyes or their cameras. They infer an internal representation of the world which should: (i) have spatial memory (e.g. occlusions), (ii) fill partially observable or noisy inputs (e.g. when blinded by sunlight), and (iii) be able to reason about unobservable events probabilistically (e.g. predict different possible futures). They are embodied intelligent agents that can predict, plan, and act in the physical world through their world model. In this thesis we present a general framework to train a world model and a policy, parameterised by deep neural networks, from camera observations and expert demonstrations. We leverage important computer vision concepts such as geometry, semantics, and motion to scale world models to complex urban driving scenes. In our framework, we derive the probabilistic model of this active inference setting where the goal is to infer the latent dynamics that explain the observations and actions of the active agent. We optimise the lower bound of the log evidence by ensuring the model predicts accurate reconstructions as well as plausible actions and transitions. First, we propose a model that predicts important quantities in computer vision: depth, semantic segmentation, and optical flow. We then use 3D geometry as an inductive bias to operate in the bird's eye view space. We present for the first time a model that can predict probabilistic future trajectories of dynamic agents in bird's-eye view from 360◦ surround monocular cameras only. Finally, we demonstrate the benefits of learning a world model in closed-loop driving. Our model can jointly predict static scene, dynamic scene, and ego-behaviour in an urban driving environment. We show that the learned world model and driving policy can generate predictions of over 1 hour (2000x longer than the training sequence size).

Scene representation is the process of converting sensory observations of an environment into compact descriptions. Such intelligent behavior is a cornerstone of artificial intelligence. Scientists have long sought to reproduce the extraordinary ability of humans for understanding the physical environment. Taking visual sensory observations of an environment as input, modern intelligent systems mostly aim to learn neural representations that encode fundamental scene properties such as geometry and semantics. Such representations can be leveraged to support other downstream tasks and finally realize autonomous perception and interaction within the complex 3D world. Impressive performance has been exhibited in recent deep neural networks that excel at modeling geometric and semantic information in neural scene representations. However, constructing robust systems is still highly challenging due to the fragility in uncontrolled real-world scenarios. This presents significant complexity for scene representation learning due to the variance of sensory observations with respect to scene changes, the domain gap among different types of visual representations, and the requirement for efficient perception for multiple categories of information. To overcome these challenges, we pursue robust, unified and informative scene representations, learning with geometry and semantics from different types of visual inputs, paving the way towards intelligent machines that autonomously learn to understand the world around them. In this context, this thesis makes three core contributions in the fields of visual localization, pixel-point matching, and semantic surface reconstruction. In this thesis, we start by estimating the 6 Degree-of-Freedom (DoF) camera pose from single images. To learn scene representations that are robust to environmental changes and sensor operation, a neural network together with a self-attention module is proposed to model complex geometry relationships from which given images are taken relative to a reference environment. Then, we build a more general framework to find unified representations across 2D images and 3D point clouds based on the inherent constraints from epipolar geometry and stereo vision. By introducing an ultra-wide reception mechanism together with novel loss functions, a dual fully-convolutional framework is presented that maps 2D and 3D inputs into a shared latent representation space to simultaneously describe and detect keypoints, bridging the gap between 2D and 3D representations. Finally, we extend our study to develop informative representations, which are generally required for intelligent systems to operate in real-world scenarios for multiple purposes at the same time. By drawing on previous work in point-based networks, we introduce a brand-new end-to-end neural implicit function that can jointly estimate precise 3D surface and semantics from raw and large-scale point clouds. Overall, this thesis develops a series of novel deep neural frameworks to advance the field of machine learning of scene representations towards artificial intelligence that can fully perceive our real-world 3D environment.

Empowering machines to understand compositionality is considered by many (Lake et al., 2017; Lake and Baroni, 2018; Schölkopf et al., 2021) a promising path towards improved representational interpretability and out-of-distribution generalization. Yet, discovering the compositional structures of raw sensory data requires solving a factorization problem, i.e. decomposing the unstructured observations into modular components. Handling the factorization problem presents numerous technical challenges, especially in unsupervised settings which we explore to avoid the heavy burden of human annotation. In this thesis, we approach the factorization problem from a generative perspective. Specifically, we develop unsupervised machine learning models to recover the compositional data-generation mechanisms around objects from visual scene observations. First, we present MulMON as the first feasible unsupervised solution to the multi-view object-centric representation learning problem. MulMON resolves the spatial ambiguities arising from single-image observations of static scenes, e.g. optical illusions and occlusion, with a multi-view inference design. We demonstrate that not only can MulMON perform better scene object factorization with less uncertainty than single-view methods, but it can also predict a scene's appearances and object segmentations for novel viewpoints. Next, we present a technique, namely for latent duplicate suppression (abbr. LDS), and demonstrate its effectiveness in fixing a common scene object factorization issue that exists in various unsupervised object-centric learning models---i.e. inferring duplicate representations for the same objects. Finally, we present DyMON as the first unsupervised learner that can recover object-centric compositional generative mechanism from moving-view-dynamic-scene observational data. We demonstrate that not only can DyMON factorize dynamic scenes in terms of objects, but it can also factorize the entangled effects of observer motions and object dynamics that function independently. Furthermore, we demonstrate that DyMON can predict a scene's appearances and segmentations at arbitrary times (querying across time) and from arbitrary viewpoints (querying across space)---i.e. answer counterfactual questions. The scene modeling explored in this thesis is a proof of concept, which we hope will inspire: 1) a broader range of downstream applications (e.g. "world modelling'' and environment interactions) and 2) generative factorization research that targets more complex compositional structures (e.g. complex textures, multi-granularity compositions).

Recently, Deep learning (DL), which involves powerful black box predictors, has outperformed human experts in several medical diagnostic problems. However, these methods focus exclusively on improving the accuracy of point predictions without assessing their outputs' quality and ignore the asymmetric cost involved in different types of misclassification errors. Neural networks also do not deliver confidence in predictions and suffer from over and under confidence, i.e. are not well calibrated. Knowing how much confidence there is in a prediction is essential for gaining clinicians' trust in the technology. Calibrated uncertainty quantification is a challenging problem as no ground truth is available. To address this, we make two observations: (i) cost-sensitive deep neural networks with Dropweights models better quantify calibrated predictive uncertainty, and (ii) estimated uncertainty with point predictions in Deep Ensembles Bayesian Neural Networks with DropWeights can lead to a more informed decision and improve prediction quality. This dissertation focuses on quantifying uncertainty using concepts from cost-sensitive neural networks, calibration of confidence, and Dropweights ensemble method. First, we show how to improve predictive uncertainty by deep ensembles of neural networks with Dropweights learning an approximate distribution over its weights in medical image segmentation and its application in active learning. Second, we use the Jackknife resampling technique to correct bias in quantified uncertainty in image classification and propose metrics to measure uncertainty performance. The third part of the thesis is motivated by the discrepancy between the model predictive error and the objective in quantified uncertainty when costs for misclassification errors or unbalanced datasets are asymmetric. We develop cost-sensitive modifications of the neural networks in disease detection and propose metrics to measure the quality of quantified uncertainty. Finally, we leverage an adaptive binning strategy to measure uncertainty calibration error that directly corresponds to estimated uncertainty performance and address problematic evaluation methods. We evaluate the effectiveness of the tools on nuclei images segmentation, multi-class Brain MRI image classification, multi-level cell type-specific protein expression prediction in ImmunoHistoChemistry (IHC) images and cost-sensitive classification for Covid-19 detection from X-Rays and CT image dataset. Our approach is thoroughly validated by measuring the quality of uncertainty. It produces an equally good or better result and paves the way for the future that addresses the practical problems at the intersection of deep learning and Bayesian decision theory. In conclusion, our study highlights the opportunities and challenges of the application of estimated uncertainty in deep learning models of medical images, representing the confidence of the model's prediction, and the uncertainty quality metrics show a significant improvement when using Deep Ensembles Bayesian Neural Networks with DropWeights.

Prostate cancer is a disease affecting growing numbers of people, which can form metastases and cause death. The risk of mortality and complications from prostate cancer is assessed by visual inspection of tumour morphology in Haematoxylin & Eosin-stained tissue samples. The morphology of prostate cancer is highly heterogeneous: during tumorigenesis, the normal glands that traverse and branch through the prostate lose their structural organisation, and develop a range of aberrant morphological characteristics, such as smaller size, larger nuclei and loss of the layer of basal epithelial cells found in normal glands. Depending on the level of structural loss, and of dedifferentiation of the glands, tissue samples are assigned a Gleason Score from 4 to 10. A higher Gleason Score is indicative of increased clinical risk for patients, and it guides the decision on what type of treatment patients should receive. The Gleason Score suffers from considerable shortcomings, including a poor reproducibility that is due to its qualitative definition. In this work, we design an algorithm to extract a quantitative representation of the morphology of prostate cancer, based on the segmentation and morphological measurement of prostate glands. We validate the use of the representation by showing it automatically captures clinically relevant morphological classes, it is predictive of the Gleason Score, and it can be used to characterise different tissue specimen types used in prostate cancer diagnosis. The morphology of glands is used to diagnose prostate cancer. In some cases, the morphology may appear abnormal, but not to an extent to make a confident cancer diagnosis possible. Immunohistochemistry is then used to detect the lack of basal cells in glands, which is indicative of prostate cancer. We design an automated tool for advance requesting immunohistochemistry for such cases of ambiguous prostate morphology, and we estimate the temporal and monetary savings of introducing the tool into the clinical workflow.

Decades of research have shed light on some of the computational elements that enable the extraordinary visual capabilities of primates, yet the brain's solution to the binding problem remains unresolved. In particular, how are the relations of separately encoded features represented in a meaningful way? More recently, machine vision has experienced significant advances with the proliferation of deep-learning techniques, although robust object recognition comparable to primate vision remains out of reach. We propose that these two unresolved goals may be related, leveraging recent theoretical work in computational neuroscience to improve the robustness of machine vision systems. Similarly, we adopt concepts from robust computer vision systems and high-level computational neuroscience models to explore a possible biological implementation of binding at the neural level. We begin by implementing a network where grid cells are employed as a basis for encoding the spatial relations of features in vision, enabling rapid feature binding for novel sensory information. The utility of this system is then demonstrated on a series of challenging machine vision tasks, including object recognition under saccade-like visual inputs, demonstrating superior performance to traditional machine learning approaches. Further analyses explore the importance of spatial information for the network, and secondary benefits of the architecture such as robustness under continual learning, and the availability of predictive representations. Our results emphasize that this form of binding can serve as a basis for designing enhanced machine vision systems, and may represent computations in the primate brain that account for our current superiority over artificial systems on a variety of visual tasks. Subsequently, machine vision systems designed for robustness, along with a high-level computational neuroscience model, form the basis for exploring how biological neurons might implement learned feature binding to support the recognition of objects. In particular, we establish the biologically and computationally constrained settings under which a combination of axonal conduction delays, coincidence detection, and Spike-Timing-Dependent Plasticity (STDP) may be sufficient to enable the emergence of object-selective neurons that are both translation invariant and form-specific in their responses. Our combined analysis and simulation results suggest that such a computation represents a feasible, albeit potentially specialized, operation that the primate brain could leverage for robust object recognition. Finally, we explore the phenomenon of adversarial examples, suggesting that these may partly be a consequence of the low-dimensional object representations typically assumed in machine vision systems. Leveraging recent developments in theoretical neuroscience, we present a novel deep-learning architecture which approximates the notion of hierarchical binding. Augmented with representations designed to better capture the rich variation of natural objects, the architecture is shown to cause a variety of adversarial attack methods to fail. Additional analyses provide evidence that these results are related to the introduced hierarchical binding, and its influence on the geometry of the network's representations, providing a novel approach for the design of more robust machine vision systems.

With the global population constantly increasing, including the population of elderly people, there is an increasing need for facilitating independent and comfortable living. The degree of at-home monitoring that is necessary to deliver sufficient independence can be provided using automated sensor technology built within smart environments. Sensors can be used to monitor a home by analysing the activities of the environment's inhabitant. It is important, however, to consider the preservation of privacy and so the unobtrusiveness of such a system should be considered as vital a characteristic as its accuracy. This thesis proposes an unobtrusive approach for detecting and recognising Activities of Daily Living (ADLs) performed within a smart environment. The research that is presented involved the use of low-resolution thermal sensors as the only means of data capture. The sensors have been used to develop original datasets for training and testing the various components of the proposed approach. The four primary components of the proposed approach are presented as four complementary chapters in this thesis. Firstly, an unobtrusive approach to recognising full body poses with thermal imagery is presented. Several machine learning algorithms were tested and their performances are compared. The second study introduces the concept of subactivities and how they can be inferred from the thermal data using only the predicted pose and the object estimated to be closest to the inhabitant. Additional sensors are considered in the third study where the effect of deep learning is investigated in order to improve upon the pose recognition performance. Lastly, the various components are combined to present the approach to detecting and recognising ADLs in a manner which exhibits both accuracy and privacy.

Techniques in computer vision have evolved over the years and seen breakthroughs in the recent past with fully data-driven approaches such as deep neural networks. Although these approaches have shown impressive capabilities when detecting objects in images, they suffer from several shortcomings such as high data dependency and lack of transparency. This is a problem when dealing with applications where data is scarce or transparency is critical to build trust in the decision making process. For example, vision applications in healthcare or self-driving technology, where the direct impact on human life is high. The aim of this thesis is to study the role of background knowledge when overcoming these limitations in neural network-based vision models. I focus on ontologies to be the source of background knowledge in the investigations because of their superior capabilities in ensuring the consistency of information and their ability to infer new information from existing information. The downstream task of choice during the experimentation is few-shot image classification which is used to evaluate the influence of background knowledge when classifying visual objects with a few examples. I propose a framework that integrates ontology-based background knowledge with a vision model which has two major components: (1) Concept embeddings that are learnt by capturing symbolic knowledge from an ontology in a continuous vector space. This study investigates methods to represent different properties of an ontology with embeddings. It further designs and applies techniques to measure their quality when representing the knowledge. (2) A vision model that is guided by the learnt embeddings during the training and inference stages. Experiments are carried out to evaluate the informed vision models with several few-shot image classification benchmarks, where they achieve superior performance compared to existing approaches. The improvement on few-shot learning capabilities of the vision models achieved through the integration of background knowledge manifests a way to overcome the challenge of high data dependency. Moreover, I argue that the use of learnt concept embeddings enhances the transparency of the vision model behaviour as the distribution of the extracted image features is decided by the embedding space. To this end, I further introduce a framework to measure the degree of error during predictions based on the background knowledge used. This study also discusses the design and construction of suitable ontologies based on the image labels of datasets used for the vision tasks.

Few-shot learning aims to scale visual recognition to open-ended growth of new classes with limited labelled examples, thus alleviating data and computation bottleneck of conventional deep learning. This thesis proposes a meta learning (a.k.a. learning to learn), paradigm to tackle the real-world few shot learning challenges. Firstly, we present a parameterized multi-metric based meta learning algorithm (RelationNet2). Existing metric learning algorithms are always based on training a global deep embedding and metric to support image similarity matching, but we propose a deep comparison network comprised of embedding and relation modules learning multiple non-linear distance metrics based on different levels of features simultaneously. Furthermore, images are represented as \todo{a} distribution rather than vectors via learning parameterized Gaussian noise regularization, reducing overfitting and enable the use of deeper embeddings. We next consider the fact that several recent competitors develop effective few-shot learners through strong conventional representations in combination with very simple classifiers, questioning whether "meta-learning" is necessary or highly effective features are sufficient. To defend meta-learning, we take an approach agnostic to the off-the-shelf features, and focus exclusively on meta-learning the final classifier layer. Specifically, we introduce MetaQDA, a Bayesian meta-learning extension of quadratic discriminant analysis classifier, that is complementary to advances in feature representations, leading to high accuracy and state-of-the-art uncertainty calibration performance in predictions. Finally, we investigate the extension of MetaQDA to more generalized real-world scenarios beyond the narrow standard few-shot benchmarks. Our model achieves both many-shot and few-shot classification accuracy in generalized few-shot learning. In terms of few-shot class-incremental learning, MetaQDA is inherently suitable to novel classes growing \todo{scenarios}. As for open-set recognition, we calculate the probability belonging to novel class by Bayes' Rule, maintaining high accuracy in both close-set recognition and open-set rejection. Overall, our contributions in few-shot meta-learning advance state of the art under both accuracy and calibration metrics, explore a series of increasingly realistic problem settings, to support more researchers and practitioners in future exploration.

Advanced Gas Cooled Reactors provide the backbone of stable low carbon nuclear energy within the United Kingdom. However, accumulation of radiation damage over time leads to a degradation of their graphite cores and a phenomenon known as keyway root cracking. In the event of an earthquake these reactors must be able to shut down safely through the insertion of control rods down core channels. The seismic safety case for these reactors is currently provided through numerical modelling using the GCORE finite element software that uses the explicit dynamic solver LS-DYNA (later ABAQUS). In order to validate the results from GCORE an experimental model was produced at the University of Bristol. This dissertation analyses the key results from the experimental configurations to probe the reactor behaviour under an earthquake scenario. Techniques are also developed to acquire detailed responses of the core, including the key behaviour, from videos taken during the tests. These methodologies are used to investigate the behaviour of a bearing key in a cracked core channel. A numerical model of the experiment, using the discrete element software 3DEC, is used to probe the physics of the core such as forces and allows rapid changes of configurations that are not practical to physically test. Advanced visualisation of the large datasets produced by the experiment is explored using virtual reality modelling language. The control column displacements are then studied for the intact core and for 9 other configurations with damage going up to levels where 50% of the fuel bricks are cracked in layers 4-7 for an 8 layer model. From this analysis the main drivers for worsening core seismic performance are seen to be increased global cracking percentage, increased seismic amplitude and broken layer spanning keys. These results may be used to direct the configurations that GCORE explores for a conservative safety case.

Electronics industry is one of the fastest evolving, innovative, and most competitive industries. In order to meet the high consumption demands on electronics components, quality standards of the products must be well-maintained. Automatic optical inspection (AOI) is one of the nondestructive techniques used in quality inspection of various products. This technique is considered robust and can replace human inspectors who are subjected to dull and fatigue in performing inspection tasks. A fully automated optical inspection system consists of hardware and software setups. Hardware setup include image sensor and illumination settings and is responsible to acquire the digital image, while the software part implements an inspection algorithm to extract the features of the acquired images and classify them into defected and non-defected based on the user requirements. This research aims to explore methods used in literature for quality and AOI of the basic building block of electronic and optoeletronic devices which are semiconductor and optoelectronic wafers. Deep Learning (DL) techniques has proven its ability in extracting the feature automatically within their hidden layers saving the export knowledge and efforts when compared to conventional image processing and feature extraction techniques. DL can also preform classification based on the examples learned by network, while conventional rulebased classification uses conditional statements to provide a decision. Based on the previous advantages, DL networks were used to be as a decision making system for classifying defect patterns and anomalies in semiconductor and optoelectronic wafers. Open source WM-811K were used in our investigation of semiconductor wafer bin maps (WBMs) patterns. It was observed that the dataset is imbalanced and some classes do not contain enough samples for training, therefore, Deep Convolutional Generative Adversarial Network (DCGAN) were used to upsample and increase the dataset. A novel improved capsule network WaferCaps were also proposed to classify the defect patterns according to eight classes. The performance of our proposed DCGAN and WaferCaps was compared with different deep learning models such as the original Capsule Network (CapsNet), Convolutional Neural Network (CNN), and Multi-layer Perceptron (MLP). In all of our experiment, WM-811K dataset was used for the data upsampling and training. The proposed approach has shown an effective performance in generating new synthetic data and classify them with training accuracy of 99.59%, validation accuracy of 97.53% and test accuracy of 91.4%. For analysing optoelectronic wafer, we focused our attention on the waveguide of quantum cascade laser (QCL) in the wafer. Due to the lack of sufficient dirt and defect samples, data augmentation approach was used to increase the number of images. Decision fusion approach was used to integrate the classification performance of CNN and WaferCaps since they performed the best in different classes. The proposed approach was compared to similar DL algorithms and it achieved an overall accuracy of 98.5%.

Masonry arch bridges have proven to be durable and underpin much of the world's transport infrastructure. However, they are an aging asset, and their effective management is challenging. Existing management focuses heavily on manual inspection, which has been shown as subjective and often leads to incomplete records regarding damage and retrofit of the structure. Improved data collection and processing technologies now provide the opportunity to create a digital record of the entire bridge surface. However, if this digital record is still inspected manually, subjectivity will persist. While considerable progress has been made in automated image-based defect detection of concrete and asphalt infrastructure, relatively little progress has been made for masonry. In this context, a new framework, or pipeline, for automated inspection of masonry arch bridges is presented: from the data capture phase all the way to the diagnosis of the underlying problems with the bridge. After presenting this framework, the focus of this work is on the automated detection of defects within realistic, deteriorated masonry surfaces that are typical of historic masonry arch bridges. This has involved the creation of a pixel-wise annotated dataset of masonry arch bridge surfaces for both different defect classes and mortar joints. This dataset is believed to be unparalleled in both scope and scale, compared to other works in the literature and therefore serves as an invaluable tool for future research in this area. Methodologies for mapping the mortar joints on masonry surfaces have been examined, including a comparison of a hardcoded deterministic algorithm based on pattern detection, with a semantic deep learning model. Both methods performed well; the deep learning model was more robust to noisier image conditions. The effect of mortar joints on automated defect detection procedures was then investigated, as these joints are one of the key distractors and differentiators between masonry and concrete/asphalt surfaces. Whilst simple models based on edge detection were unable to differentiate defects from the masonry surface without prior segmentation of mortar, a more robust classifier based on a Convolutional Neural Network architecture performed well. In addition, prior segmentation of mortar in the image is shown to have no impact on the classifier performance. Use of Class Activation Mapping shows that in most cases the classifier has learnt to ignore the mortar joint interface. The developed classifier performed equal to or better than current manual inspection procedures.

The challenge considered in this thesis is that of domesticating deep learning-based vision. Firstly, we advance the use of deep learning for indoor architectural comprehension. Secondly, we explore a taming of deep neural networks for low-cost inference such that they can be used in real-time or on constrained hardware. Room or building architecture forms the foundational level of geometry that will be visible within an indoor space, and remains static. Therefore, better understanding of architectural structure enables reasoning over a long period of time without re-training or re-mapping. We introduce Planar R-CNN which can perform unconstrained room layout estimation. Planar R-CNN detects instances of architectural planes, as well as regresses the 3D plane equation for each detection. Planar R-CNN moves room layout estimation beyond strict constraints and can infer arbitrary planar geometries. Following this, we explore visual localisation through architectural reasoning. We introduce LaLaLoc, which learns a latent representation of layout shared between rendered structure and panorama images, enabling cross-modal alignment without explicit layout prediction. LaLaLoc is able to localise across furniture configurations and in previously unvisited scenes given only a floor plan structure as a prior. LaLaLoc++ extends this by unifying representations of layout structure between 2D floor plans and images. Neural network based methods can often be too slow to meet the requirements of a mobile robot, or too large to run on its hardware. We consider low-cost network design, and in particular group convolution as a design tool. Grouped convolution has become increasingly common across the spectrum of speed and accuracy trade-offs architecture design. However, the implications of grouped convolutional layers have remained relatively unexplored. We introduce Mobile R-CNN which, through group convolutional design, performs highly accurate real-time instance segmentation. Finally, we formalise this design of grouped convolutional architectures with the GroSS representation of group convolution blocks, which ultimately enables thorough exploration of the design space where it may not have been feasible before.

Time plays an integral role in video understanding, from disambiguating gestures like "swipe left" from "swipe right" to being the dimension over which activity unfolds. It is thus of great importance to leverage information across time in an effective manner to build successful systems for video understanding. By exploring time from three different perspectives, this thesis sheds light on which parts of video are informative, how label symmetries can be used to reduce the need for so much training data, and how activities can be explicitly modeled in the representation of video. Whilst symmetries in data have been well exploited for data augmentation purposes, the relationships among the labels used to supervise models have seen less attention. Both temporal and spatial relationships between fine-grained action labels are explored for the purposes of data augmentation and zero-shot learning. The dominant approach for applying deep learning methods to video for recognition has been to sample a sequence of frames at regular intervals as input. However, not all frames are made equal; some contain less information than others or are redundant to frames already sampled. This thesis presents a method used to quantify the importance of frames from the perspective of a trained model. A variety of models have been proposed for video recognition that model temporal relationships in different ways, but direct comparison has often been difficult due to differences in evaluation protocol. A benchmark study of these common models on the EPIC-KITCHENS dataset under a common evaluation protocol is presented to assess the relative merits of each one. One of the challenges faced in the video understanding community is the difficulty of scaling models up to longer video. A novel representation of video is proposed that explicitly models observed activities within a video up to a point in time to enable efficient processing of subsequent video. Additionally, a self-supervised pre-training procedure is introduced to bootstrap the model from long unlabelled videos.

Despite the rapid progress in the field of artificial intelligence, there are still important new areas to be explored and existing methods enhanced to make machines think like humans. This thesis conducts research in four machine learning and computer vision areas in this direction. First, we study what makes some images more memorable than others and propose a new machine learning method to learn and predict image memorability, closely matching human performance. A spatial attention function is learnt to localize image regions responsible for the image retention in memory. To identify meaningful temporal segments in a video stream, we study episodic segmentation in our memory and design a novel algorithm for video summarization to mimic human capabilities. A soft, self-attention method without a recurrent network is used to learn frame importance scores for the video summarization. This simple algorithm demonstrates a performance superior to the current state-of-the-art methods. Inspired by our brain's ability to project high dimensional visual information to computationally efficient, meaningful representations, we propose a method for latent binary representations learning and methods for operations in this discrete latent space such as interpolation, novel image generation, and attribute modification outperforming more complex published methods. To advance methods targeting catastrophic interference, one of the most fundamental problems of artificial neural networks, we study elementary neural mechanisms mitigating this phenomenon in our brain's memory. Building on our insights on the function of pattern separation in the hippocampus, we propose a conceptually simple and resource-efficient method to learn high dimensional sparse binary representations for continual learning. By performing elementary binary operations or and and over a continual stream of sparse representations of novel classes, our method exhibits performance significantly exceeding the current state-of-the-art meta-learning methods on identical benchmarks.

Relative navigation is paramount in space missions that involve rendezvousing between two spacecraft. It demands accurate and continuous estimation of the six degree-of-freedom relative pose, as this stage involves close-proximity-fast-reaction operations that can last up to five orbits. This has been routinely achieved thanks to active sensors such as lidar, but their large size, cost, power and limited operational range remain a stumbling block for en masse on-board integration. With the onset of faster processing units, lighter and cheaper passive optical sensors are emerging as the suitable alternative for autonomous rendezvous in combination with computer vision algorithms. Current vision-based solutions, however, are limited by adverse illumination conditions such as solar glare, shadowing, and eclipse. These effects are exacerbated when the target does not hold cooperative markers to accommodate the estimation process and is incapable of controlling its rotational state. This thesis explores novel model-based methods that exploit sequences of monoc ular images acquired by an on-board camera to accurately carry out spacecraft relative pose estimation for non-cooperative close-range rendezvous with a known artificial target. The proposed solutions tackle the current challenges of imaging in the visible spectrum and investigate the contribution of the long wavelength infrared (or "thermal") band towards a combined multimodal approach. As part of the research, a visible-thermal synthetic dataset of a rendezvous approach with the defunct satellite Envisat is generated from the ground up using a realistic orbital camera simulator. From the rendered trajectories, the performance of several state-of-the-art feature detectors and descriptors is first evaluated for both modalities in a tailored scenario for short and wide baseline image processing transforms. Multiple combinations, including the pairing of algorithms with their non-native counterparts, are tested. Computational runtimes are assessed in an embedded hardware board. From the insight gained, a method to estimate the pose on the visible band is derived from minimising geometric constraints between online local point and edge contour features matched to keyframes generated offline from a 3D model of the target. The combination of both feature types is demonstrated to achieve a pose solution for a tumbling target using a sparse set of training images, bypassing the need for hardware-accelerated real-time renderings of the model. The proposed algorithm is then augmented with an extended Kalman filter which processes each feature-induced minimisation output as individual pseudo measurements, fusing them to estimate the relative pose and velocity states at each time-step. Both the minimisation and filtering are established using Lie group formalisms, allowing for the covariance of the solution computed by the former to be automatically incorporated as measurement noise in the latter, providing an automatic weighing of each feature type directly related to the quality of the matches. The predicted states are then used to search for new feature matches in the subsequent time-step. Furthermore, a method to derive a coarse viewpoint estimate to initialise the nominal algorithm is developed based on probabilistic modelling of the target's shape. The robustness of the complete approach is demonstrated for several synthetic and laboratory test cases involving two types of target undergoing extreme illumination conditions. Lastly, an innovative deep learning-based framework is developed by processing the features extracted by a convolutional front-end with long short-term memory cells, thus proposing the first deep recurrent convolutional neural network for spacecraft pose estimation. The framework is used to compare the performance achieved by visible-only and multimodal input sequences, where the addition of the thermal band is shown to greatly improve the performance during sunlit sequences. Potential limitations of this modality are also identified, such as when the target's thermal signature is comparable to Earth's during eclipse.

How plants develop and acquire their form is a complex issue driven by a multitude of factors. Like all biological phenomena, the processes underpinning plant development are complex both when viewed collectively, but often also when simplified and viewed in isolation from one another. Computational tools can therefore be helpful means to deal with, and make sense of, biological issues. In this thesis, four different projects are considered that each deal with distinct biological questions connected to the development of aerial plant tissues, and particularly how they acquire and maintain their shapes. A particular focus is put on using an integrative approach that combines experimental data, computational tools, and mathematical models to help forward our understanding of morphodynamic processes in the plant shoot. Firstly, a method is presented for segmenting and quantifying geometric properties in plant shoots on the tissue level. The method is applied to analyse and identify phenotypes in a broad range of different mutants, over many plants. Specifically, this high-throughput approach identifies transport proteins of the fundamental plant hormone auxin, ABCB1 and 19, as potential regulators of shoot and flower morphology. The method is also shown to be useful in identifying and parameterising plant shoots. Finally, the method is applied on a range of mutants with compromised genes involved in cell wall synthesis, where it is found that the spatial gene expressions for these have implications for shoot morphology. Secondly, a quantitative approach is used to analyse the spatial patterning of auxin signalling in initiating leaves, which is believed to have importance in the development of leaf morphology. Nuclear segmentation of ratiometric auxin reporters in 3D provide insights into how auxin is not symmetrically patterned in the early stages of leaf development, in contrast to previous belief. Thirdly, extending on the issue of auxin patterning, a quantitative approach is developed to identify the patterning of the auxin efflux transporter PIN1 in plant shoots, specifically in the context of flower organ initiation. A method is developed to investigate molecular patterning data of PIN1 from many samples, in 3D. The results indicate that PIN1 patterning is tightly linked to the initialisation of new flower organs, supporting the notion of PIN1 as a primary driver of phyllotactic patterning. However, in contrast to a recent study, the analysis shows strong convergence patterns of PIN1 polarity at initiation sites. This data is then complemented with a mathematical model simulating auxin transport driven by the extracted PIN1 data. Strikingly, it is found that whilst auxin patterning in the shoot follows that of PIN1, the two are not always similar, and local auxin maxima can appear in the tissue even before clear PIN1 concentration maxima or convergence patterns do so. Lastly, a high-throughput analysis of plants with compromised functionality of POM2/CSI1, a linker protein between cellulose synthase complexes and microtubules, is conducted. Methods from the above projects are integrated to assess phenotypes in mutant plants relative to wild-type both at cell and tissue scales. It is found that plants with defective POM2/CSI1 functionality have flattened shoot apices and compromised cell geometries. By analysing how cell geometry data is spatially patterned, and how these properties develop over time, a change in mechanical properties is assessed to be the primary cause of the phenotypes. A computational model is then used to demonstrate how reduced mechanical stiffness is sufficient to explain the flattened shoots. Collectively, the results in this thesis emphasise the applicability of computational methods in addressing questions in plant development. The integration of experiments, data quantification tools, and mathematical modelling of molecular patterning and mechanical regulation, illustrate a holistic approach to the research of these issues. As a result, this enables the investigation of questions where more singular approaches become limited.

Generating captions for ultrasound images and videos is an area that is yet to be fully studied and explored. The aim of the work in this thesis is to learn joint image-text representations to describe ultrasound images with rich vocabulary consisting of nouns, verbs, and adjectives. Preparing medical image captioning benchmarks is challenging for two reasons: (a) describing medical images with specific terminology requires expert knowledge of medical professionals; and (b) the sensitive nature of medical images prevents wide-scale annotation, for instance, using crowd-sourcing services (e.g. Amazon Mechanical Turk) and similar methods. Therefore, automatic image captioning has not been widely studied on ultrasound images before, the challenge being enhanced by the lack of readily available large datasets of ultrasound images with captions. First, the thesis explores different combinations of recurrent neural networks, concatenation techniques, word embedding vectors in different model architecture configurations. We identify in this process the configuration most suitable for the fetal ultrasound image captioning task and the dataset at hand. We show that a configuration incorporating an LSTM-RNN and word2vec embeddings and using a merge-by-concatenation operation performed best. The thesis then explores three solutions to the challenge of working with real world datasets. We introduce a curriculum learning based strategy that incorporates the complexities of the image and text information to prepare the data for training. We show that by training captioning models with the order of data samples determined by the curriculum, we can achieve higher scores on the evaluation metrics with the same amount of data. We also look into augmenting the data through the creation of pseudocaptions to pair up with caption-less images. Finally, we explore leveraging other available data from a different modality, specifically eye gaze points, to supplement available image-text data. We find that using eye gaze data can help in training models that score relatively higher on the evaluation metrics; however since the improvements are small and the pre-training steps involved are considerable, this leads us to the recommendation that improving base models should take precedence over relying on data from other modalities to improve the performance of captioning models. To the best of our knowledge, the work in this thesis is the first attempt to perform automatic image captioning on fetal ultrasound images (video frames), using sonographer spoken words to describe their scanning experience. The thesis can help serve as a blue print for future endeavours in fetal ultrasound captioning by providing guidelines to follow and pitfalls to avoid and as an aid for those attempting medical image captioning, more generally.

Human perception and learning are inherently multimodal: we interface with the world through multiple sensory streams, including vision, audition, touch, olfaction and taste. By contrast, automatic approaches for machine perception and learning have traditionally depended on single modalities, by processing, for instance, video, audio or speech separately. The goal of this thesis is instead utilizing the natural co-occurrence of audio and visual information in videos to learn useful tasks. The thesis is structured around four main themes: (i) lip reading and Audio-Visual Speech Recognition (AVSR); (ii) audio-visual speech enhancement and separation; (iii) audio-visual sound source localization and detection; (iv) sign language recognition; Lip reading is the ability to recognise speech by observing the speaker's lip movements; it is a challenging task and has many important applications including enabling speech impaired individuals to better communicate. We build and improve on recent breakthroughs by exploring the use of Transformer-based architectures, proposing attention based pooling mechanisms for representation aggregation, as well as using sub-word units instead of character tokenisation. These enhancements, combined with improvements to the training protocol, yield substantial performance boosts, resulting in state-of-the art results on the challenging LRS2 and LRS3 datasets. Moreover, we develop a method for exploiting unlabelled speech video by distilling an Automatic Speech Recognition Model into a lip-reading one. Finally we show that it is possible to identify spoken language just by observing a speaker's lip movements. Speech enhancement and separation increases the signal-to-noise ratio of noisy speech audio, by filtering out interfering voices or background noise. Until recently, works in this area focused on solving the problem by using the audio modality alone. We first propose tackling this problem audio-visually by conditioning on each speaker's lip movements. We then further improve this approach by making it robust to visual occlusions. Recent works have shown that it is possible to determine the spatial location of sound-making objects in video frames by exploiting correlations between the audio and video signals. We present a method to improve and extend these techniques, by grouping heat maps into distinct object representations that can be used for various downstream tasks, without the need for face detectors. The resulting method is entirely self-supervised and can be used for extending tasks such as active speaker detection and speech separation in new domains, e.g. videos of cartoons or puppets. We then propose a method that uses similar principles in order to train object detection models without relying on human annotation, by deriving all the necessary supervision from audio-visual correspondence cues. Finally we consider the problem of automatic sign-language recognition, which to-date remains unsolved, despite all the progress in related vision and natural language processing tasks. The main blocker is the scarcity of large-scale annotated sign-language datasets. We attempt to solve this problem by using sign-interpreted TV broadcasts footage, combined with subtitles obtained from the corresponding audio speech. Towards achieving this goal we first train Transformer models to identify and temporally localize instances of sings in continuous signed videos, thus automatically generating thousands of annotations for a large sign vocabulary. We then directly tackle the problem of temporally aligning the asynchronous subtitles to the sign language footage.

Deep learning has fueled an explosion of applications, yet training deep neural networks usually requires expensive human annotations. In this thesis we explore alternatives to avoid the substantial reliance on manual annotated examples when training deep neural networks. Specifically, we do so by either adapting self-supervised methods to automatically correct freely obtained data labels, or by completely abandoning the use of human labels and instead utilizing the natural co-occurrence of audio and visual information to learn object representations in videos. Growing collections of digital data often provide noisy labels that can be exploited to supervise the learning process. Conventional data pre-processing includes correcting/cleaning them before training recognition models, but this can require infeasible amounts of manual effort. We consider correcting the annotation noise automatically, and hence eschew the need for costly manual annotation. We build and extend recent breakthroughs with a consistency loss which enables training even without ground truth, and an spatial memory map that provides flexible instance-level registration, leading to greater generalization. We further explore multimodal sensory streams to provide self-supervision to the model by utilizing modality redundancy, i.e. the overlapping information between modalities. Representations are learned by harnessing different modalities without using any human-annotated labels. We demonstrate this technique using three different applications. First, we automatically curate a large-scale audio dataset, VGG-Sound, with more than 200k videos collected using visual guidance, training on which yields state-of-the-art models for audio recognition. Second, we present a method to improve and extend recent sound source localization techniques by introducing a mechanism to mine hard samples and add them to a contrastive learning formulation automatically. Finally, unlike existing audio-visual synchronization tasks performed on one specific domain, we propose to solve the synchronization problem in open world settings by exploring the use of several transformer-based architectures. With these models, we achieve state-of-the-art results in challenging speech datasets and show excellent generalization in a general sound dataset.

This practicebased research engages new artistic production in an examination of the aesthetics of 3D imaging technologies. In particular, the research concerns the photogrammetric image and its aesthetics as encountered in art practice. Critical discourse on photogrammetry in art practice is underexplored. Where such discourse does exist, for instance in and around the work of the research and activist group Forensic Architecture, it has tended to focus on questions of functionality. This PhD proposes a new starting point for an understanding of photogrammetric representation in its own terms. The study finds the partiality of recent critical research writing on photogrammetry to be too heavily conditioned by discourses of photography. Such discussion fails to appreciate the computational mediation at the heart of photogrammetry. The photogrammetric image is one of a range of images recently emer which are subject to heavily automated computational processes. This study sets out a conceptual framework for understanding these images; photogrammetric images being one of an emergent range of 'Spatially-immersive Networked Composites' or 'SiNCs'. The research outlines a way of foregrounding qualities of layering and assemblage through computation as pivotal to understanding the image. These images are created through algorithmic analysis resulting in the formation of a computational, navigable environment. The project engages sculptural practice, video, Augmented Reality, and media installations. It provokes a plurality of encounters to be enlisted into the research, thus demonstrating the necessity of art making in this research. New forms of Media Archaeological methods are employed, focusing on glitch practices that explore this evolving technology. Under certain conditions, peculiar errors and aberrations occur. These attributes reveal a glimpse of the image's materiality by showing estimations and extrapolations of algorithms. Methods devised include generating the conditions for such errors to better understand the aesthetics of Spatially-immersive Networked Composites (SiNCs), both on screen and removed from navigable, screen-based space. The urgency of the research is evident in a digital media environment in which, through automation and algorithmic agency, image production and dissemination are changing rapidly. This research sets the conditions for discussion for emergent forms of imagery, encouraging wider and more critical engagement with the photogrammetric image and its associated, evolving technologies.

This thesis explores how a computer can learn the structure of visual objects in the absence of strong supervision using self-supervised learning. We demonstrate that we can learn structural representations of objects using an autoencoding framework with reconstruction as the key learning signal. We do this by engineering bottlenecks that disentangle object structure from other factors of variation. Moreover, we design the bottlenecks to represent the object structure in the form of 2D and 3D object landmarks or 3D mesh. Specifically, we develop a method that automatically discovers 2D object landmarks without any annotations using a conditional autoencoder with 2D keypoint bottleneck that disentangles pose, represented as 2D keypoints, and appearance. Despite the ability of self-supervised learning methods to learn stable object landmarks, the automatically discovered landmarks are not aligned with landmarks that would be annotated by human annotators. To address this, we present a method that can inject an unpaired empirical prior into a conditional autoencoder by introducing a novel landmark autoencoding that can leverage powerful image discriminators used in adversarial learning. A by-product of these conditional autoencoding methods is that the generation can be interactively controlled by manipulating the keypoints in the bottleneck. We leverage this feature in a novel method for interactive 3D shape deformation. The method is trained in a self-supervised way to use automatically discovered 3D landmarks to align pairs of 3D shapes. In the test time, the method allows the user to interactively deform the object shape via the discovered 3D object landmarks. Finally, we present a method that uses a photo-geometric autoencoder to recover 3D shape of an object category without any 3D annotations. It uses videos for training and learns to disentangle an image input into a rigid pose, texture and deformable shape model.

Vision loss is rarely absolute; most people referred to as blind have some form of residual vision. This residual vision can create a confusing view of the world. In this thesis, I explore how a camera system and a wearable display can augment an image of the word to improve its visibility and understanding to a visually impaired wearer. I focus on two crucial parts of life; recognising faces and reading. In the first part of this thesis, I design and validate a new face perception test to assess different image enhancement forms. I used this new test to measure the impact of image enhancement on a visually impaired population (n=64). The results found that approximately 40% of participants showed a significant improvement in test scores. Performance was related to reading acuity and lower spatial frequency loss. I then develop and integrate a complete real-time image processing solution for face perception enhancement with an augmented reality display, demonstrating a practical application of the image enhancements. The second part of this thesis's focus is reading. I compare paper-based reading performance against reading with a virtual reality (VR) display with a visually impaired population (n=53). I further assess the impact of different colours on reading with the VR display. The results found no difference in reading performance, comparing VR to paper- based. The colour of text and background had a significant effect, with 75% of participants having a preferred colour combination. A participant with a higher overall magnitude of colour vision loss is more likely to benefit from white text on a black background than black text on a yellow background. Augmenting an individual's residual vision can improve the quality of information they receive about the visual world. While not universal, this improvement has the potential to enhance a visually impaired person's quality of life significantly.

State estimation is a fundamental requirement for mobile robots. An accurate and robust estimate of a platform's location and environment is a key input for many other robot systems, including control, navigation, and task planning. The aim of this thesis is to develop a state estimation system based on probabilistic factor graph optimisation. The proposed algorithm is called VILENS (Visual Inertial Lidar Legged Navigation System). It uses different combinations of camera, lidar, inertial, and kinematic sensors to estimate the robot's state, and is designed to be robust to individual sensor failure. This thesis presents several novel contributions. First is a new method of integrating kinematic measurements into a factor graph. The proposed preintegrated twist factor aims to reduce the drift caused by slippage or deformation by estimating kinematics biases online. These effects are not modelled in other works. Second, a novel method for tracking and optimising sparse geometric features from lidar scans is presented. This differs from the literature which focuses on scan-to-scan or scan-to-map matching. Third, the complete VILENS system, which is the first attempt at integrating visual, inertial, lidar, and kinematic measurements into a factor graph. This makes use of a novel camera-lidar synchronisation method to reduce the graph size and complexity. A key benefit of this tight fusion is the natural handling of degenerate scenarios without hard switching. This algorithm has been widely tested on a variety of platforms, including legged robots and handheld sensor rigs. This thesis includes one of the first extensive real-world demonstrations of factor graphs for legged robot state estimation and has achieved relative position errors as low as 0.4% per distance travelled in challenging underground environments. Several large-scale deployments of mobile robots have either motivated its development or directly incorporated VILENS for state estimation. These include sewer inspection, 3D building reconstruction, and the DARPA Subterranean Challenge.

Expressing ideas in our minds which are inevitably visual into words had been a necessity. Lack of tools to express oneself creatively using visual communication had been a bottleneck. With the advent of AI and generative models starting to be applied to graphics based tasks, this elusive elysian fields seems to be a step closer. This thesis tries to tackle the challenge using the 3 pronged approach of GANs, VAEs and Continuous Normalizing Flows. We first propose MAD-GAN, an intuitive generalization to the Generative Adversarial Networks (GANs) and its conditional variants to address the well known problem of mode collapse. First, MAD-GAN is a multi-agent GAN architecture incorporating multiple generators and one discriminator. Second, to enforce that different generators capture diverse high probability modes, the discriminator of MAD-GAN is designed such that along with finding the real and fake samples, it is also required to identify the generator that generated the given fake sample. Intuitively, to succeed in this task, the discriminator must learn to push different generators towards different identifiable modes. We perform extensive experiments on synthetic and real datasets and compare MAD-GAN with different variants of GAN. We show high quality diverse sample generations for challenging tasks such as image to image translation and face generation. In addition, we also show that MAD-GAN is able to disentangle different modalities when trained using highly challenging diverse-class dataset (For example the dataset with images of forests, icebergs, and bedrooms). In the end, we show its efficacy on the unsupervised feature representation task. We introduce a similarity based competing objective (MAD-GAN-Sim) which encourages different generators to generate diverse samples based on a user defined similarity metric. We show its performance on the image-to-image translation, and also show its effectiveness on the unsupervised feature representation task. We then propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects. As the user starts to draw a sketch of a desired object type, the network interactively recommends plausible completions, and shows a corresponding synthesized image to the user. This enables a feedback loop, where the user can edit their sketch based on the network's recommendations, visualizing both the completed shape and final rendered image while they draw. In order to use a single trained model across a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network. In Appendix C we propose a semi supervision based strategy for deep generative modelling for human body analysis. Deep generative modelling for human body analysis is an emerging problem with many interesting applications. However, the latent space learned by such approaches is typically not interpretable, resulting in less flexibility. In this work, we present deep generative models for human body analysis in which the body pose and the visual appearance are disentangled. Such a disentanglement allows independent manipulation of pose and appearance, and hence enables applications such as pose-transfer without specific training for such a task. Our proposed models, the Conditional-DGPose and the Semi-DGPose, have different characteristics. In the first, body pose labels are taken as conditioners, from a fully-supervised training set. In the second, our structured semi-supervised approach allows for pose estimation to be performed by the model itself and relaxes the need for labelled data. Therefore, the Semi-DGPose aims for the joint understanding and generation of people in images. It is not only capable of mapping images to interpretable latent representations but also able to map these representations back to the image space. We compare our models with relevant baselines, the ClothNet-Body and the Pose Guided Person Generation networks, demonstrating their merits on the Human3.6M, ChictopiaPlus and DeepFashion benchmarks. Finally, Training Neural Ordinary Differential Equations (ODEs) is often computationally expensive. Indeed, computing the forward pass of such models involves solving an ODE which can become arbitrarily complex during training. Recent works have shown that regularizing the dynamics of the ODE can partially alleviate this. In this paper we propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. Further, the technique is orthogonal to several other methods proposed to regularize the dynamics of ODEs and as such can be used in conjunction with them. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models.

Across many sectors concerned with animal husbandry, there is a growing need for automated tools for continuously monitoring animals under our care. In farmyards, zoos, veterinary centres, animal research facilities and many others, humans are typically responsible for identifying signs of disease or distress within their animal populations. While this can be effective, a significant challenge is posed when a small number of humans are expected to care for large animal groups. Existing automated systems often install cameras and use computer vision algorithms to track motion and/or predict behaviours. However, these approaches are either low fidelity or cannot be readily applied to identifying ill health due to ethical concerns around data collection. This thesis proposes a solution based on using 3D morphable models (3DMMs) as a useful intermediary 3D animal representation, enabling identification of adverse events and reducing the task-specific data requirements for downstream behaviour/welfare prediction tasks. A 3DMM is a specially designed mesh, supplied with parameters that constrain deformations. Typical reconstruction pipelines tackle reconstructing a 3D articulated structure as estimating the 3DMM parameters per input frame. This thesis focuses on designing methods for 3D animal reconstruction, making use of suitable 3DMMs. We as humans are highly proficient at estimating the 3D structure of articulated subjects, such as animals or people. Even from a single image or video sequence, a human can with reasonable accuracy, predict the 3D locations of the animal's limbs, estimate body proportions and even the camera's location and viewing direction. However, these remain challenging tasks for computers. For this reason, 3DMMs provide a useful intermediate representation for downstream tasks since they help to explain the system's output - if the recovered 3D model looks wrong, a human will notice immediately. Recently, great progress has been made in 3D human reconstruction but there are particular chal- lenges which impede naively transferring these techniques to animal categories. Firstly, human recon- struction techniques use large, specialized datasets of images with corresponding 2D and 3D annotations to learn accurate 3DMMs and train neural networks. Unfortunately, animal datasets are extremely limited in size, number and variability which has led to significant prior work in animals requiring per-frame manual annotations at test time. This lack of data also contributes to difficulties in designing detailed 3D priors, which are needed to cover the enormous pose and shape diversity among animal species when compared to humans. Other challenges faced in this thesis are common to humans and animals alike, such as when tackling ambiguous input imagery, for example caused by self or environmental occlusions. This thesis proposes a series of methods which are designed to tackle these challenges. Chapter 3 begins by introducing two new animal datasets which are relied upon in subsequent chapters. Chapter 4 discusses the first approach that performs fully automatic 3D reconstruction on a wide range of quadruped species, based on a graphics pipeline for generating synthetic data. Chapter 5 introduces vi WLDO, an automatic and real-time system for 3D dog reconstruction which covers an approach for improving the representational power of a low-fidelity 3DMM; a common problem for animals, since the lack of 3D training data necessitates learning from other sources, such as artist-designed figurines. The approach operates without a real 3D dataset and produces state-of-the-art accuracy on a challenging new dataset StanfordExtra, outperforming energy minimization approaches even when they are given ground-truth test-time annotations. Chapter 6 proposes a technique for reconstructing images with heavy occlusion, which is an open problem across 3D reconstruction literature and a common failure mode of existing systems. The approach achieves state-of-the-art performance on challenging benchmarks.

Computer Vision involves many challenging problems. While early work utilized classic methods, in recent years solutions have often relied on deep neural networks. In this study, we explore those two classes of methods through two applications that are at the limit of the ability of current computer vision algorithms, i.e., faint edge detection and multispectral image registration. We show that the detection of edges at a low signal-to-noise ratio is a demanding task with proven lower bounds. The introduced method processes straight and curved edges in nearly linear complexity. Moreover, performance is of high quality on noisy simulations, boundary datasets, and real images. However, in order to improve accuracy and runtime, a deep solution was also explored. It utilizes a multiscale neural network for the detection of edges in binary images using edge preservation loss. The second group of work that is considered in this study addresses multispectral image alignment. Since multispectral fusion is particularly informative, robust image alignment algorithms are required. However, as this cannot be carried out by single-channel registration methods, we propose a traditional approach that relies on a novel edge descriptor using a feature-based registration scheme. Experiments demonstrate that, although it is able to align a wide field of spectral channels, it lacks robustness to deal with every geometric transformation. To that end, we developed a deep approach for such alignment. Contrarily to the previously suggested edge descriptor, our deep approach uses an invariant representation for spectral patches via metric learning that can be seen as a teacher-student method. All those pieces of work are reported in five published papers with state-of-the-art experimental results and proven theory. As a whole, this research reveals that, while traditional methods are rooted in theoretical principles and are robust to a wide field of images, deep approaches are faster to run and achieve better performance if, not only sufficient training data are available, but also they are of the same image type as the data on which they are applied.

In this thesis, a scene analysis mainly focusing on vision-based techniques have been explored. The vision-based scene analysis techniques have a wide range of applications from surveillance, security to agriculture. A vision sensor can provide rich information about the environment such as colour, depth, shape, size and much more. This information can be further processed to have an in-depth knowledge of the scene such as type of environment, objects and distances. Hence, this thesis covers initially the background on human detection in particular pedestrian and crowd detection methods and introduces various vision-based techniques used in human detection. Followed by a detailed analysis of the use of synthetic data to improve the performance of state-of-the-art Deep Learning techniques and a multi-purpose synthetic data generation tool is proposed. The tool is a real-time graphics simulator which generates multiple types of synthetic data applicable for pedestrian detection, crowd density estimation, image segmentation, depth estimation, and 3D pose estimation. In the second part of the thesis, a novel technique has been proposed to improve the quality of the synthetic data. The inter-reflection also known as global illumination is a naturally occurring phenomena and is a major problem for 3D scene generation from an image. Thus, the proposed methods utilised a reverted ray-tracing technique to reduce the effect of inter-reflection problem and increased the quality of generated data. In addition, a method to improve the quality of the density map is discussed in the following chapter. The density map is the most commonly used technique to estimate crowds. However, the current procedure used to generate the map is not content-aware i.e., density map does not highlight the humans' heads according to their size in the image. Thus, a novel method to generate a content-aware density map was proposed and demonstrated that the use of such maps can elevate the performance of an existing Deep Learning architecture. In the final part, a Deep Learning architecture has been proposed to estimate the crowd in the wild. The architecture tackled the challenging aspect such as perspective distortion by implementing several techniques like pyramid style inputs, scale aggregation method and self-attention mechanism to estimate a crowd density map and achieved state-of-the-art results at the time.