Search

This thesis explores the challenges of enabling human physiological sensing by leveraging head-worn wearable computer systems. In particular, we want to answer a fundamental question, i.e., could we leverage head-worn wearables to enable accurate and socially-acceptable solutions to improve human healthcare and prevent life-threatening conditions in our daily lives? To that end, we will study the techniques that utilise the unique advantages of wearable computers to (1) facilitate new sensing capabilities to capture various biosignals from the brain, the eyes, facial muscles, sweat glands, and blood vessels, (2) address motion artefacts and environmental noise in real-time with signal processing algorithms and hardware design techniques, and (3) enable long-term, high-fidelity biosignal monitoring with efficient on-chip intelligence and pattern-driven compressive sensing algorithms. We first demonstrate the ability to capture the activities of the user's brain, eyes, facial muscles, and sweat glands by proposing WAKE, a novel behind-the-ear biosignal sensing wearable. By studying the human anatomy in the ear area, we propose a wearable design to capture brain waves (EEG), eye movements (EOG), facial muscle contractions (EMG), and sweat gland activities (EDA) with a minimal number of sensors. Furthermore, we introduce a Three-fold Cascaded Amplifying (3CA) technique and signal processing algorithms to tame the motion artefacts and environmental noises for capturing high-fidelity signals in real time. We devise a machine-learning model based on the captured signals to detect microsleep with a high temporal resolution. Second, we will discuss our work on developing an efficient Pattern-dRiven Compressive Sensing framework (PROS) to enable long-term biosignal monitoring on low-power wearables. The system introduces tiny on-chip pattern recognition primitives (TinyPR) and a novel pattern-driven compressive sensing technique (PDCS) that exploits the sparsity of biosignals. They provide the ability to capture high-fidelity biosignals with an ultra-low power footprint. This development will unlock long-term healthcare applications on wearable computers, such as epileptic seizure monitoring, microsleep detection, etc. These applications were previously impractical on energy and resource-constrained wearable computers due to the limited battery lifetime, slow response rate, and inadequate biosignal quality. Finally, we will further explore the possibility of capturing the activities of a blood vessel (i.e., superficial temporal artery) lying deep inside the user's ear using an ear-worn wearable computer. The captured optical pulse signals (PPG) are used to develop a frequent and comfortable blood pressure monitoring system called eBP. In contrast to existing devices, eBP introduces a novel in-ear wearable system design and algorithms to eliminate the need to block the blood flow inside the ear, alleviating the user's discomfort.

Giving machines the ability to precisely perceive and understand the 3D visual world is the fundamental step to allow them to interact competently within our physical world. However, the research on large-scale 3D scene understanding and perception is still in its infancy, due to the complex geometrical structure of 3D shapes and limited high-quality data resources. Among various 3D representations, point clouds have attracted increasing attention due to its flexibility, compactness, and the characteristic of being close to raw sensory data. Despite this, the semantic understanding of large-scale 3D point clouds remains a challenge due to its orderless, unstructured, and non-uniform proper-ties. To this end, this thesis makes three core contributions from a high-quality urban-scale dataset, and then fully-supervised semantic understanding to weakly supervised label efficient learning of large-scale 3D point clouds. The main contributions of this thesis are three-fold. In Chapter 3, we start by building an urban-scale photogrammetric point cloud dataset with nearly three billion richly annotated points. We also identify a number of open and unique challenges faced by urban-scale 3D scene understanding and conduct comprehensive experimental analysis to address these challenges. This work points out the limitations of existing algorithms and provides several insightful conclusions for understanding large-scale 3D point clouds in urban environments. With high-quality large-scale 3D datasets, we further investigate the research problem of efficient semantic segmentation of large-scale 3D point clouds in Chapter 4. We begin by analyzing the strengths and weaknesses of existing downsampling strategies and find that random sampling is a suitable component for efficient learning on large-scale point clouds. Additionally, we propose a local feature aggregation module to increase the receptive field hierarchically and preserve important features. We then build an efficient, lightweight neural architecture called RandLA-Net that can directly infer per-point semantics for large-scale point clouds with millions of points that span hundreds of meters. In Chapter 5, we take a step further to investigate the label-efficient learning of large-scale 3D point clouds. That is, achieving high-quality semantic segmentation with limited annotations. We first examine key issues in weakly supervised learning of 3D point clouds, including different weak supervision schemes and the critical point of weak annotations. Through a pilot study, we find that dense 3D annotations are actually redundant and unnecessary. Motivated by this, we propose a novel weak supervision framework that implicitly augments the total amount of available supervision signals by leveraging the semantic homogeneity between neighboring points. This is achieved using a point neighborhood query to allow the sparse training signals to be back-propagated to a wider context. All of the algorithms and datasets presented in this thesis have been open-source on GitHub to facilitate future research. The RandLA-Net algorithm was recognized as one of the most influential papers at the CVPR 2020 conference and has been integrated into code libraries such as Open3D and TorchPoints3D. The SensatUrban dataset was used as the platform for two Urban3D challenges, contributing to the advancement of semantic understanding of city-scale point clouds. Overall, this thesis presents a high-quality dataset and two novel data-driven algorithms, aiming to achieve efficient, scalable, and effective learning-based semantic understanding of large-scale 3D point clouds, eventually improving the real-time 3D perception capacity of intelligent machines in practice.

Scene representation is the process of converting sensory observations of an environment into compact descriptions. Such intelligent behavior is a cornerstone of artificial intelligence. Scientists have long sought to reproduce the extraordinary ability of humans for understanding the physical environment. Taking visual sensory observations of an environment as input, modern intelligent systems mostly aim to learn neural representations that encode fundamental scene properties such as geometry and semantics. Such representations can be leveraged to support other downstream tasks and finally realize autonomous perception and interaction within the complex 3D world. Impressive performance has been exhibited in recent deep neural networks that excel at modeling geometric and semantic information in neural scene representations. However, constructing robust systems is still highly challenging due to the fragility in uncontrolled real-world scenarios. This presents significant complexity for scene representation learning due to the variance of sensory observations with respect to scene changes, the domain gap among different types of visual representations, and the requirement for efficient perception for multiple categories of information. To overcome these challenges, we pursue robust, unified and informative scene representations, learning with geometry and semantics from different types of visual inputs, paving the way towards intelligent machines that autonomously learn to understand the world around them. In this context, this thesis makes three core contributions in the fields of visual localization, pixel-point matching, and semantic surface reconstruction. In this thesis, we start by estimating the 6 Degree-of-Freedom (DoF) camera pose from single images. To learn scene representations that are robust to environmental changes and sensor operation, a neural network together with a self-attention module is proposed to model complex geometry relationships from which given images are taken relative to a reference environment. Then, we build a more general framework to find unified representations across 2D images and 3D point clouds based on the inherent constraints from epipolar geometry and stereo vision. By introducing an ultra-wide reception mechanism together with novel loss functions, a dual fully-convolutional framework is presented that maps 2D and 3D inputs into a shared latent representation space to simultaneously describe and detect keypoints, bridging the gap between 2D and 3D representations. Finally, we extend our study to develop informative representations, which are generally required for intelligent systems to operate in real-world scenarios for multiple purposes at the same time. By drawing on previous work in point-based networks, we introduce a brand-new end-to-end neural implicit function that can jointly estimate precise 3D surface and semantics from raw and large-scale point clouds. Overall, this thesis develops a series of novel deep neural frameworks to advance the field of machine learning of scene representations towards artificial intelligence that can fully perceive our real-world 3D environment.

Localization is a fundamental problem in many application scenarios, including robotics, computer vision and autonomous driving. In this thesis, localization is defined as two sub-tasks: odometry and relocalization. Odometry estimates the relative poses of a sensor in real-time when provided with a sequential series of sensory input data, and relocalization predicts the global poses of a sensor from previously traversed scenes. Recently, deep neural networks have been applied to visual localization and shown promising results. However, vision sensors such as cameras are greatly impacted by environmental conditions which limits the application of purely visual methods in the real world. Active sensors, such as LiDAR and radar, can capture rich geometric and semantic information in the forms of point clouds and polar scans and they are significantly more robust to environmental variations. Inspired by the recent deep visual localization methods, this thesis studies how to apply deep learning for the localization task using LiDAR and radar sensors. Specifically, this thesis studies three aspects of LiDAR and radar based localization with deep neural networks. First, an end-to-end parallel neural network termed DeepPCO is proposed for the LiDAR odometry task. It consists of two sub-networks to predict translation and orientation respectively. Second, a novel LiDAR point cloud-based method for deep LiDAR relocalization is introduced, termed PointLoc. It learns features from point clouds directly for estimating global poses in an end-to-end manner. In addition, a new indoor LiDAR-visual dataset termed vReLoc is released for studying the LiDAR relocalization problem. Third, a radar-based relocalization system, RadarLoc, is proposed by utilizing deep learning and FMCW radar scans. It designs a selfattention module with a nested encoder-decoder architecture to mask out outliers. Geometric constraints are leveraged in the loss function to improve the overall performance. Extensive experiments on large-scale datasets have been conducted to demonstrate the effectiveness of the above three approaches. This thesis also discusses implications and future research directions of this field.

Machine learning (ML) has caused a fundamental shift in how we practice science, with many now placing learning from data at the focal point of their research. As the complexity of the scientific problems we want to study increases, and the amount of data generated by today's scientific experiments grows, ML is helping to automate, accelerate and enhance traditional workflows. Emerging at the forefront of this revolution is a field called scientific machine learning (SciML). The central goal of SciML is to more tightly combine existing scientific understanding with ML, generating powerful ML algorithms which are informed by our prior knowledge. A plethora of approaches exist for incorporating scientific principles into ML and expectations are rising for SciML to address some of the biggest challenges in science. However, the field is burgeoning and many questions are still arising. A major one is whether SciML approaches can scale to more complex, real-world problems. Much SciML research is at a proof-of-concept stage, where techniques are validated on simplified, toy problems. Yet, understanding how well they scale to more complex problems is essential for them to become widely applicable. This question is of central focus in this thesis. Firstly, multiple different physics-informed ML approaches are designed for three complex, real-world, domain-specific case studies taken from the fields of lunar science and geophysics, and their performance and scalability is assessed. Secondly, the scalability of physics-informed neural networks, a popular and general SciML approach, for solving differential equations with large domains and high frequency solutions is evaluated and improved. Common observations across these studies are discussed, and significant advantages and underlying limitations are identified, highlighting the importance of designing scalable SciML techniques.

The deployment of millions of embedded sensors plagued by resource constraints in sophisticated, complex and dynamic Internet of Things (IoT) environments continues to inspire the need to build novel architectures and models for automated, efficient inference and communication. In such settings, practical challenges related to energy efficiency, computational power and reliability, tedious design implementation, effective communication, optimal sampling and accurate event classification, prediction and detection exist. Sensors operating in such environments must be capable of overcoming such challenges and enable scalable monitoring of dynamic phenomena whilst respecting limitations computational capabilities and battery energy. We continue to see a shift from reliance on cloud-centric to edge-centric architectures for data processing, inference and actuation. Distributed edge inference techniques address real-time, connectivity, network bandwidth and latency challenges in spatially dis- tributed IoT applications. Achieving efficient, resource-aware communication in such systems is a longstanding challenge. Many current approaches require complex, hand-engineered communication protocols. The development of Machine Learning (ML) continues to motivate a new wave of innovative solutions that intermarry embedded sensors, IoT, and ML to enable various applications. In this thesis, we explore the idea of maximizing the shared utility of multiple nodes (agents) seeking to achieve a sensing objective using a data-driven approach. Among other considerations, it is imperative that agents not only learn to communicate vital information, but also learn to know when to communicate. Firstly, we describe an in-network processing computing paradigm in which modern distributed Deep Networks are employed for inference on sensed objectives. This computing approach unifies deep learning and decentralized computation. Secondly, we propose a distributed communicating architecture based on Recurrent Neural Networks (RNNs) that can be instantiated on resource- constrained devices observing unique data and performing automated dis- tributed inference via hidden-state communication. We prove that our framework is able to use a data-driven approach to collectively solve various distributed objectives using sensor-like data, as evidenced by a series of systematic analyses we present. Thirdly, we present a novel scalable, data-driven and communication- efficient Convolutional Recurrent Neural Network (C-RNN) framework for distributed tasks. Our framework can extract relevant features on visual data and automatically learns communication protocols to solve tasks in distributed visual settings. Finally, we demonstrate that our framework can be extended to visual tracking scenarios. In doing this, we introduce novel and scalable methods to setup, train, and deploy communication-efficient distributed nodes using Deep Networks. Our empirical and systematic analyses of model convergence, node scalability, energy-efficiency, computational complexity and communication- cost on varying node topologies demonstrate that our proposed frame- works are able to employ Deep Networks to discover and optimize communication protocols on data with different modalities and can be extended to many distributed applications in resource-constrained systems.

Ubiquitous sensing is a key enabler for smart environment applications, and mmWave radar is an ideal candidate due to its high usability, low cost and privacy preserving features. Tracking and recognition are two fundamental functions of mmWave radar sensing, but with conventional methods, their performance is often affected by many factors such as noise, limited resolution, and multi-path effects. In this thesis, we explore the use of novel deep learning techniques in radar signal processing chain for more accurate and robust tracking and recognition, to better support smart environment applications. First we explore the problem of using radar point cloud for human tracking and identification. In the proposed system named 'mID', we first associate clusters between frames to track human targets, followed by using deep neural networks to take a sequence of voxelized point clusters as input for human identification, as well as using the latent features for intruder detection. In the following chapter we propose 'mBeats', a system to monitor user's heart rate using deep learning with a robot mounted mmWave radar. 'mBeats' shows superior performance compared to conventional methods for heart rate prediction, and can also estimate the prediction uncertainty, which is very important for health-related applications. Next, we introduce using a deep learning based method for 3D tracking of an unmodified drone. The proposed technique can achieve a tracking accuracy 3 times better than conventional methods, with improved robustness and computational efficiency. In the last work, we use an end-to-end neural network, replacing the conventional Discrete Fourier Transform (DFT) based pre-processing with 'CubeLearn', which is our proposed learnable pre-processing module, for human gesture and activity recognition. 'CubeLearn' can adaptively extract the most informative features from raw data which best fit the target task, and shows consistently better performance than the conventional DFT pre-processing module with different models and on different tasks. Through the research introduced in this thesis, we demonstrate that applying learning based techniques for mmWave radar data processing can significantly enhance tracking and recognition performance, and benefit downstream smart environment applications.

Wild animals have evolved a diverse range of energy-budgeting tactics to cope in situ with rapid environmental change. My thesis investigates these tactics and their underlying mechanisms in a population of wild European badgers (Meles meles): 1) over the course of an individual's lifetime and 2) in response to fine-scale changes in energetic context. Chapter 1 derives a novel minimum number alive abundance estimator, incorporating variable capture efficiency to more accurately quantify changing social context for subsequent analyses. Chapter 2 exposes how changing population metrics and weather drive sex-specific life-history responses, producing diverse co-occurring paces-of-life and potentially contributing to substantial sex ratio skew at higher densities. Chapter 3 uncovers a threshold-bounded relationship between body condition and survival probability, whereby most badgers exhibit flexible somatic phenotypes within what I term an "energetic safe zone"; adverse weather and life-history risk factors can tip individuals over the threshold, but a minority of high-quality individuals exhibit high reproduction probability and lower mortality risk despite reduced body condition. Chapter 4 uses 3D printing to streamline production and reusability of bespoke multi-sensor collars, enabling simultaneous fine-scale quantification of overall dynamic body acceleration (ODBA, measuring mechanical energy expenditure) for multiple individuals. Chapter 5 applies this technology and finds season-specific fine-scale activity responses to prevailing weather conditions, according to individual energetic context; in particular, low body condition constrains energetic tactics. Chapter 6 combines ODBA with quantification of daily energy expenditure (DEE) using doubly-labelled water, demonstrating that individuals expend energy differently under the same conditions; ODBA/DEE, measuring proportional energy expenditure on activity, correlated negatively with next-season body condition. The range of energetic expenditure tactics sustained in this population support the existence of adaptive heterogeneity. I present an argument predicated on Jensen's Inequality for the maintenance of this diversity, and reflect on implications for resilience to ongoing anthropogenic change.

Accurate camera ego-motion estimation, widely known as Visual Odometry (VO), remains a key prerequisite for many applications in computer vision and robotics. Conventional VO, which relies on hand-crafted feature engineering, is prone to drift and can easily lose track as the extracted features contain outliers and unknown noise. This is problematic for application which requires robustness and high level accuracy such as for tracking UAV position in underground tunnel or estimating firefighter position in emergency operation. To alleviate the problem of noisy feature engineering, machine learning algorithms, especially Deep Neural Networks (DNN), have been used in the past few years to automatically learn robust odometry features from large amounts of data. However, despite some promising results, several fundamental drawbacks still exist in terms of accuracy, efficiency, and applicability to visually-denied environments. The work presented in this thesis tackles these shortcomings by proposing a novel network architecture and optimization strategy for DNN-based odometry estimation. To address issues of accuracy and long-term consistency, we propose to train DNN-based VO using both a windowed-based composite transformation loss and relative transformation loss through curriculum learning. With this approach, we can improve the generalization ability of the network for both translation and rotation by 21% and 16% respectively. We also propose the use of an attention network to conditionally re-weight image features such that the network can produce more accurate poses whilst being more amenable to interpretation. This method improves translation and rotation estimation by 27.8% and 43.1% respectively over the model without attention. The second contribution deals with the efficiency problem of DNN-based VO by proposing the first distillation approach for camera pose regression. We demonstrate that distilling knowledge from a deep pose regression network can be done effectively if we emphasize the knowledge transfer only when we trust the teacher network prediction. We also show that a distilled network can be further compressed with factorization and could be more generalizable due to low-rank constraints. Our proposed approach can reduce the number of student parameters by up to 92.95% (2.12× faster) whilst keeping the prediction accuracy very close to that of the teacher. Finally, we deal with the issue of tracking in visually-denied environments by proposing the first DNN-based thermal-inertial odometry system. Since thermal images inherently lack robust features, we design the network to not only extract features from the thermal images, but to also hallucinate visual features given thermal image as the input. Through extensive evaluation across two datasets, we conclude that our proposed method can produces accurate odometry estimation with less than 2 m absolute trajectory errors on average.

To endow machines with the ability to perceive the real-world in a three dimensional representation as we do as humans is a fundamental and long-standing topic in Artificial Intelligence. Given different types of visual inputs such as images or point clouds acquired by 2D/3D sensors, one important goal is to understand the geometric structure and semantics of the 3D environment. Traditional approaches usually leverage hand-crafted features to estimate the shape and semantics of objects or scenes. However, they are difficult to generalize to novel objects and scenarios, and struggle to overcome critical issues caused by visual occlusions. By contrast, we aim to understand scenes and the objects within them by learning general and robust representations using deep neural networks, trained on large-scale real-world 3D data. To achieve these aims, this thesis makes three core contributions from object-level 3D shape estimation from single or multiple views to scene-level semantic understanding. In Chapter 3, we start by estimating the full 3D shape of an object from a single image. To recover a dense 3D shape with geometric details, a powerful encoder-decoder architecture together with adversarial learning is proposed to learn feasible geometric priors from large-scale 3D object repositories. In Chapter 4, we build a more general framework to estimate accurate 3D shapes of objects from an arbitrary number of images. By introducing a novel attention-based aggregation module together with a two-stage training algorithm, our framework is able to integrate a variable number of input views, predicting robust and consistent 3D shapes for objects. In Chapter 5, we extend our study to 3D scenes which are generally a complex collection of individual objects. Real-world 3D scenes such as point clouds are usually cluttered, unstructured, occluded, and incomplete. By drawing on previous work in point-based networks, we introduce a brand-new end-to-end pipeline to recognize, detect, and segment all objects simultaneously in a 3D point cloud. Overall, this thesis develops a series of novel data-driven algorithms to allow machines to perceive our real-world 3D environment, arguably pushing the boundaries of Artificial Intelligence and machine understanding.

Location awareness is a fundamental need for intelligent systems, such as self-driving vehicles, delivery drones, and mobile devices. Given their on-board sensors (e.g. camera, inertial sensor and LIDAR), previous researchers have developed a variety of localization systems, by building hand-crafted models and algorithms. Under ideal conditions, these sensors and models are able to accurately estimate system states without time bound. However, in real-world environments, many issues such as imperfect sensor measurements, inaccurate system modelling, complex environmental dynamics and unrealistic constraints, degrade the accuracy and reliability of localization systems. Therefore, this thesis aims to leverage machine learning approaches to overcome the intrinsic problems of the human-designed localization models. This research presents learning methods to estimate self-motion using multimodal sensor data to achieve accurate and robust localization. Firstly, we exploit inertial sensor, a completely ego-centric and relatively robust sensor, to develop Inertial Odometry Neural Network (IONet) that learns motion transformation from raw inertial data, and reconstructs accurate trajectories. This inertial only solution shows impressive performance in locating people and wheeled objects without being influenced by environmental issues. IONet was further refined as L-IONet, a lightweight framework, to reduce the computational burden of model training and testing, and enable real-time inference on low-end devices. As a first trial in this direction, we collected and released Oxford Inertial Odometry Dataset (OxIOD) with a very large amount of inertial motion data collection containing 158 sequences totalling 42 km, to train and comprehensively evaluate our proposed models. Secondly, we present a novel generic framework to learn selective sensor fusion in enabling more robust and accurate odometry estimation and localization in real-world scenarios. Two fusion strategies are proposed: soft fusion, implemented in a deterministic fashion; and hard fusion, which introduces stochastic noise and intuitively learns to keep the most relevant feature representations, while discarding useless or misleading information. Both are trained in an end-to-end fashion, and can be applied to complimentary pairs of sensor modalities, e.g. RGB images, inertial measurements, depth images, and LIDAR point clouds. We offer a visualization and interpretation of fusion masks to give deeper insights into the relative strengths of each stream. Finally, we leverage deep generative models to propose Sequential Invariant Domain Adaptation (SIDA) to mitigate the domain shift problem of the deep neural network based localization models. This framework works well on long continuous sensor data. Its key novelty is to use a shared encoder to convert the input sequence into a domain-invariant hidden representation, to encourage the useful semantic features obtained, whilst discarding the domain specific features. We employ proposed SIDA on deep learning based inertial odometry and human activity recognition to demonstrate its effectiveness in improving the generalization ability in new domains. We show that SIDA is able to transform raw sensor data into an accurate trajectory in new unlabelled domains, benefiting from the knowledge transferred from the labelled source domain. Through extensive experiments, all our proposed methods demonstrate their effectiveness and potential in achieving accurate and robust localization in real-world environments.

Navigation is the one of the most fundamental capabilities required for mobile robots, allowing them to traverse from a source to a destination. Conventional approaches rely heavily on the existence of a predefined map which is costly both in time and labour to acquire. In addition, maps are only accurate at the time of acquisition and due to environmental changes degrade over time. We argue that this strict requirement of having access to a high-quality map fundamentally limits the realisability of robotic systems in our dynamic world. In this thesis, we investigate how to develop 'practical robotic navigation', motivated by the paradigm of mapless navigation and inspired by recent developments in Deep Reinforcement Learning (DRL). One of the major issues for DRL is the requirement of a diverse experimental setup with millions of repeated trials. This clearly is not feasible to acquire from a real robot through trial and error, so instead we learn from a simulated environment. This leads to the first fundamental problem which is that of bridging the reality gap from simulated to real environments, tackled in Chapter 3. We focus on the particular challenge of monocular visual obstacle avoidance as a low-level navigation primitive. We develop a DRL approach that is trained within a simulated world yet can generalise well to the real world. Another issue which limits the adoption of DRL techniques for mobile robotics in the real world is the high variance of the trained policies. This leads to poor convergence and low overall reward, due to the complex and high dimensional search space. In Chapter 4, we leverage simple classical controllers to provide guidance to the task of local navigation with DRL, avoiding purely random initial exploration. We demonstrate that this novel accelerated approach greatly reduces sample variance and significantly increases achievable average reward. The last challenge we consider is that of sparse visual guidance for mapless navigation. In Chapter 5, we present an innovative approach to navigate based on a few waypoint images, in contrast to traditional video based teach and repeat. We demonstrate that the policy learnt in simulation can be directly transferred to the real world and has ability to generalise well to unseen scenarios with minimal description of the environment. We develop and test novel approaches towards the key issues of obstacle avoidance, local guidance and global navigation, towards our vision of enabling practical robotic navigation. We show how DRL can be used as a powerful model-free approach to tackle these issues.

In this work, I will focus on ways in which we can build machine learning models that appropriately account for uncertainty, whether with computationally cheap estimates or with more expensive and reliable ones. In particular, I will explore how we can model distributions with Bayesian neural networks and how we can manipulate them depending on the task. The two main techniques for performing inference in Bayesian neural networks are variational inference and Markov chain Monte Carlo. I will look into the advantages and disadvantages of both methods and apply them to real-world problems. The emphasis is on how to achieve calibrated uncertainty estimates without compromising scalability. One contribution of this work is to offer a new method for implementing Bayesian neural networks within the framework of Bayesian decision theory, where Bayesian decision theory is important in all decision-making applications. A further contribution is developing sampling techniques that provide more reliable uncertainties, especially over data that lie outside the training distribution. Finally I also introduce a method for using Bayesian neural networks in an astrophysical application where it is vital that uncertainties are calibrated appropriately for the task.

Recent technological advances are revolutionizing the field of ecology and conservation by providing tools and techniques to overcome many of the challenges that have previously limited the understanding of particular animal species and their environments. Significant improvements in acoustic sensing tools have led to an increase in their use for the research and management of vocally active species, particularly those that are difficult to monitor visually and continuously. However, the application of acoustic monitoring methods is often limited by high equipment costs and poor knowledge of many species’ vocal behaviour. This is indeed the case for African lions (Panthera leo) which are well known for their characteristic roars, but for which an in-depth understanding of vocal behaviour has been difficult to acquire due to the complexities of directly observing and manually recording vocalizations. In this work, I investigate the use of animal-borne and static acoustic recording technology for improving the understanding of lion behaviour and explore its application as a monitoring tool for the species. I begin by testing a novel acoustic biologging approach for recording animal behaviour remotely and find it to be an effective technique for acquiring rich, long-term datasets on lion activities. Coupled with machine learning methods, I then apply this approach to investigate patterns of lion vocal behaviour and show that territorial males are more likely to roar under atmospheric conditions that favour long-distance sound transmission and also adjust their vocal behaviour depending on spatial attributes. Next, I present an affordable, custom-designed passive acoustic sensing tool capable of localizing animal call signals over long ranges (> 1 km) owing to a novel multi-microphone hardware design. Finally, I demonstrate the use of this tool for localizing lion roars and, combined with on-animal audio recordings, discover an individually unique call feature that enables vocal differentiation of individual lions which could facilitate collar-free tracking. Collectively, the findings presented in this thesis highlight the value of acoustic sensing for both understanding animal behaviour and monitoring individual movements over time and space.

A long-term vision within the realm of ubiquitous computing is the creation of smart, digital environments that provide seamless human-computer interaction, allowing computation to recede into the background of everyday life. Key to realizing this vision is the ability for machines to recognize people, so that spaces can become truly personalized. However, the unpredictability of real-world environments impacts robust recognition, limiting usability. In real conditions, human identification systems have to handle issues such as out-of-set subjects and domain deviations, where conventional supervised learning approaches for training and inference are poorly suited. The inability of supervised methods to cope with this inherent diversity could be overcome if equivalently diverse training data were readily available. Unfortunately, obtaining such comprehensive training datasets would incur huge enrolment effort and would be costly to stage. With the rapid development of Internet of Things (IoT), we advocate a new labelling method in this thesis that exploits signals of opportunity hidden in heterogeneous IoT data. The key insight is that one sensor modality can leverage the signals measured by other co-located sensor modalities to improve its own labelling performance. If identity associations between heterogeneous sensor data can be discovered, it is possible to automatically label data, leading to more robust human recognition, without manual labelling or enrolment. We believe that many currently unsolved identification problems could be addressed through our advocated concept. Specifically, this thesis demonstrates that leveraging the signals of opportunity in physical and digital observations of subjects can overcome many obstacles surrounding robust human identification, and we comprehensively tackle this in a number of research threads. Firstly, we propose scan, a general algorithm for cross-modality association, designed to automatically label biometric data sensed in the wild. Secondly, in order to mitigate the errors in the automatically labelled data, we further present autotune, a generic framework that iteratively adapts the biometric model and updates sensor observations. Lastly, we comprehensively investigate the privacy implication of our advocated concept, with an application on smartwatch password inference and countermeasures. We demonstrate snoopy, a password inference framework which is able to accurately intercept passwords entered on the touchscreens of smartwatches of out-of-set victims, just by eavesdropping on motion sensors. To mitigate this attack, we propose a countermeasure deepauth, which is a second-factor authentication system on smartwatches based on behavioural signatures. We prove that the co-located secondary sensor not only can be maliciously used as a leakage channel, but can be effectively employed as a defence channel as well. All the proposed approaches are comprehensively evaluated through large-scale experiments and the results demonstrate their potential impact in a broad spectrum of identification scenarios.

Precise pose information is a fundamental prerequisite for numerous applications in robotics, AI and mobile computing. Monocular cameras are the ideal sensor for this purpose - they are cheap, lightweight and ubiquitous. As such, monocular visual localization is widely regarded as a cornerstone requirement of machine perception. However, a large gap still exists between the performance that these applications require and that which is achievable through existing monocular perception algorithms. In this thesis we directly tackle the issue of robust egocentric visual localization and mapping through a data-centric approach. As a first major contribution we propose novel learnt models for visual odometry which form the basis of the ego-motion estimates used in later chapters. The proposed approaches are less fragile and much more robust than existing approaches. We present experimental evidence that these approaches can not only approach the accuracy of standard methods but in many cases also show major improvements in computational and memory efficiency. To cope with the drift inherent to the odometry methods, we then introduce a novel learnt spatio-temporal model for performing global relocalization updates. The proposed approach allows one to efficiently infer the global location of an image stream at the fraction of the time of traditional feature-based approaches with minimal loss in localization accuracy. Finally, we present a novel SLAM system integrating our learnt priors for creating 3D maps from monocular image sequences. The approach is designed to harness multiple input sources, including prior depth and ego-motion estimates and incorporates both loop-closure and relocalization updates. The approach, based on the well-established standard visual-inertial structure-from-motion process, allows us to perform accurate posterior inference of camera poses and scene structure to significantly boost the reconstruction robustness and fidelity. Through our qualitative and quantitative experimentation on a wide range of datasets, we conclude that the proposed methods can bring accurate visual localization to a wide class of consumer devices and robotic platforms.

Over the last decade, we have witnessed an unprecedented interest in indoor positioning technologies, with a variety of solutions developed in academic and industrial research labs. Although the field has reached a significant level of maturity, there is still no dominant solution and, as a consequence, positioning services are still lacking in many buildings. In order for a solution to be widely implemented and adopted, two key requirements must be satisfied: low cost and high accuracy. The dichotomy between cost and accuracy has fragmented the technology landscape, leading to a plethora of competing solutions that cannot satisfy both requirements simultaneously. The key objective of this thesis is to investigate how to unify the two disparate camps, providing high positioning accuracy with very low cost. Many approaches have tried to achieve this goal by fusing different sensor modalities. However, the majority of existing work has only investigated how to fuse sequences of measurements for which the associations with the targets are known (i.e. device personal data). Sensor fusion techniques that combine device personal data and anonymous sensor streams (where the associations between the measurements and the targets are not known) remain under-explored as of today. In this thesis, we investigate how to efficiently combine device sensor data and anonymous sensor streams from various sensor modalities in order to build low cost and high accuracy positioning systems. By combining these two types of sensor modalities in one system we see a great potential in designing cost-effective and accurate positioning systems for challenging environments such as for tracking people in highly dynamic industrial settings. Our goal is to design a multi-target multi-sensor tracking framework which will utilise existing sensor infrastructure found in industrial environments and large public buildings (e.g museums) in order to provide reliable positioning services.