Search

Swarm robotics, inspired by swarms in nature, has great potential. The resilience, scalability, robustness and redundancy of having many robots collectively perform tasks such as mapping, disaster recovery, pollution control, and cleaning make for a compelling vision. To achieve this, we need to design swarm robot systems to have a desired collective behaviour, but the design of controllers for the individual robots of a swarm such that this behaviour emerges from the interaction of the individual robots is difficult. Current solutions often use off-line automatic discovery by artificial evolution of robot controllers, which are then transferred into the swarm. This is problematic for two important reasons. Firstly, since there is the need for additional supporting infrastructure, both to evolve the new controllers and to communicate them to the swarm, the swarm is not self-sufficient. Secondly, the evolved controllers are often opaque and hard to understand, an important consideration for safety and explainability reasons. In this work we tackle both of these issues. We build a swarm of robots with very high computing performance using recently available mobile computation devices. This high performance allows us to move the evolutionary process, dependent on processing power for simulation, into the swarm. Because the computational power of the swarm grows with the size of the swarm, it is both autonomous and scalable. We use behaviour trees as the individual robot controller architecture. They are modular, hierarchical and human readable. By developing automatic tools to simplify large evolved trees, we can understand, explain, and even improve the evolved controllers. By moving the evolutionary process into the swarm, and by using understandable controllers, we make the swarm autonomous, scalable, and understandable, necessary steps towards their real-world deployment.

This thesis presents two novel algorithms for the evolutionary optimisation of agent populations through divergent search of low dimensional phenotypic landscapes. As the eld of Evolutionary Robotics (ER) develops towards more complex domains, which often involve deception and uncertainty, the promotion of phenotypic diversity has become of increasing interest. Divergent exploration of the phenotypic feature space has been shown to avoid convergence towards local optima and to provide diverse sets of solutions to a given objective. Novelty Search (NS) and the more recent Multi-dimensional Archive of Phenotypic Elites (MAP-Elites), are two state of the art algorithms which utilise divergent phenotypic search. In this thesis, the individual merits and weaknesses of these algorithms are built upon in order to further develop the study of divergent phenotypic search within ER. An observation that the diverse range of individuals produced through the optimisation of novelty will likely contain solutions to multiple independent objectives is utilised to develop Multiple Assessment Directed Novelty Search (MADNS). The MADNS algorithm is introduced as an extension to NS for the simultaneous optimisation of multiple independent objectives, and is shown to become more e ective than NS as the size of the state space increases. The central contribution of this thesis is the introduction of a novel algorithm for rapid and thorough divergent search of low dimensional phenotypic landscapes. The Spatial, Hierarchical, Illuminated NeuroEvolution (SHINE) algorithm di ers from previous divergent search algorithms, in that it utilises a tree structure for the maintenance and selection of potential candidates. Unlike previous approaches, SHINE iteratively focusses upon sparsely visited areas of the phenotypic landscape without the computationally expensive distance comparison required by NS; rather, the sparseness of the area within the landscape where a potential solution resides is inferred through its depth within the tree. Experimental results in a range of domains show that SHINE signi cantly outperforms NS and MAP-Elites in both performance and exploration.

In this thesis, the evolution of adaptive behaviour in artificial agents is studied. More specifically, two types of adaptive behaviours are studied: articulated and cognitive ones. Chapter 1 presents a general introduction together with a brief presentation of the research area of this thesis, its main goals and a brief overview of the experimental studies done, the results and conclusions obtained. On chapter 2, I briefly present some promising methods that automatically generate robot controllers and/or body plans and potentially could help in the development of adaptive robots. Among these methods I present in details evolutionary robotics, a method inspired on natural evolution, and the biological background regarding adaptive behaviours in biological organisms, which provided inspiration for the studies presented in this thesis. On chapter 3, I present a detailed study regarding the evolution of articulated behaviours, i.e., behaviours that are organized in functional sub-parts, and that are combined and used in a sequential and context-dependent way, regardless if there is a structural division in the robot controller or not. The experiments performed with a single goal task, a cleaning task, showed that it is possible to evolve articulated behaviours even in this condition and without structural division of the robot controller. Also the analysis of the results showed that this type of integrated modular behaviours brought performance advantages compared to structural divided controllers. Analysis of robots' behaviours helped to clarify that the evolution of this type of behaviour depended on the characteristics of the neural network controllers and the robot's sensorimotor capacities, that in turn defined the capacity of the robot to generate opportunity for actions, which in psychological literature is often called affordances. In chapter 4, a study seeking to understand the role of reactive strategies in the evolution of cognitive solutions, i.e. those capable of integrating information over time encoding it on internal states that will regulate the robot's behaviour in the future, is presented. More specifically I tried to understand whether the existence of sub-optimal reactive strategies prevent the development of cognitive solutions, or they can promote the evolution of solutions capable of combining reactive strategies and the use of internal information for solving a response delayed task, the double t-maze. The results obtained showed that reactive strategies capable of offloading cognitive work to the agent/environmental relation can promote, rather than prevent the evolution of solutions relying on internal information. The analysis of these results clarified how these two mechanisms interact producing a hybrid superior and robust solution for the delayed response task.

In this thesis I report the development of FARSA (Framework for Autonomous Robotics Simulation and Analysis), a simulation tool for the study of the interaction between language and action in cognitive robots and more in general for experiments in embodied cognitive science. Before presenting the tools, I will describe a series of experiments that involve simulated humanoid robots that acquire their behavioural and language skills autonomously through a trial-and-error adaptive process in which random variations of the free parameters of the robots’ controller are retained or discarded on the basis of their effect on the overall behaviour exhibited by the robot in interaction with the environment. More specifically the first series of experiments shows how the availability of linguistic stimuli provided by a caretaker, that indicate the elementary actions that need to be carried out in order to accomplish a certain complex action, facilitates the acquisition of the required behavioural capacity. The second series of experiments shows how a robot trained to comprehend a set of command phrases by executing the corresponding appropriate behaviour can generalize its knowledge by comprehending new, never experienced sentences, and by producing new appropriate actions. Together with their scientific relevance, these experiments provide a series of requirements that have been taken into account during the development of FARSA. The objective of this project is that to reduce the complexity barrier that currently discourages part of the researchers interested in the study of behaviour and cognition from initiating experimental activity in this area. FARSA is the only available tools that provide an integrated framework for carrying on experiments of this type, i.e. it is the only tool that provides ready to use integrated components that enable to define the characteristics of the robots and of the environment, the characteristics of the robots’ controller, and the characteristics of the adaptive process. Overall this enables users to quickly setup experiments, including complex experiments, and to quickly start collecting results.

The work presented herein focuses on the design of distributed autonomous controllers for collective behaviour of Micro-unmanned Aerial Vehicles (MAVs). Two alternative approaches to this topic are introduced: one based upon the Evolutionary Robotics (ER) paradigm, the other one upon flocking principles. Three computer simulators have been developed in order to carry out the required experiments, all of them having their focus on the modelling of fixed-wing aircraft flight dynamics. The employment of fixed-wing aircraft rather than the omni-directional robots typically employed in collective robotics significantly increases the complexity of the challenges that an autonomous controller has to face. This is mostly due to the strict motion constraints associated with fixed-wing platforms, that require a high degree of accuracy by the controller. Concerning the ER approach, the experimental setups elaborated have resulted in controllers that have been evolved in simulation with the following capabilities: (1) navigation across unknown environments, (2) obstacle avoidance, (3) tracking of a moving target, and (4) execution of cooperative and coordinated behaviours based on implicit communication strategies. The design methodology based upon flocking principles has involved tests on computer simulations and subsequent experimentation on real-world robotic platforms. A customised implementation of Reynolds’ flocking algorithm has been developed and successfully validated through flight tests performed with the swinglet MAV. It has been notably demonstrated how the Evolutionary Robotics approach could be successfully extended to the domain of fixed-wing aerial robotics, which has never received a great deal of attention in the past. The investigations performed have also shown that complex and real physics-based computer simulators are not a compulsory requirement when approaching the domain of aerial robotics, as long as proper autopilot systems (taking care of the ”reality gap” issue) are used on the real robots.

The attempt to evolve complete embodied and situated artificial creatures in which both morphological and control characteristics are adapted during the evolutionary process has been and still represents a long term goal key for the artificial life and the evolutionary robotics community. Loosely inspired by ancient biological organisms which are not provided with a central nervous system and by simple organisms such as stick insects, this thesis proposes a new genotype encoding which allows development and evolution of mor- phology and neural controller in artificial agents provided with a distributed neural network. In order to understand if this kind of network is appropriate for the evolution of non trivial behaviours in artificial agents, two experiments (description and results will be shown in chapter 3) in which evolution was applied only to the controller’s parameters were performed. The results obtained in the first experiment demonstrated how distributed neural networks can achieve a good level of organization by synchronizing the output of oscillatory elements exploiting acceleration/deceleration mechanisms based on local interactions. In the second experiment few variants on the topology of neural architecture were introduced. Results showed how this new control system was able to coordinate the legs of a simulated hexapod robot on two different gaits on the basis of the external circumstances. After this preliminary and successful investigation, a new genotype encoding able to develop and evolve artificial agents with no fixed morphology and with a distributed neural controller was proposed. A second set of experiments was thus performed and the results obtained confirmed both the effectiveness of genotype encoding and the ability of distributed neural network to perform the given task. The results have also shown the strength of genotype both in generating a wide range of different morphological structures and in favouring a direct co-adaptation between neural controller and morphology during the evolutionary process. Furthermore the simplicity of the proposed model has showed the effective role of specific elements in evolutionary experiments. In particular it has demonstrated the importance of the environment and its complexity in evolving non-trivial behaviours and also how adding an independent component to the fitness function could help the evolutionary process exploring a larger space solutions avoiding a premature convergence towards suboptimal solutions.

The ability of a robot to adapt in-mission to achieve an assigned goal is highly desirable. This thesis project places an emphasis on employing learning-based intelligent control methodologies to the development and implementation of an autonomous unmanned aerial vehicle (UAV). Flight control is carried out by evolving spiking neural networks (SNNs) with Hebbian plasticity. The proposed implementation is capable of learning and self-adaptation to model variations and uncertainties when the controller learned in simulation is deployed on a physical platform. Controller development for small multicopters often relies on simulations as an intermediate step, providing cheap, parallelisable, observable and reproducible optimisation with no risk of damage to hardware. Although model-based approaches have been widely utilised in the process of development, loss of performance can be observed on the target platform due to simplification of system dynamics in simulation (e.g., aerodynamics, servo dynamics, sensor uncertainties). Ignorance of these effects in simulation can significantly deteriorate performance when the controller is deployed. Previous approaches often require mathematical or simulation models with a high level of accuracy which can be difficult to obtain. This thesis, on the other hand, attempts to cross the reality gap between a low-fidelity simulation and the real platform. This is done using synaptic plasticity to adapt the SNN controller evolved in simulation to the actual UAV dynamics. The primary contribution of this work is the implementation of a procedural methodology for SNN control that integrates bioinspired learning mechanisms with artificial evolution, with an SNN library package (i.e. eSpinn) developed by the author. Distinct from existing SNN simulators that mainly focus on large-scale neuron interactions and learning mechanisms from a neuroscience perspective, the eSpinn library draws particular attention to embedded implementations on hardware that is applicable for problems in the robotic domain. This C++ software package is not only able to support simulations in the MATLAB and Python environment, allowing rapid prototyping and validation in simulation; but also capable of seamless transition between simulation and deployment on the embedded platforms. This work implements a modified version of the NEAT neuroevolution algorithm and leverages the power of evolutionary computation to discover functional controller compositions and optimise plasticity mechanisms for online adaptation. With the eSpinn software package the development of spiking neurocontrollers for all degrees of freedom of the UAV is demonstrated in simulation. Plastic height control is carried out on a physical hexacopter platform. Through a set of experiments it is shown that the evolved plastic controller can maintain its functionality by self-adapting to model changes and uncertainties that take place after evolutionary training, and consequently exhibit better performance than its non-plastic counterpart.

The current state of robotics relies largely on hand designed morphologies and controllers. This paradigm of robotics is well suited for controlled and static environments like warehouses or factory floors, but this type of robot often fails to extrapolate to autonomous behaviors in unpredictable and dynamic environments. In contrast, biological animals have evolved to seamlessly interact with the uncertainty of the real world. They accomplish this feat, in part, through specialized and complex morphologies that employ compliant materials. In this work, I explore the interactions of autonomous embodied agents’ brains and bodies with each other, and with the outside environment, through the evolution of soft robot morphologies and controllers. These interactions are first explored by evolving robots that perform complex and effective behaviors without high-level controllers in order to demonstrate the potential of morphological computation in compliant bodies. The study of morphological computation is further explored by also demonstrating effective behavior in tasks which are unapproachable with traditional rigid body robots (like squeezing and folding oneself). The focus on morphologically-driven behaviors is extended by evolving soft robots with neural-esque spiking muscles and demonstrating the optimization of physically embodied information pathways, exemplify the continuum between morphologies and controllers in embodied systems. I then turn to the simultaneous optimization of complex morphologies and high-level controllers, using the theory of embodied cognition to hypothesize that the specialization of morphologies and controllers to one another has been hindering the evolution of complex embodied machines. Results here demonstrate that a proposed algorithm for “morphological innovation protection”, which temporarily reduces selection pressure on newly mutated morphologies to enable readaptation of the coupled brain-body systems, produces significantly more fit robots and allows for their sustained optimization over evolutionary time. Generalizing the above methods, the design automation techniques employed here also are applied to problems outside of soft robots – demonstrating the optimization of object topologies towards a desired mechanical resonance. I hope that the work described in this dissertation will help to inform the automated design of embodied machines, like robots, for engineering applications, while also contributing to the fundamental and general understanding of embodied intelligent agents, and their evolution in natural systems.

The control of insect-sized flapping-wing micro air vehicles is attracting increasing interest. Solution of the problem requires construction of a controller that is physically small, extremely power efficient, and capable. In addition, process variation in the creation of very small wings and armatures as well as the potential for accumulating damage and wear over the course of a vehicle's lifetime suggest that controllers be able to self-adapt to the specific and possibly changing nature of the vehicles in which they are embedded. Previous work with Evolvable Hardware Continuous Time Recurrent Neural Networks (CTRNNs) as applied to adaptive control of walking in legged robots suggests that CTRNNs may provide a suitable control solution for flapping-wing micro air vehicles. However, upon complete analysis, it can be seen that perceived similarities between the two problems are somewhat superficial, and that flapping-wing vehicle control requires its own study. This dissertation constitutes the first attempt to apply evolved CTRNN devices to the control of a feasible flapping-wing micro air vehicle. It is organized as a sequence of control experiments of increasing difficulty and explores the following issues, development of behavior-based analog circuit modules, architectures to combine those modules into multi-functional controllers, low-level circuit analyses to explain how evolved modules operate and interact. Also included are experiments in the creation of physically polymorphic behavior modules that combine multiple flight functions into a monolithic analog device. In addition to providing first-of-its-kind feasibility results, this dissertation develops a new frequency-grouping based analysis method to explain the operation of evolved devices.

Unmanned aerial vehicles (UAVs) have become increasingly popular for many applications, including search and rescue, surveillance, and electronic warfare, but almost all UAVs are controlled remotely by humans. Methods of control must be developed before UAVs can become truly autonomous. While the field of evolutionary robotics (ER) has made strides in using evolutionary computation (EC) to develop controllers for wheeled mobile robots, little attention has been paid to applying EC to UAV control. EC is an attractive method for developing UAV controllers because it allows the human designer to specify the set of high level goals that are to be solved by artificial evolution. In this research, autonomous navigation controllers were developed using multi-objective genetic programming (GP) for fixed wing UAV applications. Four behavioral fitness functions were derived from flight simulations. Multi-objective GP used these fitness functions to evolve controllers that were able to locate an electromagnetic energy source, to navigate the UAV to that source efficiently using on-board sensor measurements, and to circle around the emitter. Controllers were evolved in simulation. To narrow the gap between simulated and real controllers, the simulation environment employed noisy radar signals and a sensor model with realistic inaccuracies. All computations were performed on a 92-processor Beowulf cluster parallel computer. To gauge the success of evolution, baseline fitness values for a successful controller were established by selecting values for a minimally successful controller. Two sets of experiments were performed, the first evolving controllers directly from random initial populations, the second using incremental evolution. In each set of experiments, autonomous navigation controllers were evolved for a variety of radar types. Both the direct evolution and incremental evolution experiments were able to evolve controllers that performed acceptably. However, incremental evolution vastly increased the success rate of incremental evolution over direct evolution. The final incremental evolution experiment on the most complex radar investigated in this research evolved controllers that were able to handle all of the radar types. Evolved UAV controllers were successfully transferred to a wheeled mobile robot. An acoustic array on-board the mobile robot replaced the radar sensor, and a speaker emitting a tone was used as the target. Using the evolved navigation controllers, the mobile robot moved to the speaker and circled around it. Future research will include testing the best evolved controllers by using them to fly real UAVs.

Evolutionary Robotics (ER) is a field of research that applies evolutionary computing methods to the automated design and synthesis of behavioral robotics controllers. In the general case, reinforcement learning (RL) using high-level task performance feedback is applied to the evolution of controllers for autonomous mobile robots. This form of RL learning is required for the evolution of complex and non-trivial behaviors because a direct error-feedback signal is generally not available. Only the high-level behavior or task is known, not the complex sensor-motor signal mappings that will generate that behavior. Most work in the field has used evolutionary neural computing methods. Over the course of the preceding decade, ER research has been largely focused on proof-of-concept experiments. Such work has demonstrated both the evolvablility of neural network controllers and the feasibility of implementation of those evolved controllers on real robots. However, these proof-of-concept results leave important questions unanswered. In particular, no ER work to date has shown that it is possible to evolve complex controllers in the general case. The research described in this work addresses issues relevant to the extension of ER to generalized automated behavioral robotics controller synthesis. In particular, we focus on fitness selection function specification. The case is made that current methods of fitness selection represent the primary factor limiting the further development of ER. We formulate a fitness function that accommodates the Bootstrap Problem during early evolution, but that limits human bias in selection later in evolution. In addition, we apply ER methods to evolve networks that have far more inputs, and are of a much greater complexity than those used in other ER work. We focus on the evolution of robot controllers for the competitive team game Capture the Flag. Games are played in a variety of maze environments. The robots use processed video data requiring 150 or more neural network inputs for sensing of their environment. The evolvable artificial neural network (ANN) controllers are of a general variable-size architecture that allows for arbitrary connectivity. Resulting evolved ANN controllers contain on the order of 5000 weights. The evolved controllers are tested in competitions of 240 games against hand-coded knowledge-based controllers. Results show that evolved controllers are competitive with the knowledge-based controllers and can win a modest majority of games in a large tournament in a challenging world configuration.

The automatic synthesis of embodied creatures through artificial evolution has become a key area of research in robotics, artificial life and the cognitive sciences. However, the research has mainly focused on genetic encodings and fitness functions. Considerably less has been said about the role of controllers and how they affect the evolution of morphologies and behaviors in artificial creatures. Furthermore, the evolutionary algorithms used to evolve the controllers and morphologies are pre-dominantly based on a single objective or a weighted combination of multiple objectives, and a large majority of the behaviors evolved are for wheeled or abstract artifacts. In this thesis, we present a systematic study of evolving artificial neural network (ANN) controllers for the legged locomotion of embodied organisms. A virtual but physically accurate world is used to simulate the evolution of locomotion behavior in a quadruped creature. An algorithm using a self-adaptive Pareto multi-objective evolutionary optimization approach is developed. The experiments are designed to address five research aims investigating: (1) the search space characteristics associated with four classes of ANNs with different connectivity types, (2) the effect of selection pressure from a self-adaptive Pareto approach on the nature of the locomotion behavior and capacity (VC-dimension) of the ANN controller generated, (3) the effciency of the proposed approach against more conventional methods of evolutionary optimization in terms of computational cost and quality of solutions, (4) a multi-objective approach towards the comparison of evolved creature complexities, (5) the impact of relaxing certain morphological constraints on evolving locomotion controllers. The results showed that: (1) the search space is highly heterogeneous with both rugged and smooth landscape regions, (2) pure reactive controllers not requiring any hidden layer transformations were able to produce sufficiently good legged locomotion, (3) the proposed approach yielded competitive locomotion controllers while requiring significantly less computational cost, (4) multi-objectivity provided a practical and mathematically-founded methodology for comparing the complexities of evolved creatures, (5) co-evolution of morphology and mind produced significantly different creature designs that were able to generate similarly good locomotion behaviors. These findings attest that a Pareto multi-objective paradigm can spawn highly beneficial robotics and virtual reality applications.