Search

In reinforcement learning (RL), an agent learns to solve a task by interacting with its environment. In order to scale these RL agents on high-dimensional complex tasks, recent methods successfully proposed to integrate deep learning methods in RL, creating the field of deep RL (DRL). However, the semantic meaning of a task still derives from an expert reward function that guide the agent in its learning process. This paradigm contrasts with how animals and humans learn: psychology suggests that humans are intrinsically motivated to autonomously acquire knowledge about their environment. In this dissertation, our objective is to investigate how intrinsic motivation can solve the issues experienced by DRL algorithms. First of all, we stress out how intrinsic motivations currently address some issues of DRL algorithms. We classify them, analyze the current limitations of these methods. To highlight their importance, we emphasize that these deadlocks may prevent an agent to learn both increasing complex skills and environment representations. These are key elements to make autonomous agent learn in a more human-like way. Following this analysis, we introduce two new models that can learn diverse and task-specific skills in an end-to-end way. The first one, ELSIM, builds a discrete tree of skills in the direction of the feedbacks of the environment. Our results show that this learning paradigm favors exploration in sparse-rewards environments and allows to reuse skills over different correlated tasks. We highlight the drawbacks of ELSIM and propose a novel model, DisTop, that tackle them. It progressively builds a discrete topology of the environment using an unsupervised contrastive loss, a self-organizing network and a goal-conditioned policy. Then, DisTop can smartly control which skill to improve or forget. This way, DisTop competes with state-of-art algorithms on three different benchmarks, including a hierarchical environment with sparse rewards. To conclude the dissertation, we discuss the outlooks of the domain and future directions for research., En apprentissage par renforcement (RL), un agent apprend à résoudre une tâche en interagissant avec son environnement. Afin de faire passer à l’échelle ces agents sur des tâches complexes, les méthodes récentes ont proposé avec succès d’intégrer les méthodes d’apprentissage profond au RL, créant le domaine d’apprentissage profond par renforcement (DRL). Cependant, la signification sémantique d’une tâche est toujours pourvue par une fonction de récompense experte qui guide l’agent dans son processus d’apprentissage. Ce paradigme contraste avec la manière dont les animaux et humains apprennent: les travaux de psychologie suggèrent que les humains sont intrinsèquement motivés à acquérir de nouvelles connaissances à propos de leur environnement. Dans cette thèse, notre objectif est d’étudier comment la motivation intrinsèque permet de résoudre les problèmes expérimentés par le DRL. Tout d’abord, nous mettons en évidence comment les motivations intrinsèques actuelles attaquent certains problèmes du DRL. Nous classifions et formalisons les méthodes, puis analysons leurs limites. Afin d’exhiber leur importance, nous mettons en avant que ces verrous peuvent empêcher un agent d’apprendre des compétences et représentations de l’environnement de plus en plus complexes. Ce sont des éléments-clés pour faire apprendre des agents autonomes comme des humains. Á partir de cette analyse, nous introduisons deux nouveaux modèles qui peuvent apprendre des compétences diverses et spécifiques à une tâche de bout en bout. Le premier, ELSIM, construit un arbre discret de compétences dans la direction des récompenses de l’environnement. Nos résultats montrent que ce paradigme d’apprentissage améliore l’exploration dans des environnements avec des récompenses éparses et permet d’utiliser des compétences sur différentes tâches corrélées. Nous mettons en avant les inconvénients d’ELSIM et proposons un autre modèle, DisTop, pour les corriger. DisTop construit progressivement une topologie de l’environnement en utilisant une fonction de coût contrastive, un réseau auto-organisé et une politique dépendante d’objectifs. L’agent peut alors intelligemment contrôler quelles compétences apprendre ou oublier. De cette manière, DisTop est compétitif avec des algorithmes de l’état de l’art sur trois types de tâches différentes, incluant une tâche hiérarchique avec des récompenses éparses. Pour conclure la thèse, nous discutons des perspectives du domaines et des directions futures de notre recherche.

Reinforcement learning (RL) is today more popular than ever, but certain basic skills are still out of reach of this paradigm: object manipulation, sensorimotor control, natural interaction with other agents. A possible approach to address these challenges consist in taking inspiration from human development, or even trying to reproduce it. In this thesis, we study the intersection of two crucial topics in developmental sciences and how to apply them to RL in order to tackle the aforementioned challenges: interactive learning and intrinsic motivation. Interactive learning and intrinsic motivation have already been studied, separately, in combination with RL, but in order to improve quantitatively existing agents performances, rather than to learn in a developmental fashion. We thus focus our efforts on the developmental aspect of these subjects. Our work touches the self-organisation of learning in developmental trajectories through an intrinsically motivated for learning progress, and the interaction of this organisation with goal-directed learning and imitation learning. We show that these mechanisms, when implemented in open-ended environments with no task predefined, can interact to produce learning behaviors that are sound from a developmental standpoint, and richer than those produced by each mechanism separately.; L'apprentissage par renforcement est aujourd'hui plus populaire que jamais, mais plusieurs compétences simples lui restent hors de portée: manipulation d'objets, contrôle sensorimoteur, interaction naturelle avec d'autres agents. Une approche possible pour aborder ces défis consiste à s'inspirer du développement humain, voire de tenter de le reproduire. Dans cette thèse, nous étudions l'intersection de deux sujets cruciaux en sciences du développement, et leur application à l'apprentissage par renforcement dans le but d'aborder ces défis: l'apprentissage social et la motivation intrinsèque. L'interaction et la motivation intrinsèque ont déjà été étudiées, séparément, en combinaison avec l'apprentissage par renforcement, mais avec l'objectif d'améliorer les performances d'agents existants plutôt que d'apprendre de manière développementale. Nous concentrons donc à l'inverse notre étude sur l'aspect développemental de ces deux sujets. Nos travaux de thèse abordent en particulier l'auto-organisation de l'apprentissage en trajectoires développementale par recherche intrinsèquement motivée du progrès d'apprentissage, et l'interaction de cette organisation avec l'apprentissage dirigé vers des buts et l'apprentissage par imitation. Nous montrons que ces différents mécanismes, lorsqu'ils sont mis en place dans des environnements ouverts sans tâche prédéfinie, peuvent interagir pour produire des comportements d'apprentissage satisfaisants d'un point de vue développemental, et plus riches que ceux produits par chaque mécanisme séparément: génération automatique de curriculum, imitation sélective, suivi non-supervisé de démonstrations.

Reinforcement learning (RL) is today more popular than ever, but certain basic skills are still out of reach of this paradigm: object manipulation, sensorimotor control, natural interaction with other agents. A possible approach to address these challenges consist in taking inspiration from human development, or even trying to reproduce it. In this thesis, we study the intersection of two crucial topics in developmental sciences and how to apply them to RL in order to tackle the aforementioned challenges: interactive learning and intrinsic motivation. Interactive learning and intrinsic motivation have already been studied, separately, in combination with RL, but in order to improve quantitatively existing agents performances, rather than to learn in a developmental fashion. We thus focus our efforts on the developmental aspect of these subjects. Our work touches the self-organisation of learning in developmental trajectories through an intrinsically motivated for learning progress, and the interaction of this organisation with goal-directed learning and imitation learning. We show that these mechanisms, when implemented in open-ended environments with no task predefined, can interact to produce learning behaviors that are sound from a developmental standpoint, and richer than those produced by each mechanism separately.; L'apprentissage par renforcement est aujourd'hui plus populaire que jamais, mais plusieurs compétences simples lui restent hors de portée: manipulation d'objets, contrôle sensorimoteur, interaction naturelle avec d'autres agents. Une approche possible pour aborder ces défis consiste à s'inspirer du développement humain, voire de tenter de le reproduire. Dans cette thèse, nous étudions l'intersection de deux sujets cruciaux en sciences du développement, et leur application à l'apprentissage par renforcement dans le but d'aborder ces défis: l'apprentissage social et la motivation intrinsèque. L'interaction et la motivation intrinsèque ont déjà été étudiées, séparément, en combinaison avec l'apprentissage par renforcement, mais avec l'objectif d'améliorer les performances d'agents existants plutôt que d'apprendre de manière développementale. Nous concentrons donc à l'inverse notre étude sur l'aspect développemental de ces deux sujets. Nos travaux de thèse abordent en particulier l'auto-organisation de l'apprentissage en trajectoires développementale par recherche intrinsèquement motivée du progrès d'apprentissage, et l'interaction de cette organisation avec l'apprentissage dirigé vers des buts et l'apprentissage par imitation. Nous montrons que ces différents mécanismes, lorsqu'ils sont mis en place dans des environnements ouverts sans tâche prédéfinie, peuvent interagir pour produire des comportements d'apprentissage satisfaisants d'un point de vue développemental, et plus riches que ceux produits par chaque mécanisme séparément: génération automatique de curriculum, imitation sélective, suivi non-supervisé de démonstrations.

International audience; Most AI algorithms consider input data as "percepts" that the agent receives from the environment. Constructivist epistemology, however, suggests an alternative approach that considers the algorithm's input data as feedback resulting from the agent's actions. This paper introduces a constructivist algorithm to let an agent learn regularities of actions and feedback. The agent organizes its behaviors to fulfill a form of intentionality defined independently of a specific task. The experiment shows that this algorithm constructs a Petri net whose nodes represent hypothetical stable states afforded by the agent/environment coupling, and arcs represent transitions between such states. Since this Petri net allows the algorithm to predict the consequences of the agent's actions, we argue that it constitutes a rudimentary causal model of the "world" (agent+environment) learned by the agent through experience of interaction. This work opens the way to studying how an autonomous agent car learn more complex causal models of more complex worlds, in particular by explaining regularities of interaction through the presence of objects in the agent's surrounding space.

International audience; This paper reports preliminary research work on applying developmental learning to social robotics for making human-robot interactions more instinctive and more natural. Developmental learning is an unsupervised learning strategy relying on the fact that the learning agent is intrinsically motivated, and is able to incrementally build its own representation of the world through its experiences of interaction with it. Our claim is that using developmental learning in social robots could dramatically change the way we envision human-robot interaction, notably by giving the robot an active role in the interaction building process, and even more importantly, in the way it autonomously learns suitable behaviors over time. Developmental learning appears to be an appropriate approach to develop a form of "interactional intelligence" for social robots. In this work, our goal was to set up a common framework for implementing, experimenting and evaluating developmental learning algorithms with various social robots.

Little AI is a pedagogical game aimed at presenting the founding concepts of constructivist learning and developmental Artificial Intelligence. It primarily targets students in computer science and cognitive science but it can also interest the general public curious about these topics. It requires no particular scientific background; even children can find it entertaining. Professors can use it as a pedagogical resource in class or in online courses. The player presses buttons to control a simulated “baby robot”. The player cannot see the robot and its environment, and initially ignores the effects of the commands. The only information received by the player is feedback from the player’s commands. The player must learn, at the same time, the functioning of the robot’s body and the structure of the environment from patterns in the stream of commands and feedback. We argue that this situation is analogous to how infants engage in early-stage developmental learning (e.g., Piaget (1937), [1]). Keywords: Pedagogical game, Artificial intelligence, Developmental learning, Constructivist learning

International audience; We report lessons learned from our experience creating and administering a MOOC introducing students to developmental artificial intelligence. Our course, entitled Implementation of DEvelopmentAl Learning (IDEAL), covered advanced findings in developmental artificial intelligence spanning the fields of cognitive science and computer science. Its objectives consisted of conveying these findings to the general public, teaching developmental AI techniques to programmers and roboticists, and supporting international multidisciplinary collaboration amongst actors in the domain (academics, industrials, and hobbyists). Teaching materials included textual descriptions, short videos, and programming and writing activities. The course also supported community forums that allowed participants to engage in debates and work in teams. This form proved to be well fitted to the objectives. Participants reported that they found this design more efficient than lectures; and the forums proved useful in creating a productive community and conveying advanced scientific ideas.

International audience; We propose an architecture for self-motivated agents allowing them to construct their own knowledge of objects and of geometrical properties of space through interaction with their environment. Self-motivation is defined here as a tendency to experiment and to respond to behavioral opportunities afforded by the environment. Interactions have predefined valences that specify inborn behavioral preferences. The long-term goal is to design agents that construct their own knowledge of their environment through experience, rather than exploiting pre-coded knowledge. Over time, the agent learns relations between elements of the environment that afford its interactions, and its perception of these elements, in the form of data structures called signatures of interactions. These signatures allow the agent to attribute a low level semantics to elements that constitute its environment based on valences of interactions, without predefined knowledge about these elements and regardless of the number of element types. Signatures of interaction are then used to localize elements in space and to construct data structures that characterize spatial properties of space, called signatures of places and signatures of presence. Signatures of place and of presence characterize space using interactions rather than geometrical or topological properties. The agent uses these structures to maintain an egocentric representation of affordances of the surrounding environment, without any preconception about the elements that compose the environment, and without using notions of geometrical space. Experiments with simulated agents show that they learn to behave in their environment, taking into account multiple surrounding objects, reaching or avoiding objects according to the valence of the interactions that they afford.

International audience; In the context of visual object search and localization, saliency maps provide an efficient way to find object candidates in images. Unlike most approaches, we propose a way to learn saliency maps directly on a robot, by exploring the environment, discovering salient objects using geometric cues, and learning their visual aspects. More importantly, we provide an autonomous exploration strategy able to drive the robot for the task of learning saliency. For that, we describe the Reinforcement Learning-Intelligent Adaptive Curiosity algorithm (RL-IAC), a mechanism based on IAC (Intelligent Adaptive Curiosity) able to guide the robot through areas of the space where learning progress is high, while minimizing the time spent to move in its environment without learning. We demonstrate first that our saliency approach is an efficient tool to generate relevant object boxes proposal in the input image and significantly outperforms state-of-the-art algorithms. Second, we show that RL-IAC can drastically decrease the required time for learning saliency compared to random exploration.

International audience; We present an architecture for self-motivated agents to generate behaviors in a dynamic environment according to its possibilities of interactions. Some interactions have predefined valences that specify inborn behavioral preferences. Over time, the agent learns to recognize affordances in its surrounding environment under the form of structures called signatures of interactions. The agent keeps track of enacted interactions in a spatial memory to generate a completed context in which it can use signatures to recognize and localize distant possibilities of interactions, and generates behaviors that satisfy its motivation principles.

This thesis is a part of the IDEAL project (Implementing DEvelopmentAl Learning) funded by the Agence Nationale de la Recherche (ANR). The ability of perceiving, memorizing and interpreting the surrounding environment is a vital ability found in numerous living beings. This ability allows them to generate context adapted behaviors, or escaping from a predator that escape from their sensory system. The objective of this thesis consists in implementing such a capacity in artificial agents. We propose a theoretical model that allows artificial agent to generate a usable knowledge of elements that compose its environment and a structure able to characterize the structure of surrounding space. This model is based on the sensorimotor contingency theory, and implements a form of intrinsic motivation. Indeed, this model begin with a set of indivisible structures, called interactions, that characterize the interaction possibilities between the agent and its environment. The learning is developmental and emerges from the interaction that occurs between the agent and the environment, without the need of any external intervention (like reward). Our model propose a set of mechanisms that allow to organize and exploit emerging knowledge in order to generate behaviors. We propose implementations of our model to demonstrate the emerging knowledge based on agent-environment interaction, and behaviors that can emerge from this knowledge, Cette thèse s'inscrit dans le cadre du projet IDEAL (Implementing DevelopmentAl Learning) financé par l'Agence Nationale de la Recherche (ANR). La capacité à percevoir, mémoriser et interpréter l'environnement qui nous entoure est une capacité vitale que l'on retrouve chez de nombreux êtres vivants. Cette capacité leur permet de générer des comportements adaptés à leur contexte, ou d'échapper à un prédateur sorti de leur champ de vision. L'objectif de cette thèse consiste à doter un agent artificiel de cette capacité. Nous proposons un modèle théorique permettant à un agent artificiel de générer des connaissances exploitables des éléments constituant son environnement et une structure reflétant l'espace. Ce modèle est basé sur la théorie de la contingence sensorimotrice, et implémente une forme de motivation intrinsèque. En effet, ce modèle débute avec un ensemble de structure indivisibles, appelées interactions, caractérisant les échanges entre l'agent et son environnement. L'apprentissage des connaissances est développemental et émerge de l'interaction entre l'agent et son environnement, sans qu'aucune intervention extérieure (récompense), ne soit nécessaire. Notre modèle propose des mécanismes permettant d'organiser et d'exploiter ces connaissances émergentes dans le but de générer des comportements. Nous proposons des implémentations de ce modèle pour démontrer l'émergence d'une connaissance à partir de l'interaction entre l'agent et son environnement, et les comportements qui émergent de cette connaissance

International audience; The scientific, technological, and application challenges that arise from the mutual interaction of developmental robotics and computational human behavior understanding give rise to two different perspectives. Robots need to be capable to learn dynamically and incrementally how to interpret, and thus understand multimodal human behavior, which means behavior analysis can be performed for developmental robotics. On the other hand, behavior analysis can also be performed through developmental robotics, since developmental social robots can offer stimulating opportunities for improving scientific understanding of human behavior, and especially to allow a deeper analysis of the semantics and structure of human behavior. The contributions to the Special Issue explore these two perspectives.

International audience; Robotics is evolving today toward applications in which robots enter into everyday life and thus need to be able to interact with non-engineers; robots that assist people with disabilities in their homes are an example [5]. For such applications, robots must be able to manipulate and use objects in the human environment that cannot be known and modelled beforehand by engineers. For reasons of safety and energy efficiency, the use of soft materials and soft actuators is flourishing in these applications [12]; difficult control problems arise, however, because the dynamics of these soft materials are difficult to model analytically. For these reasons, a crucial goal has become the development of learning methods that allow robots to acquire models of the dynamics of their own bodies and their interactions with new objects.