Search

This thesis addresses the topic of grounding in human-robot interaction, that is, the process by which the human and robot can ensure mutual understanding. To explore this topic, the scenario of a robot holding a presentation to a human audience is used, where the robot has to process multimodal feedback from the human in order to adapt the presentation to the human's level of understanding. First, the use of behaviour trees to model real-time interactive processes of the presentation is addressed. A system based on the behaviour tree architecture is used in a semi-automated Wizard-of-oz experiment, showing that audience members prefer an adaptive system to a non-adaptive alternative. Next, the thesis addresses the use of knowledge graphs to represent the content of the presentation given by the robot. By building a small, local knowledge graph containing properties (edges) that represent facts about the presentation, the system can iterate over that graph and consistently find ways to refer to entities by referring to previously grounded content. A system based on this architecture is implemented, and an evaluation using simulated users is presented. The results show that crowdworkers comparing different adaptation strategies are sensitive to the types of adaptation enabled by the knowledge graph approach. In a face-to-face presentation setting, feedback from the audience can potentially be expressed through various modalities, including speech, head movements, gaze, facial gestures and body pose. The thesis explores how such feedback can be automatically classified. A corpus of human-robot interactions is annotated, and models are trained to classify human feedback as positive, negative or neutral. A relatively high accuracy is achieved by training simple classifiers with signals found mainly in the speech and head movements. When knowledge graphs are used as the underlying representation of the system's presentation, some consistent way of generating text, that c, Denna avhandling behandlar ämnet multimodal kommunikativ grundning (grounding) mellan robotar och människor. Detta är processen för hur en människa och en robot kan säkerställa att de har en gemensam förståelse. För att utforska detta ämne ämne, används ett scenario där en robot håller en presentation för en mänsklig publik. Roboten måste analysera multimodala signaler från människan för att anpassa presentationen till människans nivå av förståelse. Först undersöks hur beteendeträd kan användas för att modellera realtidsaspekterna av interaktionen mellan robotpresentatören och dess publik. Ett system som baseras på beteendeträdsarkitekturen används i ett delvis automatiskt, delvis människostyrt experiment, där det visas att publikmedlemmar i labbmiljö föredrar ett system som anpassar presentationen till deras reaktioner över ett som inte anpassar sin presentation. Efter detta, urdersöker också avhandlingen hur kunskapsgrafer kan användas för att representera innehållet som roboten presenterar. Om en liten, lokal kunskapsgraf byggs så att den innehåller relationer (kanter) som representerar fakta i presentationen, så kan roboten iterera över grafen och konsekvent hitta refererande uttryck som använder sig av kunskap som publiken redan har. Ett system som baseras på denna arkitektur implementeras, och ett experiment med simulerade interaktioner utförs och presenteras. Experimentets resultat visar att utvärderare som jämför olika anpassningsstrategier föredrar ett system som kan utföra den sortens anpassning som grafmetoden tillåter. Publikens reaktioner i ett presentationsscenario kan ske genom olika modaliteter, som tal, huvudrörelser, blickriktning, ansiktsuttryck och kroppsspråk. För att klassificera kommunikativ återmatning (feedback) av dessa modaliteter från presentationspubliken, utforskas hur sådana signaler kan analyseras automatiskt. En datamängd med interaktioner mellan en människa och vår robot annoteras, och statistiska modeller tränas för att klassifice, QC 20231017

Intelligent agents interacting with humans through conversation (such as a robot, embodied conversational agent, or chatbot) need to receive feedback from the human to make sure that its communicative acts have the intended consequences. At the same time, the human interacting with the agent will also seek feedback, in order to ensure that her communicative acts have the intended consequences. In this review article, we give an overview of past and current research on how intelligent agents should be able to both give meaningful feedback toward humans, as well as understanding feedback given by the users. The review covers feedback across different modalities (e.g., speech, head gestures, gaze, and facial expression), different forms of feedback (e.g., backchannels, clarification requests), and models for allowing the agent to assess the user's level of understanding and adapt its behavior accordingly. Finally, we analyse some shortcomings of current approaches to modeling feedback, and identify important directions for future research., QC 20220404, COIN-SSF, Constructing Explainability (438445824)

Feedback is an essential part of all communication, and agents communicating with humans must be able to both give and receive feedback in order to ensure mutual understanding. In this paper, we analyse multimodal feedback given by humans towards a robot that is presenting a piece of art in a shared environment, similar to a museum setting. The data analysed contains both video and audio recordings of 28 participants, and the data has been richly annotated both in terms of multimodal cues (speech, gaze, head gestures, facial expressions, and body pose), as well as the polarity of any feedback (negative, positive, or neutral). We train statistical and machine learning models on the dataset, and find that random forest models and multinomial regression models perform well on predicting the polarity of the participants' reactions. An analysis of the different modalities shows that most information is found in the participants' speech and head gestures, while much less information is found in their facial expressions, body pose and gaze. An analysis of the timing of the feedback shows that most feedback is given when the robot makes pauses (and thereby invites feedback), but that the more exact timing of the feedback does not affect its meaning., QC 20220112 QC 20220216, Co-adaptive Human-Robot Interactive Systems