Search

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD.

When developing interactive systems for children, such as serious games in the context of educational technology, it is important to take into account and address relevant cognitive and emotional child's experiences that may influence learning outcomes. Some works were done to analyze and automatically recognize these cognitive and affective states from nonverbal expressive behaviors. However, there is a lack of knowledge about visually impaired children and their body language to convey those states during learning tasks. In this paper, we present an analysis of nonverbal expressive behaviors of both blind and low-vision children, aiming at understanding what type of body communication can be an indicator of two cognitive states: engagement and confidence. In the study we consider the data collected along the EU-ICT H2020 weDRAW Project, while children were asked to solve mathematical tasks with their body. For such a dataset, we propose a list of 31 nonverbal behaviors, annotated both by rehabilitators used to work with visually impaired children and by naive observers. In the last part of the paper, we propose a preliminary study on automatic recognition of engagement and confidence states from 2D positional data. The classification results are up to 0.71 (F-score) on a three-class classification task.

A poster abstract for the Consortium of European Research on Emotion (CERE) 2018.

At school, the visual channel is often the most frequently exploited one for teaching, and the other channels are left a marginal role only. The visual signal, however, is not always the more powerful and effective channel for perception. Recent results from psychophysics and developmental psychologyshow indeed that children have a preferential sensory channel to learn specific (spatial and/or temporal) concepts. In this work, we explore the possibility of developingand evaluating a new methodology, supported by novel multisensory technologies, to teaching and learning arithmetic (time) and geometry (space). The main novelty of the proposed approach comes from a deeper understanding of the role of communication between sensory modalities during development, that is, that specific sensory systems have specific roles for learning specific concepts. Results suggest that it is possible to open a new teaching/learning channel, personalized for each student based on the child’s sensory skills. Multisensory interactive technologies including a hardware and software platform to support this approach are presented and discussed.

At school, the visual channel is often the most frequently exploited one for teaching, and the other channels are left a marginal role only. The visual signal, however, is not always the more powerful and effective channel for perception. Recent results from psychophysics and developmental psychology show indeed that children have a preferential sensory channel to learn specific (spatial and/or temporal) concepts. In this work, we explore the possibility of developing and evaluating a new methodology, supported by novel multisensory technologies, to teaching and learning arithmetic (time) and geometry (space). The main novelty of the proposed approach comes from a deeper understanding of the role of communication between sensory modalities during development, that is, that specific sensory systems have specific roles for learning specific concepts. Results suggest that it is possible to open a new teaching/learning channel, personalized for each student based on the child’s sensory skills. Multisensory interactive technologies including a hardware and software platform to support this approach are presented and discussed.

This paper discusses self-efficacy, curiosity, and reflectivity as cognitive and affective states that are critical to learning but are overlooked in the context of affect-aware technology for learning. This discussion sits within the opportunities offered by the weDRAW project aiming at an embodied approach to the design of technology to support exploration and learning of mathematical concepts. We first review existing literature to clarify how the three states facilitate learning and how, if not supported, they may instead hinder learning. We then review the literature to understand how bodily expressions communicate these states and how technology could be used to monitor them. We conclude by presenting initial movement cues currently explored in the context of weDRAW.

The International Workshop on Multimodal Interaction for Education aims at investigating how multimodal interactive systems, firmly grounded on psychophysical, psychological, and pedagogical bases, can be designed, developed, and exploited for enhancing teaching and learning processes in different learning environments, with a special focus on children in the classroom. Whilst the usage of multisensory technologies in the education area is rapidly expanding, the need for solid scientific bases, design guidelines, and appropriate procedures for evaluation is emerging. Moreover, the introduction of multimodal interactive systems in the learning environment needs to develop at the same time suitable pedagogical paradigms. This workshop aims at bringing together researchers and practitioners from different disciplines, including pedagogy, psychology, psychophysics, and computer science - with a particular focus on human-computer interaction, affective computing, and social signal processing - to discuss such challenges under a multidisciplinary perspective. The workshop is partially supported by the EU-H2020-ICT Project weDRAW (http://www.wedraw.eu).