Search

This systematic review analyzes the differential use of gestures in learning by children with intellectual disability (ID) compared to typically developing ones (TD). Eleven studies published between 2000 and 2020 fulfilled the inclusion criteria (N = 364 participants). The results identify three key elements: (1) Children with ID accompany their spoken language with more gestures than TD children; (2) Specifically, they produce more iconic gestures that provide access to the conceptualization process and understanding in students with ID; (3) Children with ID rely on gesture more than TD children to carry meaning (i.e., produce unimodal gestural utterances without accompanying speech). Possible implications for teaching and guidelines for future research are proposed.

This paper discusses how multimodal creative inquiry might be conceptualised and implemented for children’s meaning-making in science. We consider Halliday’s (1978) and Vygotsky’s (1987, 2016) theoretical ideas for showing how the most important characteristics of social semiotics are connected to imagination, play-based and creative inquiry for children’s science meaning-making. Qualitative data was analysed from two preschool classroom video observations of 40 children’s playful interactions with technologies, such as robotic toys, semiotic artefacts, two teachers’ reflective journal documentation and children’s artefacts. Findings show children participate and discuss elements of scientific concepts in inquiry-based dialogues and make sense of science concepts whilst becoming creators of multimodal representations arising from their interests and curiosity. The semiotic resources that operate through technologies such as apps provide a medium for creative inquiry affording communication spaces and multimodal (visual, haptic [digital touch], text) meaning-making around everyday science phenomena. Practical implications lie in upskilling educators’ integration of semiotic resources such as robotic toys and deploying a multimodal creative inquiry approach for reconfiguring children’s science learning opportunities in early childhood educational practices.

Self-generated memory cues support recall of target information more robustly than memory cues generated by others. Across two experiments, we tested whether the benefit of self-generated cues in part reflects a meta-mnemonic effect rather than a pure generation effect. In other words, can learners select better memory cues for themselves than others can? Participants generated as many possible memory cues for each to-be-remembered target as they could and then selected the cue they thought would be most effective. Self-selected memory cues elicited better cued recall than cues the generator did not select and cues selected by observers. Critically, this effect cannot be attributed to the process of generating a cue itself because all of the cues were self-generated. Further analysis indicated that differences in cue selection arise because generators and observers valued different cue characteristics; specifically, observers valued the commonality of the cue more than the generators, while generators valued the distinctiveness of a cue more than observers. Together, results suggest that self-generated cues are effective at supporting memory, in part, because learners select cues that are tailored to their specific memory needs.

The current generation of virtual reality (VR) technologies has improved substantially from legacy systems, particularly in the resolution and latency of their visual display. The presentation of haptic cues remains challenging, however, because haptic systems do not generalise well over the range of stimuli (both tactile and proprioceptive) normally present when interacting with objects in the world. This study investigated whether veridical tactile and proprioceptive cues lead to more efficient interaction with a virtual environment. Interaction in the world results in spatial and temporal correlation of tactile, proprioceptive and visual cues. When cues in VR are similarly correlated, observers experience a sense of embodiment and agency of their avatars. We investigated whether sensorimotor performance was mediated by embodiment of the avatar hands. Participants performed a Fitts’ tapping task in different conditions (VR with no haptics, active haptics, passive haptics, and on a real touchscreen). The active-haptic condition provided abstract tactile cues and the passive haptic condition provided veridical tactile and proprioceptive cues. An additional (hybrid haptics) condition simulated an ideal passive haptic system. Movement efficiency (throughput) and embodiment were higher for the passive than for the active and no-haptics conditions. However, components of embodiment (perceived agency and ownership) did not predict unique variance in throughput. Improved sensorimotor performance and ratings of presence and realism support the use of passive haptics in VR environments where objects are in known and stable locations, regardless of whether performance was mediated by the sense of embodiment.

One of the most important ways to improve pedestrian spaces is to quantitatively analyze the preferences of pedestrians through concrete and empirical data. This requires a psychological evaluation on pedestrians and an analysis on correlations between pedestrian space elements and pedestrians’ preferences. In this study, the relationship between landscape elements in pedestrian space and the preferences of pedestrians was analyzed. The quantitative information of landscape elements was obtained from landscape sequences because pedestrians experience the landscape as a sequence, not individual scenes. The analysis was used to develop a comprehensive index of how pedestrians look at pedestrian spaces. A psychological evaluation was performed to obtain pedestrian preferences for different spaces and the influencing factors. This study clarified the landscape elements preferred by pedestrians, and the results can be used to develop basic guidelines for the planning and design of pedestrian spaces.

The cold-start problem is a grand challenge in music recommender systems aiming to provide users with a better and continuous music listening experience. When a new user creates a playlist, the recommender system remains in a cold-start state until enough information is collected to identify the user’s musical taste. In such cases, playlist metadata, such as title or description, have been successfully employed to create intent recommendation models. In this paper, we propose a multi-stage retrieval system utilizing user-generated titles to alleviate the cold-start problem in automatic playlist continuation. Initially, playlists are clustered to form a music documents collection. Then, the system applies latent semantic indexing to the collection to discover hidden patterns between tracks and playlist titles. For similarity calculation, singular value decomposition is performed on a track-cluster matrix. When the system is given a new playlist as a cold-start instance, it first retrieves neighboring clusters and then produces a ranked list of recommendations by weighting candidate tracks in these clusters. We scrutinize the performance of the proposed system on a large, real-world music playlists dataset supplied by the Spotify platform. Our empirical results show that the proposed system outperforms the state-of-the-art approaches and improves recommendation accuracy significantly in three primary evaluation metrics.

Caregivers of children with an autism spectrum disorder are often responsible for assisting their children to complete activities of daily living skills. Effective and efficient caregiver training methods are needed to train caregivers. The present study used two concurrent multiple-baseline across-participants designs to evaluate the effects of real-time feedback and behavioral skills training on training eight caregivers to implement teaching procedures for activities of daily living skills with their child. We assessed caregivers’ accuracy and correct implementation of the six-component teaching procedure after they received either real-time feedback or behavioral skills training. Caregivers from both groups mastered and maintained correct implementation of the teaching procedures with their child. The overall results suggest that real-time feedback and behavioral skills training are efficacious to train caregivers to implement activities of daily living skills procedures with their children, and that real-time feedback may be an efficient alternative method to train caregivers. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40617-020-00513-z.

This paper traces the origins of the scaffolding construct, placing it in its theoretical-historical context. The paper discusses the connection between Vygotsky’s Zone of Proximal Development (ZPD), and the notion of scaffolding, and explicates the differences between scaffolding and scaffolds. The paper then presents a discussion of the changes that the notion of scaffolding has undergone, especially when it comes to supporting students in classroom contexts. In classrooms where one teacher supports multiple students, scaffolding is distributed across various tools and social scaffolds. A discussion of the notion of distributed scaffolding is presented, to describe how students in classrooms may be supported by various tools and social scaffolds. The paper then introduces the kinds of distribution and interactions between tools and social scaffolds that need to be considered to support multiple students in classroom contexts. Finally, distributed scaffolding is discussed with reference to the key features of scaffolding, especially fading and transfer of responsibility.

This article aims to offer an overview of Tele’Drama as a method and its pioneering role in implementing action and experiential methods via online communication, as well as its purpose in applying J. L. Moreno’s view on international sociometry. The main author of this article is the creator of the method, and believes that Tele‘Drama is a particularly important part of the future of action methods as it is also a bridge between locations and cultures.

Autism Spectrum Disorder is a neurological and developmental disorder. Children diagnosed with this disorder have persistent deficits in their social-emotional reciprocity skills, nonverbal communication, and developing, maintaining, and understanding relationships. Besides, autistic children usually have motor deficits that influence their imitation and gesture production ability. The present study aims to review and analyze the current research findings in using robot-based and virtual reality-based intervention to support the therapy of improving the social, communication, emotional, and academic deficits of children with autism. Experimental data from the surveyed works are analyzed regarding the target behaviors and how each technology, robot, or virtual reality, was used during therapy sessions to improve the targeted behaviors. Furthermore, this study explores the different therapeutic roles that robots and virtual reality were observed to play. Finally, this study shares perspectives on the affordances and challenges of applying these technologies.

The development of information accessibility is receiving more and more attention. One challenging task for the blind is to input Braille while by no way could they sense the location information on touch screens. The existing Braille input methods are suffering from problems including inaccurate positioning and lack of interactive prompts. In this paper, touch gestures are recognized by trained RBF network while combined gestures are modeled by Petri net that introduces logic, timing and spatial relationship description. By doing so, the Braille input concerning multi-touch gesture recognition is then implemented. The experimental results show that the method is effective and blind people can friendly Braille Input with almost real-time interaction. The input method makes full use of the inherent logic of Braille, making the blind easy to learn and remember, providing a new method for human-computer interaction between the blind and the touch screen.

While observing gesture has been shown to benefit narrative recall and learning, research has yet to show whether gestures that provide information that is missing from speech benefit narrative recall. This study explored whether observing gestures that relay the same information as speech and gestures that provide information missing from speech differentially affect narrative recall in university students. Participants were presented with a videotaped narrative told in one of four conditions: with gestures and no missing verbal information, with gestures and missing verbal information, with no gestures and no missing verbal information, or with no gestures and missing verbal information. Results showed that observing gestures that provided additional information to speech (i.e., when the speech was missing vital information) enhanced narrative recall compared to observing no gestures, while observing gestures that did not provide additional information to speech were no more beneficial than observing no gestures at all. Findings from the current study provide valuable insight into the beneficial effect of iconic gesture on narrative recall, with important implications for education and learning.

In this paper, we present a whole new perspective on generating music. The method proposed in this paper is the first to be used which uses the frequency domain characteristics of pitch contour curve to generate music melody with long-term structure controllable. The music generated by this method has a good long-term structure that other basic music generation methods do not have. This method has great development potential and application ability, can be combined with other music generation methods, and improve the performance of long-term structure. This method firstly uses the neural network to fit the pitch contour curve in frequency domain, then combines the vocabulary matching method to perfect the detailed characteristics of melody generated in time domain and control the long-term trend of notes generated with respect to label information, finally generates music melody with real and controllable long-term structure. Through a large number of experiments, it can be seen that compared with the music generated based on the LSTM, the music generated by the proposed method has better long-term structure and has similar statistical characteristics.

This paper presents a systematic review of the literature on Tangible User Interfaces (TUIs) and interactions in young children’s education by identifying 155 studies published between 2001 and 2019. The review was based on a set of clear research questions addressing application domains, forms of tangible objects, TUI design and assessment. The results indicate that (i) the form of tangible object is closely related to the application domain, (ii) the manipulatives are the most dominant form of tangible object, (iii) the majority of studies addressed all three stages of TUI development (design, implementation and evaluation) and declared a small sample of young children as a major shortcoming, and (iv) additional empirical research is required to collect evidence that TUIs are truly beneficial for children’s acquisition of knowledge. This review also identifies gaps in the current work, thus providing suggestions for future research in TUIs application in educational context expected to be beneficial for researchers, curriculum designers and practitioners in early years’ education. To the authors’ knowledge, this is the first systematic review specific to TUIs’ studies in early years’ education and is an asset to the scientific community.

Working memory and writing share a common resource. The authors investigated whether increasing the complexity of a finger-tapping task while simultaneously performing a serial verbal recall task while writing would decrease performance for recall, writing fluency, and tapping speed. Participants completed three verbal serial recall tasks which incrementally increased attentional load. Participants recalled word lists after listening to words while, (1) producing pseudo-handwriting movements without finger tapping (no-tapping), (2) producing pseudo-handwriting movements while tapping with a single finger (single-tapping), (3) producing pseudo-handwriting movements while tapping with two fingers (double tapping). The results showed that the double tapping condition caused a decrease in performance for recall, handwriting fluency, and tapping speed compared to the no-tapping condition and the single tapping conditions. However, no differences occurred between the no-tapping condition and the single tapping conditions. The authors concluded that by incrementally increasing the complexity of a concurrent tapping task we can achieve a decrease in performance across multiple cognitive processes. The results provided support for a central pool of shared resources that are utilised by non-working memory tasks and those reliant on working memory. The observed decreases in cognitive performance were dependent on task complexity rather than just performing a secondary task. The findings have implications for how multi-tasking while taking notes is detrimental for memory retention and handwriting fluency.

A short reminiscence of Andrew Henderson’s interaction with the author is provided, as well as a general overview of his palm research.

This paper draws upon the theory of embodied cognition to provide a robust account of how gestural interactions with and around multi-touch tabletops can play an important role in facilitating collaborative meaning-making, particularly in the context of science data visualizations. Embodied cognition is a theory of learning that implies that thinking and perception are shaped by interactions with the physical environment. Previous research has used embodied cognition as a theoretical framework to inform the design of large touchscreen learning applications such as for multi-touch tabletops. However, this prior work has primarily assumed that learning is occurring during any motion or interaction, without considering how specific interactions may be linked to particular instances of collaborative learning supported by embodiment. We investigate this question in the context of collaborative learning from data visualizations of global phenomena such as ocean temperatures. We followed a user-centered, iterative design approach to build a tabletop prototype that facilitated collaborative meaning-making and used this prototype as a testbed in a laboratory study with 11 family groups. We qualitatively analyzed learner groups’ co-occurring utterances and gestures to identify the nature of gestural interactions groups used when their utterances signaled the occurrence of embodiment during collaborative meaning-making. Our findings present an analysis of both touchscreen and above-the-surface gestural interactions that were associated with instances of embodied cognition. We identified four types of gestural interactions that promote scientific discussion and collaborative meaning-making through embodied cognition: (T1) gestures for orienting the group; (T2) cooperative gestures for facilitating group meaning-making; (T3) individual intentional gestures for facilitating group meaning-making; and (T4) gestures for articulating conceptual understanding to the group. Our work illustrates interaction design opportunities for affording embodied cognition and will inform the design of future interactive tabletop experiences in the domain of science learning.

Although visual and auditory inputs are initially processed in separate perception systems, studies have built on the idea that to maintain spatial information these modalities share a component of working memory. The present study used working memory navigation tasks to examine functional similarities and dissimilarities in the performance of updating tasks. Participants mentally updated the spatial location of a target in a virtual array in response to sequential pictorial and sonant directional cues before identifying the target’s final location. We predicted that if working memory representations are modality-specific, mixed-modality cues would demonstrate a cost of modality switching relative to unimodal cues. The results indicate that updating performance using visual unimodal cues positively correlated with that using auditory unimodal cues. Task performance using unimodal cues was comparable to that using mixed modality cues. The results of a subsequent experiment involving updating of target traces were consistent with those of the preceding experiments and support the view of modality-nonspecific memory.

This document is part of a global investigation that aims to establish best practices in usability testing of mobile applications, that fit the specific needs of persons with cognitive disabilities. The motivating factor is to improve the quality of life of people with special needs. As a first step, we want to discover what are the skills of people with DS (DS) when using a mobile device. Thanks to its direct manipulation interaction style, multi-touch technology is the ideal mechanism for learning activities with a view to the social inclusion of people with DS. This paper investigates the most common touchscreen gestures on commercial existing software. A commercial analysis was carried out to discover the touch screen gestures used by 103 free software Apps running on a mobile touch screen tablet of 11 where the applications run”. The commercial analysis showed that most applications support tap and drag operations on multi-touch technology. Additionally, the research sought to discover the adults with DS (19–54 years of age) ability to perform other gestures on multi-touch surfaces. A DS user skill study was performed to assess the ability of this user segment to interact with multi-touch surfaces. The analysis involved 53 participants, aged between 19 and 54 years, from two vocational training centers attended by people with DS in Madrid. Authors used the Gesture Games App for experimenting with multi-touch gestures such as tap, double tap, long press, drag, scale up, scale down, and one-finger rotation. Tap, double tap, and drag were the three gestures that the most participant could use. In contrast, participants had difficulty performing one-finger rotation and long press gesture. Our statistical analysis showed that the ability to perform each gesture was independent of gender, age group, and previous experience with touchscreen of a participant.

Objective The objective of this study is to evaluate associations between webcam use in the neonatal intensive care unit (NICU) with parental stress and nursing work-related stress and burnout. Design Prospective validated and de novo questionnaires administered to NICU parents and nurses during two observation periods: (1) no webcam access (off webcam) and (2) webcam access (on webcam). Results Seventy-nine “off webcam” parents, 80 “on webcam” parents, and 35 nurses were included. Parental stress levels were significantly lower “on webcam” and perceptions of the technology were overwhelmingly positive. There were no significant differences in nursing stress levels and burnout between periods. Only 14% of nurses believed that webcam use improves infant’s quality of care. Majority nurses felt that webcams increase parental and nursing stress. Conclusions Webcam use in the NICU is associated with lower parental stress levels and has no effect on nursing stress levels or work-related burnout. These findings contradict nurses’ beliefs that webcams increase parent and nurse stress.

On-body input interfaces that can be used accurately without visual attention could have a wide range of applications where vision is needed for a primary task: emergency responders, pilots, astronauts, and people with vision impairments could benefit by making interfaces accessible. This paper describes a between-participant study (104 participants) to determine how well users can locate e-textile interface discrete target touch points on the forearm without visual attention. We examine whether the addition of active touch embroidery and passive touch nubs (metal snaps with vibro-tactile stimulation) helps in locating input touch points accurately. We found that touch points towards the middle of the interface on the forearm were more difficult to touch accurately than at the ends. We also found that the addition of vibro-tactile stimulation aids in the accuracy of touch interactions by over 9% on average, and by almost 17% in the middle of the interface.

People generate reminders in a variety of ways (e.g. putting items in special places or creating to-do lists) to support their memories. Successful remindings can result in retroactive facilitation of earlier information; in contrast, failures to remind can produce interference between memory for related information. Here, we compared the efficacy of different kinds of reminders, including participant’s self-generated reminders, reminders created by prior participants, and normatively associated reminders. Self-generated reminders boosted memory for the earlier target words more than normatively associated reminders in recall tests. Reminders generated by others enhanced memory as much as self-generated reminders when we controlled output order during recall. The results suggest that self-generated reminders boost memory for earlier studied information because they distinctly point towards the target information.

In music recommender systems, automatic playlist continuation is an emerging task that aims to improve users’ listening experience by recommending music in line with their musical taste. The typical approach towards this goal is to identify playlist characteristics by inspecting the existing tracks (i.e., seeds) in target playlists. However, seeds are not always available, especially when users create new playlists. For such cold-start situations, user-generated titles can be a good starting point to understand the intended purpose of users. This paper investigates the effectiveness of titles as an auxiliary data source for playlists suffering from the cold-start problem. Employing three naive recommendation models, we conduct experiments on one million music playlists from the Spotify platform. Our analyses show that the prevalent attitude in naming playlists results in highly accurate recommendations for playlists concerning a specific theme, such as albums, artists, and soundtracks. As the title space moves away from a particular theme, recommendation accuracy drops. Furthermore, the correlation between the common preference of a title and its usability in recommendation is quite weak; a title without a common sense may be useless in recommender systems, even though many users favor that title. Consequently, our findings serve as a guideline to develop title-aware recommendation approaches that can provide coherent continuations to the cold-start playlists.

In this paper, a new haptic interaction is presented where the operator is in contact with the haptic device (HD) only when she/he is in contact with the virtual environment (VE). This is in contrast with traditional haptic systems, where the operator is always in contact with the HD, even if she/he is out of the VE. In this haptic interaction, a visual tracking system is used to track the operator’s finger. When the finger is out of the VE, the HD tracks the finger so that the stylus of the HD keeps a constant distance of about 2 cm from the finger. When the finger gets close to the VE, the stylus slows down and stops upon reaching the VE; it then waits until the operator touches the stylus and feels the VE. Some advantages of this haptic interaction include more immersivity, higher margins of stability, bigger workspace, smaller actuators and, more feasible impact simulation. The speed of the HD at the onset of contacting the VE plays a significant role in the stability of the haptic system. The lower the collision speed, the bigger the maximum stiffness of the VE will be. The stability improvement of this presented haptic interaction is compared with the traditional one for both low and medium collision speeds for several time delays. For low collision speeds, theoretical and experimental results show an increase of 72% and 40% in the maximum stiffness of the VE, respectively. Similarly, for medium collision speeds, 44% and 28% increase in the maximum stiffness of the VE are achieved for theoretical and experimental results, respectively.

This work describes the study of modeling and controller on wheeled mobile robots designed the motors which is driving the wheels. According to the structure and design of wheeled mobile robot, DC motors are the best suited for the motion control. The kinematical model is required for the designing process of the wheels in the WMR. The analysis of the mathematical model is divided into angle and velocity of the dc motor build in wheeled mobile robot because of the importance of motor parameters for stability. The main focus of the work is to develop an efficient controller to control the speed of the dc motor applied in the wheels of the robot. PID tuning has been implemented in designing of the controller for the speed control of dc motor. The open loop and closed loop performance of a two wheeled mobile robot with PID and LQR controllers are obtained and compared by using MATLAB programs and simulations.

Vision of the body has been reported to improve tactile acuity even when vision is not informative about the actual tactile stimulation. However, it is currently unclear whether this effect is limited to body parts such as hand, forearm or foot that can be normally viewed, or it also generalizes to body locations, such as the shoulder, that are rarely before our own eyes. In this study, subjects consecutively performed a detection threshold task and a numerosity judgment task of tactile stimuli on the shoulder. Meanwhile, they watched either a real-time video showing their shoulder or simply a fixation cross as control condition. We show that non-informative vision improves tactile numerosity judgment which might involve tactile acuity, but not tactile sensitivity. Furthermore, the improvement in tactile accuracy modulated by vision seems to be due to an enhanced ability in discriminating the number of adjacent active electrodes. These results are consistent with the view that bimodal visuotactile neurons sharp tactile receptive fields in an early somatosensory map, probably via top-down modulation of lateral inhibition.

According to narratology or narrative theory, a piece of artwork should tell a story based on its various tensions. In this study, an automated music composition algorithm using musical tension energy was proposed; this algorithm can generate a musical piece by changing the musical tension. The proposed innovative Algorithmic Composition Musical Tension Energy (ACMTE) method uses the level of musical tension; this level is determined primarily by the chord progression and also the musical parameters of pitch interval and rhythm. The effects of musical tension energy on those parameters were analyzed. This paper presents a formula that unifies all generated parts. The experimental results demonstrate that thousands of beautiful pieces can easily be made without the use of a music database. This algorithmic composition method can be easily applied in both streaming media and to portable music devices, such as smart phones, notebooks, and MP3 players.

Braille is a system of raised dots. Normal human beings read braille code with their eyes whereas visually impaired person with their fingers. Braille is not a language but is a code. With Braille code many languages such as English, French, Spanish etc. may be written and read. All over the world many people use this code in their own native language, thereby providing literacy to one and all. Braille is traditionally written on an embossed paper which is static in nature. Advanced Braille allows people to read digital documents with the use of refreshable braille displays. The aim is to design and implement a system capable of making the process of Braille reading easier and more economic to the visually impaired users. In this paper a new and improvised design for Braille Cell is proposed and implemented using the Arduino UNO and electromagnetic relays as tactile device. The proposed device outperforms the existing devices in terms of manufacturing complexity, cost and ease of use. A prototype was developed and tested with several visually impaired people to validate the operation. If implemented on a larger scale, the product promises to be of an efficient replacement to the existing braille books.

Mobile organisms make use of spatial cues to navigate effectively in the world, such as visual and self-motion cues. Over the past decade, researchers have investigated how human navigators combine spatial cues, and whether cue combination is optimal according to statistical principles, by varying the number of cues available in homing tasks. The methodological approaches employed by researchers have varied, however. One important methodological difference exists in the number of cues available to the navigator during the outbound path for single-cue trials. In some studies, navigators have access to all spatial cues on the outbound path and all but one cue is eliminated prior to execution of the return path in the single-cue conditions; in other studies, navigators only have access to one spatial cue on the outbound and return paths in the single-cue conditions. If navigators can integrate cues along the outbound path, single-cue estimates may be contaminated by the undesired cue, which will in turn affect the predictions of models of optimal cue integration. In the current experiment, we manipulated the number of cues available during the outbound path for single-cue trials, while keeping dual-cue trials constant. This variable did not affect performance in the homing task; in particular, homing performance was better in dual-cue conditions than in single-cue conditions and was statistically optimal. Both methodological approaches to measuring spatial cue integration during navigation are appropriate.

Collaborative Mixed Reality (MR) technologies enable remote people to work together by sharing communication cues intrinsic to face-to-face conversations, such as eye gaze and hand gestures. While the role of visual cues has been investigated in many collaborative MR systems, the use of spatial auditory cues remains underexplored. In this paper, we present an MR remote collaboration system that shares both spatial auditory and visual cues between collaborators to help them complete a search task. Through two user studies in a large office, we found that compared to non-spatialized audio, the spatialized remote expert’s voice and auditory beacons enabled local workers to find small occluded objects with significantly stronger spatial perception. We also found that while the spatial auditory cues could indicate the spatial layout and a general direction to search for the target object, visual head frustum and hand gestures intuitively demonstrated the remote expert’s movements and the position of the target. Integrating visual cues (especially the head frustum) with the spatial auditory cues significantly improved the local worker’s task performance, social presence, and spatial perception of the environment.

In this paper, we present a tactile rendering algorithm applied to an electrostatic tactile display that adjusts three parameters of the driving signal (amplitude, frequency, and waveform) to modulate the tangential friction force between a user’s finger and a touch screen. The aim of this work is to find an effective electrostatic tactile rendering algorithm to improve the tactile perception of image textures. The key idea is to jointly adjust the three parameters of the driving signal to increase the perceptual difference interval between image textures. We first explore the tactile representation characteristics of amplitude, frequency and waveform through subjective perception experiments. Based on these characteristics, we establish tactile mapping models between the three parameters of the driving signal and image textures. Finally, we use subjective evaluation experiments to verify the effectiveness of the proposed rendering method. The results show that the proposed rendering method can achieve a better tactile experience compared with rendering methods that realize tactile representation by only varying amplitude.

Automatic music composition could dramatically decrease music production costs, lower the threshold for the non-professionals to compose as well as improve the efficiency of music creation. In this paper, we proposed an intelligent music composition neutral network to automatically generate a specific style of music. The advantage of our model is the innovative structure: we obtained the music sequence through an actor’s long short term memory, then fixed the probability of sequence by a reward-based procedure which serves as feedback to improve the performance of music composition. The music theoretical rule is introduced to constrain the style of generated music. We also utilized a subjective validation in experiment to guarantee the superiority of our model compared with state-of-the-art works.

Healthy people use a keyboard and mouse as standard input devices for controlling a computer. However, these input devices are usually not suitable for people with severe physical disabilities. This study aims to design and implement a suitable and reliable human–computer interactive (HCI) interface for disabled users by integrating both eye tracking technology and lip motion recognition. Eye movements control the cursor position on the computer screen. An eye gaze followed by a lip motion of the mouth opening served as a mouse click action. Seven lip motion features were extracted to discriminate mouth openings and mouth closures with the cumulative sum control chart algorithm. A novel smoothing technique called the threshold-based Savitzky–Golay smoothing filter was proposed to stabilize the cursor movement due to the inherent jittery motions of eyes and to reduce eye tracking latencies. In this study, a fixation experiment with nine dots was carried out to evaluate the efficacy of eye gaze data smoothing. A Chinese text entry experiment based on the on-screen keyboard with four keypad sizes was designed to evaluate the influence of the keypad size on the Chinese text entry rate. The results of the fixation experiment indicated that the threshold-based Savitzky–Golay smoothing filter with a threshold value of two standard deviations, a polynomial order of 3, and a window length of 61 can significantly improve the stability of eye cursor movements by 44.86% on average. The averaged Chinese text entry rate achieved 4.41 wpm with the dynamical enlargeable keypads. The current results encouraged us to utilize the proposed HCI interface for disabled users in the future.

Digital technologies are essential for almost all human activities, so children should learn to work with them at schools. Some studies of the effect of using computers, tablets or cell phones at schools have been carried out, but in Czechia, where the research was conducted, the number of such studies is very low. We wanted to examine how tablets help pupils of the high school with learning social geography in Czech educational system. We worked with two classes of a high school in Prague. Two lessons were taught in each class, one with the Nearpod application on the tablet, the other without it. Pupils wrote pre-tests before the lesson and post-tests after it. Interviews were conducted with some of the pupils after the lesson with the tablet. Then, we worked with the results of both classes together and with the results of each class separately. We found out that pupils enjoyed the lessons with tablet, they felt more motivated and they thought, that they had learnt more with the tablet than without it. However, statistics showed us different results: the results of post-test were better when pupils had used tablet, but this finding was not valid for one of the two classes, where the difference between tablet and non-tablet lesson was not statistically significant. There were differences in pupils’ relationship to digital technologies and in level of collaboration between the classes. The level of collaboration provides a good explanation of different success of both classes.

Increasingly powerful computer technologies have enabled the development and widespread growth of touchscreen devices such as computers, tablets, and smartphones in consumers’ daily lives, including online shopping. Nonetheless, it is not clear to what extent and how direct-touch (i.e., finger) vs. indirect-touch (e.g., mouse click or stylus) interfaces have differential effects on consumers’ evaluations toward the object on the screen. Results from a lab experiment and a field study indicate that a direct-touch (vs. indirect-touch) interface has a polarizing effect on consumer evaluations. For an object about which consumers have a prior positive attitude, a direct-touch interface enhances consumer evaluations; for an object about which consumers have a prior negative attitude, a direct-touch interface lowers consumer evaluations. We find that consumers’ visual information processing style can moderate the polarizing effect. In addition, the polarizing effect can be explained by consumers’ vividness perception. These findings make useful contributions to the literature on haptic effects and human-computer interactions, as well as have significant managerial implications.

The development of touchscreen technology is providing alternative ways for learners to conceptualise, visualise, experiment with and communicate about mathematical ideas and relationships. While the multi-touch affordances of touchscreens enable children to produce and transform ‘screen objects’ with their fingers (by means of varied forms of pressure and propulsion), they also invoke an intricate interrelationship between the user’s fingers and the surface of the device itself. Drawing on a half-hour video recording of two primary school children using the TouchTimes iPad app (about multiplication) for the first time, we examine how mutually interactive the children’s fingers were, both with each other and with this particular touchscreen technology, not least their combining in ways which challenge the seemingly clear distinction between digital and physical tools, when viewed as discrete and disjoint entities. We also explore fingers being used as objects in themselves, while examining ways of doing multiplication digitally with fingers, with a particular focus on the singular role of fingers as physical intermediaries. Our aim is to consider possible ways to develop well-educated fingers in relation to engaging with mathematics.

Functional neuroimaging studies have demonstrated that following deafness, auditory regions can respond to tactile stimuli. However, research to date has not conclusively demonstrated the behavioral correlates of these functional changes, with most studies showing normal-like tactile capabilities in the deaf. It has recently been suggested that more cognitive and complex tactile processes, such as music perception, could help to uncover superior tactile capabilities in the deaf. Indeed, following deafness music seems to be perceived through vibration, but the extent to which they can perceive musical features though the tactile modality remains undetermined. The goal of this study was to investigate tactile identification of musical emotion in the deaf. Participants had to rate melodies based on their emotional perception. Stimuli were presented through an haptic glove. Data suggest that deaf and control participants were comparable in the identification of three of the four emotions tested (sad, fear/threat, peacefulness). However and most importantly, for the simplest emotion (happiness), significant differences emerged between groups, suggesting an improved tactile identification of musical emotion in the deaf. Results support the hypothesis that brain plasticity following deafness can lead to improved complex tactile ability.

This paper introduces the musical haptic wearables for audiences (MHWAs), a class of wearable devices for musical applications targeting audiences of live music performances. MHWAs are characterized by embedded intelligence, wireless connectivity to local and remote networks, a system to deliver haptic stimuli, and tracking of gestures and/or physiological parameters. They aim to enrich musical experiences by leveraging the sense of touch as well as providing new capabilities for creative participation. The embedded intelligence enables the communication with other external devices, processes input data, and generates music-related haptic stimuli. We validate our vision with two concert-experiments. The first experiment involved a duo of electronic music performers and twenty audience members. Half of the audience used an armband-based prototype of MHWA delivering vibro-tactile feedback in response to performers’ actions on their digital musical instruments, and the other half was used as a control group. In the second experiment, a smart mandolin performer played live for twelve audience members wearing a gilet-based MHWA, which provided vibro-tactile sensations in response to the performed music. Overall, results from both experiments suggest that MHWAs have the potential to enrich the experience of listening to live music in terms of arousal, valence, enjoyment, and engagement. Nevertheless, results showed that the audio-haptic experience was not homogeneous across participants, who could be grouped as those appreciative of the vibrations and those less appreciative of them. The causes for a lack of appreciation of the haptic experience were mainly identified as the sensation of unpleasantness caused by the vibrations in certain parts of the body and the lack of the comprehension of the relation between what was felt and what was heard. Based on the reported results, we offer suggestions for practitioners interested in designing wearables for enriching the musical experience of audiences of live music via the sense of touch. Such suggestions point towards the need of mechanisms of personalization, systems able to minimize the latency between the sound and the vibrations, and a time of adaptation to the vibrations.

The automatic generation of music is an emerging field of research that has attracted wide attention in Computer Science. However, most works are centered in classical music. This work develops ETHNO-MUSIC, an intelligent system that generates melodies based on popular music. ETHNO-MUSIC generates melodies with Markov models, which learns from a corpus of Spanish popular music. Then, given the importance of the lyrics in this context, ETHNO-MUSIC was integrated with Tra-La-Lyrics, an existing system that generates lyrics following a melody, which has been specifically adapted to suit this purpose. Several experiments were carried out to evaluate the quality of the results, based on human opinions towards generated pieces of music and lyrics. Overall, results are positive. Briefly, they reflect that, on the one hand, the melodies transmit a feeling of Spanish popular music, and on the other hand, the text of the lyrics is related to the topics analyzed, and the rhythm follows the melodic aspects of the music.

Shared interactive book reading (SIBR) is a broad term used to describe the act of adults reading aloud to children, while encouraging interaction by asking questions and engaging in a discussion about the book. SIBR can be used to embed naturalistic communication teaching strategies, creating learning opportunities to promote a child’s language and communication skills. This article presents practical information on how caregivers can use naturalistic communication teaching strategies during SIBR with their child with developmental delays or disabilities (DD). The intended audiences are caregivers (e.g., parents, grandparents) and practitioners (e.g., classroom teachers, speech-language pathologists, reading specialists) who work with caregivers of children with DD. We explain the importance of early communication skills, naturalistic communication teaching strategies, and SIBR, and describe how to embed naturalistic communication teaching strategies within SIBR. Through a vignette we illustrate how caregivers can implement the techniques and strategies and how practitioners can support caregivers’ implementation.

This is a theoretical paper that considers the mapping problem, a foundational issue which arises when designing a sonification, as it applies to sonic information design. We argue that this problem can be addressed by using models from the field of embodied cognitive science, including embodied image schema theory, conceptual metaphor theory and conceptual blends, and from research which treats sound and musical structures using these models, when mapping data to sound. However, there are currently very few theoretical frameworks for applying embodied cognition principles in a sonic information design context. This article describes one such framework, the embodied sonification listening model, which provides a theoretical description of sonification listening in terms of conceptual metaphor theory.

This paper explores the role of metaphor in cognition, outlining how the machine metaphor came to dominate and the problematic assumptions for culture/nature relationships and the consequences for the environment that underpin this metaphor. It then argues that the machine metaphor needs to be complemented by an animistic metaphor, which embodies and underpins indigenous relationships to nature, using examples from the New Zealand Māori.

A new high-capacity information hiding method for embedding secret messages into MIDI files is proposed. The method can preserve the original musical performance of the cover MIDI file. The property of the variable-length quantity, which expresses the magnitude of the delta time before every event in a MIDI file, is utilized for secret bit embedding. The embedding is accomplished by padding the delta times with different numbers of leading constant bytes of 8016 to represent the secret bits. The method is both reversible and blind because the original cover MIDI file can be restored completely from the stego-MIDI file by extracting the embedded data out from the resulting stego-MIDI file without referencing the original cover MIDI file. A capability of hiding a large amount of secret information is achieved since the delta time is a basic parameter that appears before every event in the MIDI file. Good experimental results yielded by the proposed method as well as a comparison of the method with five existing performance-preserving methods from the viewpoints of stego-file quality, payload capacity, and data security show the superiority and feasibility of the proposed method.

The principles for developing smart sensors for monitoring the vibrational state and for vibration diagnostics of power equipment in thermal power plants are analyzed.

Haptic interfaces can recreate the experience of touch and are necessary to improve human–robot interactions. However, at present, haptic interfaces are either electromechanical devices eliciting very limited touch sensations or devices that may provide more comprehensive kinesthetic cues but at the cost of their large volume: there is a clear trade-off between the richness of feedback and the device size. The design and manufacturing challenges in creating complex touch sensations from compact platforms still need to be solved. To overcome the physical limitation of miniaturizing force feedback robots, we adapted origami design principles to achieve portability, accuracy and scalable manufacturing. The result is Foldaway, a foldable origami robot that can render three-degrees-of-freedom force feedback in a compact platform that can fit in a pocket. This robotic platform can track the movement of the user’s fingers, apply a force of up to 2 newtons and render stiffness up to 1.2 newtons per millimetre. We experimented with different human–machine interactions to demonstrate the broad applicability of Foldaway prototypes: a portable interface for the haptic exploration of an anatomy atlas; a handheld joystick for interacting with virtual objects; and a bimanual controller for intuitive and safe teleoperation of drones. Haptic interfaces are important for the development of immersive human–machine interactions. To create a compact design with rich touch-sensitive functions, a robotic device called Foldaway, which folds flat, has been designed that can render three-degrees-of-freedom force feedback.

As one of the most important biometrics, ear biometrics is getting more and more attention. Ear recognition has unique advantages and can make identification more secure and reliable together with other biometrics (e.g. face and fingerprint). Therefore, we investigate related information about ear recognition and classify the entire process of ear recognition, including detection, preprocessing, unimodal recognition including feature extraction and decision of classification or matching, and multimodal recognition based on inter-level and intra-level fusion. Unimodal and multimodal recognition are proposed comprehensively. In addition, inter-level and intra-level fusion are divided into different fusion ways. At the same time, we compare recognition results under the same dataset and analyze the difficulty of some datasets. In the end, challenges and outlook of ear recognition are also mentioned to expect to provide readers with some help about future directions and problems that should be overcome.

This study investigates if there is an age-related positivity effect on how older adults in eastern cultures perform trustworthiness evaluations based on facial cues. Furthermore, it explores the potential method information processing affected by the positivity effect during trustworthiness evaluations. The results show that the level of trustworthiness older adults in China assign to faces is significantly higher than that of younger adults; older adults’ trustworthiness evaluations based on facial cues in the valid cueing condition is significantly higher than in the invalid cueing condition; and for younger adults, trustworthiness evaluation of untrustworthy faces in the valid cueing condition was significantly lower than in the invalid cueing condition. Overall, these results show that positive information processing in older adults is regulated by attention capacity, while negative information processing by young adults is regulated by attention capacity, which indicates that the age-related positivity effect in older adults is due to increased positive information processing.

For young children, literacy is multimodal. Visual images, oral language, gestures, numbers, and other signs are intermingled with printed words during language arts activities in preschool and kindergarten. All children, in particular children from multilingual/multicultural backgrounds, draw on the social, cultural, and emotional roles and structures they observe and experience daily in their homes and communities when presented with multiple modes of literacy in meaningful classroom contexts. This article employs social semiotics as a framework for presenting multimodal literacies used by young children to make and create meaning in their social and learning interactions. Following an overview of multimodal literacy related to children’s literacy learning, topics explored in this article include multimodal literacy activities that embed social interaction, classroom multimodal literacy events that reflect the sociocultural patterns children bring to the classroom, and approaches to implementing a culturally responsive pedagogy in order to make multimodal literacy meaningful to young learners. Classroom examples are provided throughout to illustrate scenarios with teachers who are knowledgeable about multimodal literacy experiences and social interaction, embrace a sociocultural perspective, and are committed to implementing culturally responsive pedagogy.

Through providing an external support to speech, gesture observation may benefit a student’s learning in a variety of areas, including narrative comprehension. Across two studies, we investigated factors that could moderate when gestures are most beneficial to narrative comprehension, including gesture type, task difficulty, and age, in order to help determine when gestures benefit narrative comprehension most. Crucially, observing typical gestures significantly benefited narrative comprehension (measured by specific questions relating to gesture points) compared with atypical or no gestures, which did not differ significantly. When measured through free recall and specific questions, these effects were not moderated by total task difficulty or age. This finding suggests that how beneficial a gesture is for narrative comprehension may depend more on the kind of gesture observed rather than on age or the difficulty of the task. It may be that typical gestures benefit narrative comprehension more than atypical gestures due to the typical gestures being more semantically related to accompanying speech. In a second study, typical gestures were rated as more semantically related to the speech than were atypical gestures. We argue that educators’ use of typical, frequently produced iconic gestures that are adequately semantically related to speech may enhance student understanding.

Gesture elicitation study, a technique emerging from the field of participatory design, has been extensively applied in emerging interaction and sensing technologies in recent years. However, traditional gesture elicitation study often suffers from the gesture disagreement and legacy bias problem and may not generate optimal gestures for a target system. This paper reports a research project on user-defined gestures for interacting with in-vehicle information systems. The main contribution of our research lies in a 3-stage, participatory design method we propose for deriving more reliable gestures than traditional gesture elicitation methods. Using this method, we generated a set of user-defined gestures for secondary tasks in an in-vehicle information system. Drawing on our research, we develop a set of design guidelines for freehand gesture design. We highlight the implications of this work for the gesture elicitation for all gestural interfaces.