Search

Sound source localization (SSL) in a robotic platform has been essential in the overall scheme of robot audition. It allows a robot to locate a sound source by sound alone. It has an important impact on other robot audition modules, such as source separation, and it enriches human–robot interaction by complementing the robot’s perceptual capabilities. The main objective of this review is to thoroughly map the current state of the SSL field for the reader and provide a starting point to SSL in robotics. To this effect, we present: the evolution and historical context of SSL in robotics; an extensive review and classification of SSL techniques and popular tracking methodologies; different facets of SSL as well as its state-of-the-art; evaluation methodologies used for SSL; and a set of challenges and research motivations. [ABSTRACT FROM AUTHOR]

This paper presents a user localization system based on the fusion of visual information and sound source localization, implemented on a social robot called Maggie. One of the main requisites to obtain a natural interaction between human-human and human-robot is an adequate spatial situation between the interlocutors, that is, to be orientated and situated at the right distance during the conversation in order to have a satisfactory communicative process. Our social robot uses a complete multimodal dialog system which manages the user-robot interaction during the communicative process. One of its main components is the presented user localization system. To determine the most suitable allocation of the robot in relation to the user, a proxemic study of the human-robot interaction is required, which is described in this paper. The study has been made with two groups of users: children, aged between 8 and 17, and adults. Finally, at the end of the paper, experimental results with the proposed multimodal dialog system are presented.

This paper describes an improvement in automatic speech recognition (ASR) for robot audition by introducing Missing Feature Theory (MFT) based on soft missing feature masks (MFM) to realize natural human-robot interaction. In an everyday environment, a robot’s microphones capture various sounds besides the user’s utterances. Although sound-source separation is an effective way to enhance the user’s utterances, it inevitably produces errors due to reflection and reverberation. MFT is able to cope with these errors. First, MFMs are generated based on the reliability of time-frequency components. Then ASR weighs the time-frequency components according to the MFMs. We propose a new method to automatically generate soft MFMs, consisting of continuous values from 0 to 1 based on a sigmoid function. The proposed MFM generation was implemented for HRP-2 using HARK, our open-sourced robot audition software. Preliminary results show that the soft MFM outperformed a hard (binary) MFM in recognizing three simultaneous utterances. In a human-robot interaction task, the interval limitations between two adjacent loudspeakers were reduced from 60 degrees to 30 degrees by using soft MFMs.

Dance movement is intrinsically connected to the rhythm of music and is a fundamental form of nonverbal communication present in daily human interactions. In order to enable robots to interact with humans in natural real-world environments through dance, these robots must be able to listen to music while robustly tracking the beat of continuous musical stimuli and simultaneously responding to human speech. In this paper, we propose the integration of a real-time beat tracking system with state recovery with different preprocessing solutions used in robot audition for its application to interactive dancing robots. The proposed system is assessed under different real-world acoustic conditions of increasing complexity, which consider multiple audio sources of different kinds, multiple noise sources of different natures, continuous musical and speech stimuli, and the effects of beat-synchronous ego-motion noise and of jittering in ego noise (EN). The overall results suggest improved beat tracking accuracy with lower reaction times to music transitions, while still enhancing automatic speech recognition (ASR) run in parallel in the most challenging conditions. These results corroborate the application of the proposed system for interactive dancing robots. [ABSTRACT FROM AUTHOR]

This paper attempts to provide a state-of-the-art of sound source localization in robotics. Noticeably, this context raises original constraints—e.g. embeddability, real time, broadband environments, noise and reverberation—which are seldom simultaneously taken into account in acoustics or signal processing. A comprehensive review is proposed of recent robotics achievements, be they binaural or rooted in array processing techniques. The connections are highlighted with the underlying theory as well as with elements of physiology and neurology of human hearing. [ABSTRACT FROM AUTHOR]

This paper presents an interactive quizmaster robot that can manage a multiparty speech-based quiz game. The basic flow of the quiz game is that (1) the robot reads a question, (2) one or more players answer it, and (3) the robot judges the correctness of the answers. We categorize such speech-based quiz games into school-type interaction and auction-type interaction. The former asks players to say ‘Yes’ to get the right to answer before answering a question and the latter allows players to directly answer a question without any advance notice. To realize such interaction, the robot needs the capability of recognizing utterances from multiple people using its own microphones (i.e. ears), even if those utterances are made simultaneously. To cope with such situations, the robot estimates which player made the fastest utterance and recognizes it by localizing and separating a mixture of audio signals. Experiments were conducted to evaluate the success rates of the fastest player identification and speech recognition. The results showed that the robot could identify the fastest speakers with a success rate of 90.0% more accurately than humans when only one speaker slightly preceded the other speakers. We found that although the success rate of speech recognition for the fastest speakers did not reach that of humans, the robot attains amusing quiz game interaction. [ABSTRACT FROM AUTHOR]

Hose-shaped rescue robots have been developed for searching narrow spaces such as under collapsed buildings. The posture estimation independent of the past history is critical, because conventional inertial-sensor-based posture estimation has two main problems; a cumulative error problem peculiar to inertial sensors, and a sudden posture change problem caused by external forces. For coping with the two problems, we developed a novel posture estimation method by putting an active microphone array, a set of microphones and loudspeakers, on the robot. The method calculates the time difference of arrival (TDOA) of the reference signal emitted from one loudspeaker, and estimates the posture from the distance obtained by TDOA. This concise method still has three problems: (1) external noise, (2) reverberation and reflection, and (3) obstacles. These problems are tackled by (1) TSP signal, (2) GCC-PHAT and a threshold-based onset detection, and (3) rejecting incorrect onset times, respectively. Experiments with simulated sounds and actual recordings demonstrate that the method attains the performance of estimation comparable to that of conventional methods, that is, less than 20 cm of the tip position error. Even without historical data, the method attains the similar performance while conventional methods fail. [ABSTRACT FROM AUTHOR]

This study addresses a framework for a robot audition system, including sound source localization (SSL) and sound source separation (SSS), that can robustly recognize simultaneous speeches in a real environment. Because SSL estimates not only the location of speakers but also the number of speakers, such a robust framework is essential for simultaneous speech recognition. Moreover, improvement in the performance of SSS is crucial for simultaneous speech recognition because the robot has to recognize the individual source of speeches. For simultaneous speech recognition, current robot audition systems mainly require noise-robustness, high resolution, and real-time implementation. Multiple signal classification (MUSIC) based on standard Eigenvalue decomposition (SEVD) and Geometric-constrained high-order decorrelation-based source separation (GHDSS) are techniques utilizing microphone array processing, which are used for SSL and SSS, respectively. To enhance SSL robustness against noise while detecting simultaneous speeches, we improved SEVD-MUSIC by incorporating generalized Eigenvalue decomposition (GEVD). However, GEVD-based MUSIC (GEVD-MUSIC) and GHDSS mainly have two issues: (1) the resolution of pre-measured transfer functions (TFs) determines the resolution of SSL and SSS and (2) their computational cost is expensive for real-time processing. For the first issue, we propose a TF-interpolation method integrating time-domain-based and frequency-domain-based interpolation. The interpolation achieves super-resolution robot audition, which has a higher resolution than that of the pre-measured TFs. For the second issue, we propose two methods for SSL: MUSIC based on generalized singular value decomposition (GSVD-MUSIC) and hierarchical SSL (H-SSL). GSVD-MUSIC drastically reduces the computational cost while maintaining noise-robustness for localization. In addition, H-SSL reduces the computational cost by introducing a hierarchical search algorithm instead of using a greedy search for localization. These techniques are integrated into a robot audition system using a robot-embedded microphone array. The preliminary experiments for each technique showed the following: (1) The proposed interpolation achieved approximately 1-degree resolution although the TFs are only at 30-degree intervals in both SSL and SSS; (2) GSVD-MUSIC attained 46.4 and 40.6% of the computational cost compared to that of SEVD-MUSIC and GEVD-MUSIC, respectively; (3) H-SSL reduced 71.7% of the computational cost to localize a single speaker. Finally, the robot audition system, including super-resolution SSL and SSS, is applied to robustly recognize four sources of speech occurring simultaneously in a real environment. The proposed system showed considerable performance improvements of up to 7% for the average word correct rate during simultaneous speech recognition, especially when the TFs were at more than 30-degree intervals. [ABSTRACT FROM AUTHOR]

This paper addresses online calibration of an asynchronous microphone array. Although microphone array techniques are effective for sound localization and separation, these techniques have two issues; geometry information on a microphone array or time-consuming measurements of transfer functions between a microphone array and a sound source is necessary, and a fully synchronous multichannel analog-to-digital converter should be used. To solve these issues, we proposed an online framework for microphone array calibration by combining simultaneous localization and mapping (SLAM), and beamforming. SLAM simultaneously calibrates locations of microphones and a sound source, and clocks differences between microphones every time a microphone array observes a sound event. Beamforming works as a cost function to decide the convergence of calibration by localizing the sound using the transfer functions calculated from the estimated microphone locations and clock differences. We implemented a prototype system based on the proposed framework using extended Kalman filter-based SLAM and delay-and-sum beamforming. The experimental results showed that the proposed framework successfully calibrated an eight-channel asynchronous microphone array both in a simulated and a real environment even when system parameters such as variances are set to be 10 times larger than the optimal values. Furthermore, the error of sound localization with the calibrated microphone array was as small as the desired one, that is, the grid size for beamforming. [ABSTRACT FROM PUBLISHER]

This paper presents a user localization system based on the fusion of visual information and sound source localization, implemented on a social robot called Maggie. One of the main requisites to obtain a natural interaction between human-human and human-robot is an adequate spatial situation between the interlocutors, that is, to be orientated and situated at the right distance during the conversation in order to have a satisfactory communicative process. Our social robot uses a complete multimodal dialog system which manages the user-robot interaction during the communicative process. One of its main components is the presented user localization system. To determine the most suitable allocation of the robot in relation to the user, a proxemic study of the human-robot interaction is required, which is described in this paper. The study has been made with two groups of users: children, aged between 8 and 17, and adults. Finally, at the end of the paper, experimental results with the proposed multimodal dialog system are presented. [ABSTRACT FROM AUTHOR]

This paper describes improvements in Audio-Visual Voice Activity Detection (AV-VAD) using a state transition model. Audio-Visual integration is a promising approach to improve the noise-robustness of VAD. Our proposed AV-VAD is based on a state transition model with four utterance states: speech, nonspeech, and two asynchronous states, a beginning motion state and an ending motion state, corresponding to lip activity before and after voice activity, respectively. We implanted a prototype system into an upper torso humanoid SIG and evaluated the ability of the proposed method by using auditory- and/or visually-contaminated data. Experimental results showed that the robustness of the VAD improved even when the resolution of images was low. On average, our proposed method resulted in a 7.8 point improvement in word detection rates compared with existing AV-VAD methods. [ABSTRACT FROM PUBLISHER]

The motors of a robot produce ego-motion noise that degrades the quality of recorded sounds. This paper describes an architecture that enhances the capability of a robot to perform automatic speech recognition (ASR) even as the entire body of the robot moves. The architecture consists of three blocks: (i) a multichannel noise reduction block, consisting of microphone-array-based sound localization, geometric source separation and post-filtering, (ii) a single-channel template subtraction block and (iii) an ASR block. As the first step of our analysis strategy, we divided the whole-body motion noise problem into three subdomains of arm, leg and head motion noise, according to their intensity levels and spatial location. Subsequently, by following a synthesis-by-analysis approach, we determined the best method for suppressing each type of ego-motion noise. Finally, we proposed to utilize a control module in our ASR framework; this module was designed to make decisions based on instantaneously detected motions, allowing it to switch to the most appropriate method for the current type of noise. This proposed system resulted in improvements of up to 50 points in word correct rates compared with results obtained by single microphone recognition of arm, leg and head motions. [ABSTRACT FROM AUTHOR]

We describe an architecture that gives a robot the capability to recognize speech by cancelling ego noise, even while the robot is moving. The system consists of three blocks: (1) a multi-channel noise reduction block, comprising consequent stages of microphone-array-based sound localization, geometric source separation and post-filtering; (2) a single-channel noise reduction block utilizing template subtraction; and (3) an automatic speech recognition block. In this work, we specifically investigate a missing feature theory-based automatic speech recognition (MFT-ASR) approach in block (3). This approach makes use of spectro-temporal elements derived from (1) and (2) to measure the reliability of the acoustic features, and generates masks to filter unreliable acoustic features. We then evaluated this system on a robot using word correct rates. Furthermore, we present a detailed analysis of recognition accuracy to determine optimal parameters. Implementation of the proposed MFT-ASR approach resulted in significantly higher recognition performance than single or multi-channel noise reduction methods. [ABSTRACT FROM AUTHOR]

Human-robot interaction (HRI)1 is one of the main fields in the study and research of robotics. Within this field, dialogue systems and interaction by voice play an important role. When speaking about human-robot natural dialogue we assume that the robot has the capability to accurately recognize what the human wants to transmit verbally and even its semantic meaning, but this is not always achieved. In this article we describe the steps and requirements that we went through in order to endow the personal social robot Maggie, developed at the University Carlos III of Madrid, with the capability of understanding the natural language spoken by any human. We have analyzed the different possibilities offered by current software/hardware alternatives by testing them in real environments. We have obtained accurate data related to the speech recognition capabilities in different environments, using the most modern audio acquisition systems and analyzing not so typical parameters such as user age, gender, intonation, volume, and language. Finally, we propose a new model to classify recognition results as accepted or rejected, based on a second automatic speech recognition (ASR) opinion. This new approach takes into account the precalculated success rate in noise intervals for each recognition framework, decreasing the rate of false positives and false negatives. [ABSTRACT FROM AUTHOR]

This paper proposes an adaptive step-size method for blind source separation (BSS) suitable for robot audition systems. The design of the step-size parameter is a critical consideration when we apply BSS to real-world applications such as robot audition systems, because the surrounding environment dynamically changes in the real world. It is common to use a fixed step-size parameter that was obtained empirically. However, because of environmental changes and noise, the performance of BSS with a fixed step-size parameter deteriorates and the separation matrix sometimes diverges. Several adaptive step-size methods for BSS have been proposed. However, there are difficulties when applying them to robot audition systems for example, low-computational cost requirements, being free from manual parameter adjustment and so on. We propose an adaptive step-size method suitable for robot audition systems. The proposed method has the following merits: 1) low computational cost; 2) no parameters to be adjusted manually; and 3) no additional preprocessing requirements. We applied our method to six different BSS algorithms for an eight-channel microphone array embedded in Honda's ASIMO robot. The method improved the performance of all six algorithms in experiments on separation and recognition of simultaneous speech. Moreover, the method increased the amount of calculation by less than 10% compared with the original calculation used in most BSS algorithms. [ABSTRACT FROM PUBLISHER]

This paper presents the design and implementation of the HARK robot audition software system consisting of sound source localization modules, sound source separation modules and automatic speech recognition modules of separated speech signals that works on any robot with any microphone configuration. Since a robot with ears may be deployed to various auditory environments, the robot audition system should provide an easy way to adapt to them. HARK provides a set of modules to cope with various auditory environments by using an open-sourced middleware, FlowDesigner, and reduces the overheads of data transfer between modules. HARK has been open-sourced since April 2008. The resulting implementation of HARK with MUSIC-based sound source localization, GSS-based sound source separation and Missing Feature Theory-based automatic speech recognition on Honda ASIMO, SIG2 and Robovie R2 attains recognizing three simultaneous utterances with the delay of 1.9 s at the word correct rate of 80–90% for three speakers. [ABSTRACT FROM AUTHOR]

People usually talk face to face when they communicate with their partner. Therefore, in robot audition, the recognition of the front talker is critical for smooth interactions. This paper presents an enhanced speech detection method for a humanoid robot that can separate and recognize speech signals originating from the front even in noisy home environments. The robot audition system consists of a new type of voice activity detection (VAD) based on the complex spectrum circle centroid (CSCC) method and a maximum signal-to-noise ratio (SNR) beamformer. This VAD based on CSCC can classify speech signals that are retrieved at the frontal region of two microphones embedded on the robot. The system works in real-time without needing training filter coefficients given in advance even in a noisy environment (SNR > 0 dB). It can cope with speech noise generated from televisions and audio devices that does not originate from the center. Experiments using a humanoid robot, SIG2, with two microphones showed that our system enhanced extracted target speech signals more than 12 dB (SNR) and the success rate of automatic speech recognition for Japanese words was increased by about 17 points. [ABSTRACT FROM AUTHOR]

In this work, we present a framework for constructing a spatial map of an indoor environment using the concept of echolocation. More specifically, we propose a non-linear least squares (NLS) estimator which is combined with a spatial filtering technique, e.g., beamforming, to estimate both the time-of-arrival (TOA) and direction-of-arrival (DOA) of the acoustic echoes. The proposed framework is complemented with an echo detector to classify a spurious estimate and an acoustic reflector, i.e., a wall. Based on these estimators, we propose two algorithms that complement existing range sensors and aid robotic platforms in acoustic reflector localization and mapping: single-channel localization and mapping (ScLAM) and a multi-channel localization and mapping (McLAM). Compared to commonly used sensors, such as lidar, cameras and ultrasonic sensors, our proposed model-based approach can detect transparent surfaces that are typically found in an office environment and could work in audible frequency ranges. A proof-of-concept robotic platform was built to test our algorithms. According to our evaluation, both qualitative and quantitative experiments reveal that the proposed methods can detect an acoustic reflector up to a distance of 1.5 m at a signal-to-diffuse-noise ratio (SDNR) of 0 dB in a simulated environment and 10 dB in a real environment with an accuracy of 80%. • A framework for constructing a spatial map of an indoor environment is proposed using echolocation intended for robotic platform. • A general model of the early reflections that takes ego-noise, interfering sound and background noise within the signal model. • A non-linear least squares (NLS) estimator is used to estimate acoustic reflectors for time of arrival (TOA) estimates and beamforming technique was used for direction of arrival (DOA) estimates. • A novel echo detector is proposed to classify an acoustic echoes belonging to a wall from empty space. • Two methods are proposed ScLAM and McLAM. [ABSTRACT FROM AUTHOR]

This paper describes a system that gives a mobile robot the ability to perform automatic speech recognition with simultaneous speakers. A microphone array is used along with a real-time implementation of geometric source separation (GSS) and a postfilter that gives a further reduction of interference from other sources. The postfilter is also used to estimate the reliability of spectral features and compute a missing feature mask. The mask is used in a missing feature theory-based speech recognition system to recognize the speech from simultaneous Japanese speakers in the context of a humanoid robot. Recognition rates are presented for three simultaneous speakers located at 2 m from the robot. The system was evaluated on a 200-word vocabulary at different azimuths between sources, ranging from 100 to 900. Compared to the use of the microphone array source separation alone, we demonstrate an average reduction in relative recognition error rate of 24% with the postfilter and of 42% when the missing features approach is combined with the postfilter. We demonstrate the effectiveness of our multisource microphone array postfilter and the improvement it provides when used in conjunction with the missing features theory. [ABSTRACT FROM AUTHOR]

Mobile robots in real-life settings would benefit from being able to localize and track sound sources. Such a capability can help localizing a person or an interesting event in the environment, and also provides enhanced processing for other capabilities such as speech recognition. To give this capability to a robot, the challenge is not only to localize simultaneous sound sources, but to track them over time. In this paper we propose a robust sound source localization and tracking method using an array of eight microphones. The method is based on a frequency-domain implementation of a steered beamformer along with a particle filter-based tracking algorithm. Results show that a mobile robot can localize and track in real-time multiple moving sources of different types over a range of 7 m. These new capabilities allow a mobile robot to interact using more natural means with people in real-life settings. [Copyright &y& Elsevier]

This paper presents a method to improve recognition of three simultaneous speech signals by a humanoid robot equipped with a pair of microphones. In such situations, sound separation and automatic speech recognition (ASR) of the separated speech signal are difficult, because the signal-to-noise ratio is quite low (around -3 dB) and noise is not stable due to interfering voices. To improve recognition of three simultaneous speech signals, two key ideas are introduced. One is two-layered audio-visual integration of both name (ID) and location, that is, speech and face recognition, and speech and face localization. The other is acoustical modeling of the humanoid head by scattering theory. Sound sources are separated in real-time by an active direction-pass filter (ADPF), which extracts sounds from a specified direction by using the interaural phase/intensity difference estimated by scattering theory. Since features of separated sounds vary according to the sound direction, multiple direction- and speaker-dependent acoustic models are used. The system integrates ASR results by using the sound direction and speaker information provided by face recognition as well as confidence measure of ASR results to select the best one. The resulting system shows an improvement of about 10% on average against recognition of three simultaneous speech signals, where three speakers were located around the humanoid on a 1 m radius half circle, one of them being in front of him (angle 0°) and the other two being at symmetrical positions (±θ) varying by 10° steps from 0° to 90°. [ABSTRACT FROM AUTHOR]

This paper describes our research on bio-mimetic robot audition. Among the many binaural and monaural sound localization cues in the human auditory system, the interaural time difference cue is selected as it can easily be obtained by omnidirectional microphones. We have used a three-microphone system to remove the anterior-posterior ambiguity which occurs in two-microphone (or ear) systems. The echo-avoidance model of the precedence effect is used to cope with the echoes and reverberations of real environments. We mimicked the cocktail party effect by perceptual grouping of continuous components according to the spatial information obtained by the sound localization method. A wheel-based mobile robot equipped with an auditory system was developed. The auditory system has two sound processing parts. One is a DSP-based realtime system; the other is an off-line system composed of remote computers. Experiments of localizing and separating multiple sound sources and robot navigation were conducted to demonstrate the system's ability and potential applications. [ABSTRACT FROM AUTHOR]