Search

Sensory loss causes compensatory behavior, like echolocation upon vision loss or improved visual motion detection upon deafness. This is enabled by recruitment of the deprived cortical area by the intact senses. Such cross-modal plasticity can however hamper rehabilitation via sensory substitution devices. To steer rehabilitation towards the desired outcome for the patient, having control over the cross-modal take-over is essential. Evidence accumulates to support a role for the posterior parietal cortex (PPC) in multimodal plasticity. This area shows increased activity after sensory loss, keeping similar functions but driven by other senses. Patient-specific factors like stress, social situation, age and attention, have a significant influence on the PPC and on cross-modal plasticity. We propose that understanding the response of the PPC to sensory loss and context is extremely important for determining the best possible implant-based therapies, and that mouse research holds potential to help unraveling the underlying anatomical, cellular and neuromodulatory mechanisms. ispartof: Current Opinion In Neurobiology vol:67 pages:16-25 ispartof: location:England status: Published online

Background For individuals who never achieve independent standing, rehabilitation is focused on trunk posture and balance control. Visual biofeedback has the potential to augment sitting balance training, however previous work in this area has been limited to standing. Research question To what extent do different types of visual biofeedback influence trunk sway in sitting? Methods Twelve healthy young adults sat on an articulating bench. During ‘sway referencing’ trials, the bench tilted up and down in proportion to trunk sway in the frontal plane. This paradigm increased difficulty of the balance task and required participants to rely on visual and vestibular cues. Participants were provided different visual biofeedback through a rotating needle-gage display. Trials lasted 165 s, were ordered randomly, and included either direct feedback (needle rotated in proportion to body sway), inverted feedback (needle rotated in the opposite direction of sway), time delayed feedback (0.5 s), random feedback, eyes closed, or control (eyes open with screen off). To explore the impact of inherent stability, trials were repeated with and without external trunk support. Results Body sway depended on feedback type. Specifically, direct and inverted feedback reduced root-mean-squared (RMS) sway the most, time delayed feedback had a smaller effect, and random visual feedback increased participants’ RMS sway compared to control. Frequency domain analyses demonstrated direct and inverted visual feedback reduced sway amplitude at the lower frequencies while having minimal effect on (or increasing) sway amplitude at higher frequencies. Significance This study extends previous work by showing that visual biofeedback can have powerful effects on sitting balance, even with external support. Results from the different types of feedback conditions further our understanding of how the brain interprets visual biofeedback. Frequency-based results were similar to previous studies using different modalities and suggest participants interpret biofeedback through sensory addition as opposed to sensory substitution.

The design of Sensory Substitution Devices (SSDs) often relies on the belief that the information supplied by the devices should allow the construction of spatial mental representations on the basis of which routes are planned. This study, in contrast, illustrates that navigation using an SSD can be conceived as an on-line, dynamic process, without the need for establishing a predefined plan or model of the task prior to its execution. We analyzed route selection performed with a vibrotactile SSD that could detect environmental surfaces only within a short spatial range, limiting the availability of information about remote parts of the environment to be navigated. Sixty sighted participants performed a navigation task that involved the goal of reaching a target destination while avoiding five obstacles (placed in randomly predetermined configurations). Three groups of participants differed in the sensory modality used (restricted visual, acoustic + vibrotactile, and restricted visual + vibrotactile). While participants in the visual condition had fewer obstacle collisions and reached the target location sooner, the groups coincided to a large extent in terms of the routes that they followed. Furthermore, the routes selected by participants in all groups conformed well to routes predicted by a dynamic model of visually-guided locomotion (Fajen and Warren, 2003). These findings show that local, limited information about environmental layout can support route selection equivalent to that seen when information about the full environmental layout is available.

During immersive balance rehabilitation, automatic visual-vestibular compensations occurs to reduce the patients’ visual reliance and improve the equilibrium. This paper describes the use of an identification procedure to characterise the relationship between visual stimulation features involved in this adaptive sensory compensation, and the balance improvement. The purpose is to determine the stimulus-response transfer functions (TF) associated to the equilibrium enhancement. Standing vestibular patients were stimulated by visual virtual flows, whose pattern and speed changed throughout successive stimulation sessions. The analysis of the feet centre-of-pressure, disequilibrium, and identified models parameters for one representative vestibular patient, showed that TF parameters evolved related to the gradual balance recovery boosted by the visual-vestibular compensation. This results suggest that identified TF parameters are suitable indicators for measuring the effect of sensory substitution on equilibrium recovery. This first step to model the relationship between the sensory re-weighting flexibility and the adaptation of postural commands is essential for future clinical studies using identification methods for sensorimotor evaluation in individualized vision-based balance rehabilitation

Face perception in humans and nonhuman primates is accomplished by a patchwork of specialized cortical regions. How these regions develop has remained controversial, some studies claiming that seeing faces is necessary for face-domain formation, while others contend that visual experience is not necessary for their development. In sighted individuals, facial information is primarily conveyed via the visual modality. Blind individuals, on the other hand, can still recognize shapes using auditory and tactile cues. Here we examined whether face-selective regions like the fusiform face area (FFA) show selectivity for faces in individuals with early blindness. In particular, we asked if such individuals can learn to distinguish faces from other shapes by using a sensory substitution device (SSD) presenting faces as sound-encoded stimuli. Early blind individuals and sighted controls were trained to discriminate sound-encoded faces, houses, and abstract geometrical shapes presented via an SSD. Combining functional magnetic resonance imaging (fMRI) with an N-back task, we found new evidence for preferential activation of the left FFA by sound-encoded faces in early blind adults. The results demonstrate that face-selective cortical regions can be activated by substituting vision with audition. This implies that the development of the FFA does not necessarily depend on experience with visual faces per se, but may instead depend on sensory experience with facial configurations.

When I am looking at my coffee machine that makes funny noises, this is an instance of multisensory perception I perceive this event by means of both vision and audition. But very often we only receive sensory stimulation from a multisensory event by means of one sense modality, for example, when I hear the noisy coffee machine in the next room, that is, without seeing it. The aim of this paper is to bring together empirical findings about multimodal perception and empirical findings about (visual, auditory, tactile) mental imagery and argue that on occasions like this, we have multimodal mental imagery: perceptual processing in one sense modality (here: vision) that is triggered by sensory stimulation in another sense modality (here: audition). Multimodal mental imagery is not a rare and obscure phenomenon. The vast majority of what we perceive are multisensory events: events that can be perceived in more than one sense modality like the noisy coffee machine. And most of the time we are only acquainted with these multisensory events via a subset of the sense modalities involved all the other aspects of these multisensory events are represented by means of multisensory mental imagery. This means that multisensory mental imagery is a crucial element of almost all instances of everyday perception. (C) 2017 The Author. Published by Elsevier Ltd.

The artificial sensory-substitution system can work like real human sensory tissue and participate in the perception of sensory-handicap patient. The power-supply unit is key factor for its wearable, low-cost and long-term application. Here, a new self-powered multi-perception electronic-skin has been presented as brain sensory receptor for sensory substitution of tactility, audition, olfaction, gustation and vision. The device directly outputs the triboelectric sensory signal into brain using human motion energy without external electricity power or battery, and actively involves in multi-perceptions. The sensing unit arrays act as both the power source and brain sensory receptor for tracking force trace, hearing words, smelling odor, tasting beverage and recognizing image. The mechanism can be attributed to the coupling of triboelectrification and surface effect. The device connecting to mouse brain at primary somatosensory barrel cortex can mimic the mouse perception and drive the mouse activities. This new approach can be applied to fabricating new self-powered sensory-substitution systems and multi-perception brain sensory receptor.

Electrotactile Perception Threshold (EPT) is critical for designing electrotactile displays, which is the minimum amplitude of an electrical stimulus that can be perceived. A significant concern in electrotactile displays is skin irritation and burns due to prolonged electrical stimulus with high amplitude. This study aims to propose a method for reducing the EPT using a background stimulation with a vibrotactile display at subthreshold: 90% of the Vibration Perception Threshold (VPT) at 235 Hz. A psychophysical experiment was conducted to measure EPT at the middle of the left forearm with and without the vibrotactile display using the staircase method. A reduction of 3 to 5% in EPT was observed, which can be further enhanced with varying study parameters. In addition, the comfort and safety aspects of the user’s assessment have been analyzed. Electrotactile stimulations have a higher Steven’s power exponent (1.51). Hence a reduced threshold (although it’s only 3 to 5%) would be perceptually significant and advantageous for sensory substitution and rehabilitation (vision, auditory, and gustatory).

In this article we present Ecological Augmented Reality (E-AR), an approach that questions the theoretical assumptions of mainstream Augmented Reality (AR). The development of AR systems to date presupposes an information-processing theory of perception that hinders the potential of the field. Generally, in AR devices, virtual symbolic information is superimposed upon the environment in such a way that the real and the virtual may be processed, informationally speaking, in tandem. Thus, we find information in reality itself, as well as virtual symbolic information. But by increasing the burden of symbolic crunching, AR devices run the risk of saturating the user of the technology. AR systems developed under the principles of an ecological psychology may contribute to new and better levels of performance and adaptation to the user’s perceptual abilities. Our proposal is to develop AR devices such that reality itself is augmented non-symbolically by blending real and virtual layers/information. Although there are seldom AR devices in the market that are designed ecologically, two fields of research may well bring inspiration to AR developers. These are the design and manipulation of real objects, and ecological research in the field of sensory substitution. We consider them both in turn with an eye to putting forward a framework that eschews any type of information-processing regarding the nature of our psychological processes. Ultimately, our aim is to provide some guidelines for the exploration of an ecological trend in AR applications.

Mobile devices, tablets, smart phones, and now wearable technologies have changed our world, including how we perceive and interact with others and our environment. This change has potential for both positive and negative outcomes, especially for people with sensory, cognitive, and physical impairments. This work looks at some possible eventualities, with a focus on how these technologies may change the future for people with disabilities.

Summary We examined the development of new sensing abilities in adults by training participants to perceive remote objects through their fingers. Using an Active-Sensing based sensory Substitution device (ASenSub), participants quickly learned to perceive fast via the new modality and preserved their high performance for more than 20 months. Both sighted and blind participants exhibited almost complete transfer of performance from 2D images to novel 3D physical objects. Perceptual accuracy and speed using the ASenSub were, on average, 300% and 600% better than previous reports for 2D images and 3D objects. This improvement is attributed to the ability of the participants to employ their own motor-sensory strategies. Sighted participants dominant strategy was based on motor-sensory convergence on the most informative regions of objects, similarly to fixation patterns in vision. Congenitally, blind participants did not show such a tendency, and many of their exploratory procedures resembled those observed with natural touch., Graphical Abstract, Highlights • We describe sensory substitution based on active-sensing principles (ASenSub) • Perception via ASenSub emerges rapidly and transferred easily from 2D to 3D objects • Sighted and blind users employ visual- and tactile-like strategies, respectively • Use of natural motor-sensory strategies facilitates learning and performance, Biological Sciences; Neuroscience; Sensory Neuroscience; Cognitive Neuroscience

Visual sensory substitution devices provide a non-surgical and flexible approach to vision rehabilitation in the blind. These devices convert images taken by a camera into cross-modal sensory signals that are presented as a surrogate for direct visual input. While previous work has demonstrated that the visual cortex of blind subjects is recruited during sensory substitution, the cognitive basis of this activation remains incompletely understood. To test the hypothesis that top-down input provides a significant contribution to this activation, we performed functional MRI scanning in 11 blind (7 acquired and 4 congenital) and 11 sighted subjects under two conditions: passive listening of image-encoded soundscapes before sensory substitution training and active interpretation of the same auditory sensory substitution signals after a 10-minute training session. We found that the modulation of visual cortex activity due to active interpretation was significantly stronger in the blind over sighted subjects. In addition, congenitally blind subjects showed stronger task-induced modulation in the visual cortex than acquired blind subjects. In a parallel experiment, we scanned 18 blind (11 acquired and 7 congenital) and 18 sighted subjects at rest to investigate alterations in functional connectivity due to visual deprivation. The results demonstrated that visual cortex connectivity of the blind shifted away from sensory networks and toward known areas of top-down input. Taken together, our data support the model of the brain, including the visual system, as a highly flexible task-based and not sensory-based machine.

The research field on sensory substitution devices has strong implications for theoretical work on perceptual consciousness. One of these implications concerns the extent to which the devices allow distal attribution. The present study applies a classic empirical approach on the perception of affordances to the field of sensory substitution. The reported experiment considers the perception of the stair-climbing affordance. Participants judged the climbability of steps apprehended through a vibrotactile sensory substitution device. If measured with standard metric units, climbability judgments of tall and short participants differed, but if measured in units of leg length, judgments did not differ. These results are similar to paradigmatic results in regular visual perception. We conclude that our sensory substitution device allows the perception of affordances. More generally, we argue that the theory of affordances may enrich theoretical debates concerning sensory substitution to a larger extent than has hitherto been the case.

In this paper, we present the classification of electroencephalograph (EEG) signals produced by the first-octave musical notes of the piano as stimuli. The EEG classification of musical notes is attempted for the first time, to the best of our knowledge. This type of classification could be applied towards the development of Brain-Computer Interfaces (BCIs) for the composition of music via thought as well as the definition of mappings between different stimuli for Sensory Substitution Devices (SSDs) that are based on their actual impact on brain signals and thus serve better the purpose of SSDs, which is to translate between senses at the perceptual level. Event-Related Spectral Perturbations (ERSP) are extracted as features and are fed into a Support Vector Machine (SVM) classifier. Our aim was to classify musical notes as C, C#, D, D#, E, F, F#, G, G#, A, A#, B and we have achieved it with an average accuracy of 70%.

One of the most exciting recent findings in neuroscience has been the capacity for neural plasticity in adult humans and animals. Studies of perceptual learning have provided key insights into the mechanisms of neural plasticity and the changes in functional neuroanatomy that it affords. Key questions in this field of research concern how practice of a task leads to specific or general improvement. Although much of this work has been carried out with a focus on a single sensory modality, primarily visual, there is increasing interest in multisensory perceptual learning. Here we will examine how advances in perceptual learning research both inform and can be informed by the development and advancement of sensory substitution devices for blind persons. To allow 'sight' to occur in the absence of visual input through the eyes, visual information can be transformed by a sensory substitution device into a representation that can be processed as sound or touch, and thus give one the potential to 'see' through the ears or tongue. Investigations of auditory, visual and multisensory perceptual learning can have key benefits for the advancement of sensory substitution, and the study of sensory deprivation and sensory substitution likewise will further the understanding of perceptual learning in general and the reverse hierarchy theory in particular. It also has significant importance for the developing understanding of the brain in metamodal terms, where functional brain areas might be best defined by the computations they carry out rather than by their sensory-specific processing role.

Gesture-based interaction adds a new level of immersion to video games, but players who are blind are unable to play them as these games use visual cues to indicate what gesture to provide and when. Though visual cues can be substituted with audio or haptic cues, this often requires access to the source code, which is not attainable for commercial games. We present a solution that uses real-time video analysis to detect the presence of a particular visual cue, which is then substituted with a vibrotactile cue that is provided with an external controller. A user study with 28 sighted participants with a popular commercial gesture based game showed no significant difference in performance between visual and vibrotactile feedback. A follow up study with seven visually impaired participants revealed no significant difference in performance between both groups of users. Both studies demonstrate the feasibility of real-time sensory substitution as a cost-effective approach for making gesture-based video games accessible to players who are blind.

This research considers a sensory substitution device that allows the exploration of the environment through normal walking, leaning and standing. The device includes an array of 24 coin motors placed vertically on the torso, with the intensity of vibration of each motor being a function of the distance to the first-encountered object. Thresholds were determined for the detection of ground-level obstacles (raised target platforms). On average, blindfolded participants were able to detect platforms with heights of 9 to 17 cm, but the thresholds differed for different experimental conditions. Experiment 1 showed that the detection threshold is lower for use with exploratory movements than for use without exploratory movements. Experiments 2 and 3 compared dynamic groups, who made exploratory movements and received vibrotactile flow contingent on their movements, with yoked groups, who received the same vibrotactile flow as the dynamic groups independently of their own movements. The detection thresholds were lower for the dynamic groups than for the yoked groups, meaning that the contingency of vibrotactile flow on exploratory movements is important beyond a higher-order vibrotactile flow by itself.

SummaryUsing a visual-to-auditory sensory-substitution algorithm, congenitally fully blind adults were taught to read and recognize complex images using “soundscapes”—sounds topographically representing images. fMRI was used to examine key questions regarding the visual word form area (VWFA): its selectivity for letters over other visual categories without visual experience, its feature tolerance for reading in a novel sensory modality, and its plasticity for scripts learned in adulthood. The blind activated the VWFA specifically and selectively during the processing of letter soundscapes relative to both textures and visually complex object categories and relative to mental imagery and semantic-content controls. Further, VWFA recruitment for reading soundscapes emerged after 2 hr of training in a blind adult on a novel script. Therefore, the VWFA shows category selectivity regardless of input sensory modality, visual experience, and long-term familiarity or expertise with the script. The VWFA may perform a flexible task-specific rather than sensory-specific computation, possibly linking letter shapes to phonology.

Chemical images which are often computed from hyperspectral images contain the spatial distribution of chemical information of a scene. For many applications visualizing such images on computer screens is sufficient, however there are cases where there is a need to combine the chemical images more naturally with human vision. This is especially true for interactive work where chemical images are being rapidly updated to the user. Effective integration of spatial information in general from external sources with vision is a challenge. One approach is to overlay the view of the real physical world with computer-generated graphics as in augmented reality. However such cluttering of the visual field with computer-generated graphics may confuse the user and reduce functionality. Another is projecting the chemical images back onto the scene under study in order to render the chemical information in situ to the user. This approach, however has challenges in connection with very small and very large scenes under investigation. An alternative approach is here investigated based on the possibility of enhancing the human vision system using a sensory substitution device. Such devices enables a person to sense spatial information conveyed through sensory channels other than the eye, such as hearing and sense of touch. Results presented support the claim that spatial chemical information from a hyperspectral camera can be conveyed to the brain through a sensory channel different from the eyes. As this is tested on a sighted subject it effectively provides an extension of the human vision system to incorporate chemical information which otherwise is invisible to the naked eye.

Perception is influenced both by the immediate pattern of sensory inputs and by memories acquired through prior experiences with the world. Throughout much of its illustrious history, however, study of the cellular basis of perception has focused on neuronal structures and events that underlie the detection and discrimination of sensory stimuli. Relatively little attention has been paid to the means by which memories interact with incoming sensory signals. Building upon recent neurophysiological/behavioral studies of the cortical substrates of visual associative memory, I propose a specific functional process by which stored information about the world supplements sensory inputs to yield neuronal signals that can account for visual perceptual experience. This perspective represents a significant shift in the way we think about the cellular bases of perception.

Neuronal activity in early visual cortex depends on attention shifts but the contribution to working memory has remained unclear. Here, we examine neuronal activity in the different layers of the primary visual cortex (V1) in an attention-demanding and a working memory task. A current-source density analysis reveales top-down inputs in the superficial layers and layer 5, and an increase in neuronal firing rates most pronounced in the superficial and deep layers and weaker in input layer 4. This increased activity is strongest in the attention task but it is also highly reliable during working memory delays. A visual mask erases the V1 memory activity, but it reappeares at a later point in time. These results provide new insights in the laminar circuits involved in the top-down modulation of activity in early visual cortex in the presence and absence of visual stimuli., The effect of working memory on activity in primary visual cortex (V1) is not well understood. Here the authors report a clear influence of both working memory and attention on spiking activity in the superficial and deep layers of V1 with a weaker influence on input layer 4.

Introduction/Background Despite technological advances upper limb prosthesis, myoelectric control requires a long learning process. The absence of sensory feedback is very likely to impinge the appropriation of the prosthesis. We explored a sensory substitution alternative with various configurations and settings for vibrotactile feedback and myoelectric parameters. Material and method Six vibrators were placed on a line or circumferentially around the arm. Space intervals between vibrators were absolute (2 cm) or proportional to the length or the circumference of the arm. Dispositions as longitudinal proportional and absolute, circular proportional and absolute were tested. Sixty, 100 and 140 ms of duration and intensities of 62.5; 100; and 167 mA were explored. Estimate location of the vibration, and perceived intensity between 0 (no feeling) to 3 (strong) were assessed. EMGs from biceps and triceps were recorded during isometric contraction to control the elbow velocity of an avatar displayed on a screen. EMGs were filtered and normalized from maximal contraction. A threshold of minimum muscle activity and a gain of the velocity control were adjusted to allow fast an intuitive control. Results The circular proportional disposition elicited better discrimination results than the 3 others dispositions (P A threshold between 5 and 7.5% of the maximum force, and a velocity gain varying from 0.3 to 1.2 rad.s-1 for a change in muscle contraction of 10% MVC were found to enable precise control of the avatar. Conclusion The circular proportional disposition of the vibrator is a well suited configuration for sensory substitution. Our next step will be to combine sensory feedback given by vibrator to the myoelectric control. This sensory substitution could improve prosthesis control, and may attenuate phantom limb pain.

The present study evaluated the effectiveness of electrotactile tongue biofeedback (BrainPort((R))) as a sensory substitute for the vestibular apparatus in patients with bilateral vestibular loss (BVL) who did not have a good response to conventional vestibular rehabilitation (VR). Seven patients with BVL were trained to use the device. Stimulation on the surface of the tongue was created by a dynamic pattern of electrical pulses and the patient was able to adjust the intensity of stimulation and spatially centralize the stimulus on the electrode array. Patients were directed to continuously adjust head orientation and to maintain the stimulus pattern at the center of the array. Postural tasks that present progressive difficulties were given during the use of the device. Pre- and post-treatment distribution of the sensory organization test (SOT) composite score showed an average value of 38.3+/-8.7 and 59.9+/-11.3, respectively, indicating a statistically significant improvement (p=0.01). Electrotactile tongue biofeedback significantly improved the postural control of the study group, even if they had not improved with conventional VR. The electrotactile tongue biofeedback system was able to supply additional information about head position with respect to gravitational vertical orientation in the absence of vestibular input, improving postural control. Patients with BVL can integrate electrotactile information in their postural control in order to improve stability after conventional VR. These results were obtained and verified not only by the subjective questionnaire but also by the SOT composite score. The limitations of the study are the small sample size and short duration of the follow-up. The current findings show that the sensory substitution mediated by electrotactile tongue biofeedback may contribute to the improved balance experienced by these patients compared to VR.

In this report, the phenomenology of two blind users of a sensory substitution device - "The vOICe" - that converts visual images to auditory signals is described. The users both report detailed visual phenomenology that developed within months of immersive use and has continued to evolve over a period of years. This visual phenomenology, although triggered through use of The vOICe, is likely to depend not only on online visualization of the auditory signal but also on the users' previous (albeit distant) experience of veridical vision (e.g. knowledge of shapes and visual perspective). Once established, the sensory substitution mapping between the auditory and visual domains is not confined to when the device is worn and, thus, may constitute an example of acquired synaesthesia.

The notion that blindness leads to superior non-visual abilities has been postulated for centuries. Compared to sighted individuals, blind individuals show different patterns of brain activation when performing auditory tasks. To date, no study has controlled for musical experience, which is known to influence auditory skills. The present study tested 33 blind (11 congenital, 11 early-blind, 11 late-blind) participants and 33 matched sighted controls. We showed that the performance of blind participants was better than that of sighted participants on a range of auditory perception tasks, even when musical experience was controlled for. This advantage was observed only for individuals who became blind early in life, and was even more pronounced for individuals who were blind from birth. Years of blindness did not predict task performance. Here, we provide compelling evidence that superior auditory abilities in blind individuals are not explained by musical experience alone. These results have implications for the development of sensory substitution devices, particularly for late-blind individuals.

How in real-life or through the use of technical devices can we recognize the presence of other persons and under what conditions can we differentiate them from objects? In order to approach this question, in the study reported here we explored the most basic conditions necessary for participants to recognize the presence of another person during a perceptual interaction. We created a mini-network of two minimalist devices and investigated whether participants were able to differentiate the perception of another person from the perception of a fixed and a mobile object even when the pattern of sensory stimulation was reduced to a bare minimum. We show that participants can recognize when the all-or-none tactile stimulation they experienced was attributable to an encounter with the other participant's avatar or the mobile object rather than with a fixed object. Participants were also able to establish different strategies in order to favor the situations of mutual perception. Thus, in the minimalist conditions of our experiment, the perception of another intentional subject was not based purely on any particular shape or objective trajectories of displacement; it was also based on properties that are intrinsic to the joint perceptual activity itself.

We refer to channels of communication which link the user to various electronic appliances and computers as cognitive communication channels. One especially interesting research topic related to cognitive communication channels deals with a special application called sensory substitution, when information is conveyed through a channel other than the one that is normally used. Besides offering a glimpse of hope to those living with injured sensory organs, sensory substitution can have benefits when designing user interfaces in terms of cost-effectiveness, the reduction of network delays, as well as the prevention of channel overloading. One such application in robot control could be the substitution of haptic feedback gloves (which are costly, bulky and have relatively few vibration states) using parameter-rich audio interfaces. In this paper, we experiment with a model capable of representing geometrical properties of surfaces in terms of audio-substituted tactile percepts, while at the same time modeling the more subjective dimension of user awareness.

The See Color interface transforms a small portion of a colored video image into sound sources represented by spatialized musical instruments. Basically, the conversion of colors into sounds is achieved by quantization of the HSL color system. Our purpose is to provide visually impaired individuals with a capability of perception of the environment in real time. In this work we present the system principles of design and several experiments that have been carried out by several blindfolded persons with See ColOr prototypes related to static pictures on a tablet and simple video images. The goal of the first experiment was to identify the colors of static pictures' main features and then to interpret the image scenes. Although learning all instrument sounds in only a training session was too difficult, participants found that colors were helpful to limit the possible image interpretations. The experiments on the analysis of static pictures suggested that the order of magnitude of the slow down factor related to the use of the auditory channel, instead of the visual channel could correspond to the order of magnitude related to the ratio of visual channel capacity to auditory channel capacity. Afterwards, two experiments based on a head mounted camera have been performed. The first experiment pertaining to object manipulation is based on the pairing of colored socks, while the second experiment is related to outdoor navigation with the goal of following a colored serpentine painted on the ground. The ''socks'' experiment demonstrated that blindfolded individuals were able to accurately match pairs of colored socks. The same participants with the addition of a blind individual successfully followed a red serpentine painted on the ground for more than 80m. According to task time durations, the order of magnitude of the slow down factor related to the ''socks'' and ''serpentine'' experiments could be equal to one. From a cognitive perspective this would be consistent with the fact that these two tasks are simpler than the interpretation of image scenes.

International audience; The vestibular system contributes to a wide range of functions from reflexes to spatial representation. This paper reviews behavioral, perceptive, and cognitive data that highlight the role of changes in internal spatial representation on the vestibular syndrome. Firstly, we review how visual vertical perception and postural orientation depend on multiple reference frames and multisensory integration and how reference frames are selected according to the status of the peripheral vestibular system (i.e., unilateral or bilateral hyporeflexia), the environmental constraints (i.e., sensory cues), and the postural constraints (i.e., balance control). We show how changes in reference frames are able to modify vestibular lesion-induced postural and locomotor deficits and propose that fast changes in reference frame may be considered as fast-adaptive processes after vestibular loss. Secondly, we review data dealing with the influence of vestibular loss on higher levels of internal representation sustaining spatial orientation and navigation. Particular emphasis is placed on spatial performance according to task complexity (i.e., the required level of spatial knowledge) and to the sensory cues available to de. ne the position and orientation within the environment (i.e., real navigation in darkness or visual virtual navigation without any actual self-motion). We suggest that vestibular signals are necessary for other sensory cues to be properly integrated and that vestibular cues are involved in extrapersonal space representation. In this respect, vestibular-induced changes would be based on a dynamic mental representation of space that is continuously updated and that supports fast-adaptive processes. (c) 2008 Elsevier Masson SAS. All rights reserved.

We investigated the effects of an electro-tactile vestibular substitution system (EVSS) on upright postural control in 12 unilateral vestibular-defective patients. The underlying principle of this system consists in supplying the user with additional information about his/her head orientation/motion with respect to gravitational vertical, normally provided by the vestibular system, through electro-tactile stimulation of his/her tongue. Subjects were asked to stand as immobile as possible with their eyes closed in two No-EVSS and EVSS conditions. Reduced centre-of-foot pressure displacements were observed in the EVSS relative to the No-EVSS condition. These results, demonstrating the effectiveness of the EVSS system in improving upright postural control in unilateral vestibular-defective patients, could have implications in clinical and rehabilitative areas.

Cerebral plasticity, which is the dynamic potential of the brain to reorganize itself during ontogeny, learning, or following damage, has been widely studied in the last decade, in vitro, in animals, and also in humans since the development of functional neuroimaging. In the first part of this review, the main hypotheses about the pathophysiological mechanisms underlying plasticity are presented. At a microscopic level, modulations of synaptic efficacy, unmasking of latent connections, phenotypic modifications and neurogenesis have been identified. At a macroscopic level, diaschisis, functional redundancies, sensory substitution and morphological changes have been described. In the second part, the behavioral consequences of such cerebral phenomena in physiology, namely the "natural" plasticity, are analyzed in humans. The review concludes on the therapeutic implications provided by a better understanding of these mechanisms of brain reshaping. Indeed, this plastic potential might be 'guided' in neurological diseases, using rehabilitation, pharmacological drugs, transcranial magnetic stimulation, neurosurgical methods, and even new techniques of brain-computer interface - in order to improve the quality of life of patients with damaged nervous systems.

Sensory replacement and substitution techniques based on a camera system can provide valuable spatial wayfinding information for people that are blind or visually impaired when conveyed via an underutilized sense such as touch (haptics). Mapping visual to tactile form necessitates selectivity in the information conveyed because the tactile channel is band-limited. Our focus for sensory substitution is wayfinding, in particular, providing medium distance information, up to 10 m, that is beyond the reach of the long white cane while not addressing global positioning GPS. The key research questions include what information is crucial for extraction from the computer vision system, how much spatial information can be conveyed in tactile form and what is this mapping? The engineering challenge is to embed the system into an affordable, easy to learn, non-invasive, light, portable and wearable instantiation. A portable system consisting of a stereo camera and haptic glove was developed and tested. Off-line computer vision algorithms were developed for acquiring real time depth from stereo, face detection as well as context and landmark detection. Low power and portable haptic devices of various embodiments were constructed and energy transfer properties as well as perceptual stimuli were explored. Behavioural experiments produced trajectories that were less likely to hit obstacles. Obstacles and the ground plane were adequately extracted for navigation. Intensity of haptic motor vibration was found to be appropriate for providing depth. Conveying selective camera image content via haptics can help wayfinding and provides the foundation for seeing by touch.

Objectives Direct haptic (force or tactile) feedback is not yet available in commercial robotic surgical systems. Previous work by our group and others suggests that haptic feedback might significantly enhance the execution of surgical tasks requiring fine suture manipulation, specifically those encountered in cardiothoracic surgery. We studied the effects of substituting direct haptic feedback with visual and auditory cues to provide the operating surgeon with a representation of the forces he or she is applying with robotic telemanipulators. Methods Using the robotic da Vinci surgical system (Intuitive Surgical, Inc, Sunnyvale, Calif), we compared applied forces during a standardized surgical knot-tying task under 4 different sensory-substitution scenarios: no feedback, auditory feedback, visual feedback, and combined auditory-visual feedback. Results The forces applied with these sensory-substitution modes more closely approximate suture tensions achieved under ideal haptic conditions (ie, hand ties) than forces applied without such sensory feedback. The consistency of applied forces during robot-assisted suture tying aided by visual feedback or combined auditory-visual feedback sensory substitution is superior to that achieved with hand ties. Robot-assisted ties aided with auditory feedback revealed levels of consistency that were generally equivalent or superior to those attained with hand ties. Visual feedback and auditory feedback improve the consistency of robotically applied forces. Conclusions Sensory substitution, in the form of visual feedback, auditory feedback, or both, confers quantifiable advantages in applied force accuracy and consistency during the performance of a simple surgical task.

Sensory substitution refers to the capacity of the brain to replace the functions of a lost sense by another sensory modality. This cross-modal plasticity has been documented both in animals and humans deprived of a particular sensory modality, such as vision or audition. Discovering new ways to exploit this cross-modal plasticity may help to optimize the recovery of sensory loss. The most commonly used form of sensory substitution is Braille reading, which enables the blind to read by using the somatosensory system. Recently, a human–machine interface, the tongue display unit (TDU), which uses the tongue as a medium for visual substitution in blind subjects, has been developed. We trained six congenitally blind and five blindfolded, sighted controls to use the TDU to perform a visual orientation discrimination task. Subjects were positron emission tomography (PET) scanned before and after an intensive 1-week training program with the TDU. Before training, no increased activity was measured in the visual cortex of either group during the orientation detection task. However, after training, patterned stimulation of the tongue activated the visual cortex in congenitally blind subjects. Sighted controls did not show occipital activation post-training despite equivalent performance on the same task. These data reveal the development of cross-modal plasticity in the brains of congenitally blind subjects. They further show that the time course of neuroplasticity in humans can be remarkably rapid.

A key concern in the design of a laparoscopic virtual simulator is the level of realism necessary for proper training. An analysis of tissue consistency perception of novel and expert surgeons has been performed, studying three components of this skill: visual cues, tactile information and previous knowledge and experience. Experiments of tissue identification skill with either visual or tactile stimuli have been also accomplished. Preliminary results are allowing characterizing differences on tissue consistency perception between novel and expert laparoscopic surgeons. They also suggest a possible sensory substitution of tactile feedback with visual information in laparoscopic surgery.

Recent advances in the instrumentation technology of sensory substitution have presented new opportunities to develop systems for compensation of sensory loss. In sensory substitution (e.g. of sight or vestibular function), information from an artificial receptor is coupled to the brain via a human-machine interface. The brain is able to use this information in place of that usually transmitted from an intact sense organ. Both auditory and tactile systems show promise for practical sensory substitution interface sites. This research provides experimental tools for examining brain plasticity and has implications for perceptual and cognition studies more generally.

Bach-y-Rita P. Late postacute neurologic rehabilitation: neuroscience, engineering, and clinical programs. Arch Phys Med Rehabil 2003;84:1100–8. This lecture highlights my career in rehabilitation research. My principal efforts in rehabilitation have been to study (1) mechanisms of brain plasticity related to reorganization of the brain and recovery of function; (2) late postacute rehabilitation; (3) sensory substitution; and (4) rehabilitation engineering. A principal goal has been to aid in the development of a strong scientific base in rehabilitation.

The ‘visual’ acuity of blind persons perceiving information through a newly developed human–machine interface, with an array of electrical stimulators on the tongue, has been quantified using a standard Ophthalmological test (Snellen Tumbling E). Acuity without training averaged 20/860. This doubled with 9 h of training. The interface may lead to practical devices for persons with sensory loss such as blindness, and offers a means of exploring late brain plasticity.

By modifying patterns of activity in the midbrain and thalamus, cortical feedback may be involved in selecting sensory signals for further analysis.

Electrical muscle stimulators (EMS) have been combined with a reciprocating gait orthosis (RGO) to produce an EMS-RGO system for constant velocity ambulation exercise in a spinal cord injured (SCI) individual. The objective of this study is to evaluate an auditory feedback system (AFS) used by the SCI subject when ambulating with the EMS-RGO. Three different types of auditory signals (M = metronome alone, MSL = metronome plus stride length information, and NONE = no auditory feedback) were examined at three different constant walking velocities (WV1 = 0.64 kph, WV2 = 1.22 kph, and WV3 = 1.80 kph). The experimental design was 3 x 3 full factorial with repeated measures, and the dependent variable was the absolute error (AE) of walking distance (absolute value of the distance the subject walked in one minute subtracted from the target distance). A block diagram and circuit schematic of the AFS is provided. The results indicate that at WV1 there is no significant difference of AE among the three auditory conditions (M, MSL and NONE). At WV2, the AE for the NONE auditory condition is significantly greater than the AE for M and MSL (p0.05). At WV3, the AE for the NONE auditory condition is significantly less than the AE for M and MSL (p0.05). Finally, there was no significant difference in AE between both the M and MSL auditory conditions at any of the three walking velocities. This study indicates that an adequate sensory feedback system for constant velocity control of ambulation in this SCI subject consists of metronome only auditory feedback.(ABSTRACT TRUNCATED AT 250 WORDS)

The objective of the research presented in this paper was to study the capabilities of sensory substitution for force feedback, presented to the operator of a teleoperation system, through the tactile and auditory senses. Traditional bilateral force feedback or force reflection, which applies forces to a human operator's hand or arm muscles, while generally beneficial can be limited in the operator's ability to perceive small force values. Sensory substitution for force feedback was shown in this study to increase the operator's ability to perceive small forces by allowing an increase in the effective feedback gain without risking instability or impeding the operator's inputs to the system.

Cats deprived visually from birth show few overt impairments in their natural behavior. Therefore, they seem well suited as an animal model for the study of compensatory plasticity after early vision loss. It can be demonstrated that binocularly deprived cats show improved abilities of auditory localization, and at least equal tactile behavior compared to normal controls. Within the anterior ectosylvian cortex of binocularly deprived cats, where different sensory modalities come together, the anterior ectosylvian visual area is completely taken over by auditory and somatosensory inputs. Furthermore, the auditory spatial tuning of single units in this cortical region is sharpened significantly as a result of visual deprivation. Somatosensory compensation for early loss of vision can be demonstrated by a hypertrophy of the facial vibrissae, and a corresponding expansion of their central representation in the somatosensory cortex of binocularly deprived animals. The compensatory changes in the cortex can be explained by a reorganization of sensory representations under the guidance of sensorimotor feedback rather than by instruction through an extraneous 'supervisory' signal. These processes might form the neural basis of sensory substitution in blind humans.

The purpose of this review is to consider new sensory rehabilitation avenues in the context of the brain's remarkable ability to reorganize itself following sensory deprivation. Here, deafness and blindness are taken as two illustrative models. Mainly, two promising rehabilitative strategies based on opposing theoretical principles will be considered: sensory substitution and neuroprostheses. Sensory substitution makes use of the remaining intact senses to provide blind or deaf individuals with coded information of the lost sensory system. This technique thus benefits from added neural resources in the processing of the remaining senses resulting from crossmodal plasticity, which is thought to be coupled with behavioral enhancements in the intact senses. On the other hand, neuroprostheses represent an invasive approach aimed at stimulating the deprived sensory system directly in order to restore, at least partially, its functioning. This technique therefore relies on the neuronal integrity of the brain areas normally dedicated to the deprived sense and is rather hindered by the compensatory reorganization observed in the deprived cortex. Here, we stress that our understanding of the neuroplastic changes that occur in sensory-deprived individuals may help guide the design and the implementation of such rehabilitative methods.

Sensory substitution has advanced remarkably over the past 35 years since first introduced to the scientific literature by Paul Bach-y-Rita. In this issue dedicated to his memory, we describe a collection of reviews that assess the current state of neuroscience research on sensory substitution, visual rehabilitation, and multisensory processes.

The objective of this research was to study the capabilities of sensory substitution for force feedback through the tactile and auditory senses for teleoperation tasks, with and without time delay. The motivation and potential benefits of sensory substitution for force feedback with vibrotactile and auditory displays are discussed. Teleoperator experiments that examined the presentation of basic force information through object contact tasks indicated that operator performance was improved by using the vibrotactile and auditory displays to present force information. Further, the vibrotactile and auditory displays compared favorably to traditional bilateral force feedback. Common manipulation experiments with peg-in-hole tasks of varying complexity were also conducted and showed that when the subjects' view was fully obstructed, the subjects were able to successfully complete the task by using either of the sensory substitution displays. Sensory substitution was also tested in the presence of a 3 sec time delay and significantly improved performance without instabilities.

A major limitation in the utilization of a functional electrical stimulation (FES) orthosis for routine, daily standing and walking of the spinal-cord-injured person is that visual monitoring is required to maintain balance and the walking pace. For standing and walking to be continuous and automatic with such an orthosis, a closed-loop sensory feedback system is proposed and evaluated; it provides vibrotactile feedback as a substitute to one's own visual sensation. Eight blindfolded experimental subjects were utilized as ‘imitators’ to interpret the footfalls of a second person (the pace setter). The experimental objective was to test the hypothesis that sufficient information could be transferred by way of the sensory (tactile) feedback system to the ‘imitator’ and to determine effectively foot position and anticipate the next step of the pacesetter. Quantitative analysis evaluated the effect of three different levels of training, under two different levels of cognitive load. The results disclosed a significant improvement in subject performance at the higher training levels compared with the ‘no training’ level ( P = 0.01). Neither the cognitive load nor the interaction of training and cognitive load altered significantly the effect of training on subject performance. The experimental hypothesis is therefore satisfied that sufficient information was indeed transferred using the apparatus described. Such information (when utilized in conjunction with a thorough training programme) could be used in a practical sense by a paraplegic individual to interpret his own foot steps. Through continued use and training, it is likely that this information could become subconscious and automatic. Therefore, the spinal-cord-injured person would walk with an FES orthosis in various environments and minimize or remove their reliance on visual sensory information.

Future research in the neurosciences can best be understood in the context of NASA's life sciences goals in the near term (1990-95), mid term (1995-2000), and long term (2000 and beyond). Since NASA is planning short-duration Spacelab and International Microgravity Laboratory (IML) flights for many years to come, the acute effects of exposure to microgravity will continue to be of experimental and operational interest in the near term. To this end, major new areas of research will be devoted to ground-based studies of preflight adaptation trainers and their efficacy in preventing or reducing the incidence of space motion sickness. In addition, an extensive series of studies of the vestibular system will be conducted inflight on the IML-1 mission The IML-2 mission will emphasize behavior and performance, biological rhythms, and further vestibular studies. In the mid-term period, Spacelab missions will employ new technology such as magnetic recording techniques in order to evaluate changes in the processing of sensory and motor inputs at the brainstem and cortical level during exposure to microgravity. Two Space Life Sciences (SLS) missions planned for the mid to late 1990's, SLS-4 and SLS-5, will utilize an onboard centrifuge facility that will enable investigators to study the effects of partial gravity on sensory and motor function. In the long term (2000 and beyond), Space Station Freedom and long-duration missions will provide opportunities to explore new options in the neurosciences, such as sensory substitution and augmentation, through the use of physical sensors to provide three-dimensional tactile-visual, tactile-auditory and tactile-somatosensory inputs. The use of this technology will be extremely important in the area of robotic telepresence. Finally, Space Station Freedom and proposed LifeSat missions will provide neuroscientists the opportunity to study the effects of partial gravity and microgravity on neuronal plasticity.

This paper investigates the use of tactile and auditory displays to present feedback to the pilot of a spacecraft or the operator of a space teleoperated system. Force feedback is given particular attention for teleoperator scenarios in the presence of a time delay. The motivation for and potential benefits of developing bisensory feedback displays are identified. Pioneering research in developing auditory and tactile displays is discussed. Several models of the human operator concerning the processing of bisensory information are outlined, along with a discussion of presenting redundant information across sensory modalities. Preliminary experimental results concerning sensory substitution of force feedback with a vibrotactile display are presented along with future research plans.