Search

People conceptualize auditory pitch as vertical space: low and high pitch correspond to low and high space, respectively. The strength of this cross-modal correspondence, however, seems to vary across different cultural contexts and a debate on the different factors underlying this variation is currently taking place. According to one hypothesis, pitch mappings are semantically mediated. For instance, the use of conventional metaphors such as “falling” or “rising” melodies strengthens a pitch-height mapping to the detriment of other possible mappings (e.g., pitch as bright/dark color or small/big size). Hence, entrenched pitch terms shape specific conceptualizations. The deterministic role of language is called into question by the hypothesis that different pitch mappings share a less constraining conceptual basis. As such, conceptual primitives may be concretizedad hocinto specific domains so that more local variation is possible. This claim is supported, for instance, by the finding that musicians use language-congruent (conventional) and language-incongruent (ad hoc) mappings interchangeably. The present paper substantiates this observation by investigating the head movements of musically trained and untrained speakers of Dutch in a melody reproduction task, as embodied instantiations of a vertical conceptualization of pitch. The OpenPose algorithm was used to track the movement trajectories in detail. The results show that untrained participants systematically made language-congruent movements, while trained participants showed more diverse behaviors, including language-incongruent movements. The difference between the two groups could not be attributed to the level of accuracy in the singing performances. In sum, this study argues for a joint consideration of more entrenched (e.g., linguistic metaphors) and more context-dependent (e.g., musical training and task) factors in accounting for variability in pitch representations.

Auditory pitch is represented on a vertical continuum (e.g., high vs. low). In three experiments, we examined whether the kinesthetic experience of moving in a particular direction (e.g., walking up vs. down stairs; reaching up vs. down) would affect judgments of auditory pitch. Participants listened to three tones twice each, once while moving upward and once while moving downward, and estimated the pitch of each tone. In all experiments, participants' judgments of the tones' pitch were biased in the direction of their movement. Moreover, this effect is not due to visibility of the movement or to using a numerical response method. Our results suggest that kinesthetic information from one's own bodily movements biases pitch estimation, and several possible mechanisms for the effect are discussed.

The present study compared how Cantonese-speaking children with Autism Spectrum Disorder (ASD) and their typically developing counterparts perceived speech pitch and non-speech pitch information using ERP measurements. Sixteen children with ASD (mean age = 10.42 years, SD = 2.12 years) and sixteen normal controls (mean age = 9.48 years, SD =.86 years) participated in two experiments, in which Cantonese lexical tone contrasts and non-speech pitch variations were presented to children following an oddball paradigm when they watched a silent movie. The results showed that: 1) When processing speech pitch contour, the two groups did not differ in the amplitude of mismatch response (p-MMR), while typically developing controls showed larger mismatch negativity (MMN) responses than children with ASD. In the processing of speech pitch height, more positive p-MMR was observed among children with ASD than among normal controls and stronger MMN was found for typically developing children than for children with ASD. 2) For the processing of non-speech pitch, MMN rather than p-MMR was observed and the two groups did not differ significantly with each other in the amplitudes of MMN. These results indicated that Cantonese-speaking children with ASD manifested impaired ability when processing speech pitch information (i.e., lexical tone), which was in line with previous research. However, they did not show the advantage in processing non-speech or auditory pitch information, which was not in agreement with the previous studies. Results were discussed from the perspective of how language background (i.e., Cantonese) might shape the perceptive abilities of children.

Both time-based (when) and feature-based (what) aspects of attention facilitate behavior, so it is natural to hypothesize additive effects. We tested this conjecture by recording response behavior and electroencephalographic (EEG) data to auditory pitch changes, embedded at different time lags in a continuous sound stream. Participants reacted more rapidly to larger rather than smaller feature change magnitudes (deviancy), as well as to changes appearing after longer rather than shorter waiting times (hazard rate of response times). However, the feature and time dimensions of attention separately contributed to response speed, with no significant interaction. Notably, phase coherence at low frequencies (delta and theta bands, 1–7 Hz) predominantly reflected attention capture by feature changes, while oscillatory power at higher frequency bands, alpha (8–12 Hz) and beta (13–25 Hz) reflected the orienting of attention in time. Power and phase coherence predicted different portions of response speed variance, suggesting a division of labor in encoding sensory attention in complex auditory scenes.

A common experimental task used to study the accuracy of estimating when a moving object arrives at a designated location is the time-to-contact (TTC) task. The previous studies have shown evidence that sound motion cues influence TTC estimates of a visual moving object. However, the extent to which sound can influence TTC of visual targets still remains unclear. Some studies on the crossmodal correspondence between pitch and speed suggest that descending pitch sounds are associated with faster speeds compared to ascending pitch sounds due to an internal model of gravity. Other studies have shown an opposite pitch-speed mapping (i.e., ascending pitch associated with faster speeds) and no influence of gravity heuristics. Here, we explored whether auditory pitch glides, a continuous pure tone sound either ascending or descending in pitch, influence TTC estimates of a vertically moving visual target and if any observed effects are consistent with a gravity-centered or gravity-unrelated pitch-speed mapping. Subjects estimated when a disc moving either upward or downward at a constant speed reached a visual landmark after the disc disappeared behind an occluder under three conditions: with an accompanying ascending pitch glide, with a descending pitch glide, or with no sound. Overall, subjects underestimated TTC with ascending pitch glides and overestimated TTC with descending pitch glides, compared to the no-sound condition. These biases in TTC were consistent in both disc motion directions. These results suggest that subjects adopted a gravity-unrelated pitch-speed mapping where ascending pitch is associated with faster speeds and descending pitch associated with slower speeds.

Crossmodal correspondences between auditory pitch and spatial elevation have been demonstrated extensively in adults. High- and low-pitched sounds tend to be mapped onto upper and lower spatial positions, respectively. We hypothesised that this crossmodal link could be influenced by the development of spatial and linguistic abilities during childhood. To explore this possibility, 70 children (9-12 years old) divided into three groups (4th, 5th and 6th grade of primary school) completed a crossmodal test to evaluate the perceptual correspondence between pure tones and spatial elevation. Additionally, we addressed possible correlations between the students’ performance in this crossmodal task and other auditory, spatial and linguistic measures. The participants’ auditory pitch performance was measured in a frequency classification test. The participants also completed three tests of the Wechsler Intelligence Scale for Children-IV (WISC-IV): (1) Vocabulary, to assess verbal intelligence, (2) Matrix reasoning, to measure visuospatial reasoning and (3) Blocks design, to analyse visuospatial/motor skills. The results revealed crossmodal effects between pitch and spatial elevation. Additionally, we found a correlation between the performance in the block design subtest with the pitch-elevation crossmodal correspondence and the auditory frequency classification test. No correlation was observed between auditory tasks with matrix and vocabulary subtests. This suggests (1) that the crossmodal correspondence between pitch and spatial elevation is already consolidated at the age of 9 and also (2) that good performance in a pitch-based auditory task is mildly associated, in childhood, with good performance in visuospatial/motor tasks.

Previous studies demonstrated that redundant target stimuli can enhance performance due to multisensory interplay and interactively facilitate performance enhancements due to temporal expectations (TE; faster and accurate reactions to temporally expected targets). Here we tested whether other types of multisensory interactions – i.e. interactions evoked by temporally flanking irrelevant stimuli – can result in similar performance patterns and boost not only unisensory target perception (multi-vs. unisensory sequences) but also unisensory temporal expectations (expected vs. unexpected). To test our hypothesis, we presented sequences of 12 stimuli (10 Hz) which either consisted of auditory (A), visual (V) or alternating auditory-visual stimuli (e.g. A-V-A-V-…) with either auditory (AV(A)) or visual (AV(V)) targets. Participants had to discriminate target frequency which was unpredictable by temporal regularities (expected vs. unexpected target positions) and by stimulation sequence (A, V, AV(A), AV(V)). Moreover, we ran two experiments in which we presented redundant multisensory targets and manipulated the speed of the stimulation sequence (10 vs. 15 Hz stimulus trains) to control whether the results of Experiment 1 depended on sequence speed. Performance for unisensory targets was affected by temporally flanking distractors, with multisensory interactions selectively improving unisensory visual target perception. Yet, only redundant multisensory targets reliably affected TEs. Together, these results indicate that cross-modal facilitation of unisensory target perception in fast stimulus streams is uni-directional, but also differs from multisensory interactions evoked by redundant targets; more specifically, it appears to be context-dependent (task, design etc.) whether unisensory stimulation (unlike redundant target stimulation) allows for the generation of temporal expectations.

There is a cross-modal mapping between auditory pitch and many visual properties, but the relationship between auditory pitch and motion speed is unexplored. In this article, the ball and baffle are used as the research objects, and an object collision experiment is used to explore the perceptual influence of auditory pitch on motion speed. Since cross-modal mapping can influence perceptual experience, this article also explores the influence of auditory pitch on action measures. In Experiment 1, 12 participants attempted to release a baffle to block a falling ball on the basis of speed judgment, and after each trial, they were asked to rate the speed of the ball. The speed score and baffle release time were recorded and used for analysis of variance. Since making explicit judgments about speed can alter the processing of visual paths, another group of participants in Experiment 2 completed the experiment without making explicit judgments about speed. Our results show that there is a cross-modal mapping between auditory pitch and motion speed, and high or low tones cause perception shift to faster or slower speeds.

Cross-modal correspondences refer to associations between apparently unrelated stimulus features in different senses. For example, high and low auditory pitches are associated with high and low visual elevations, respectively. Here we examined how this crossmodal correspondence between visual elevation and auditory pitch relates to auditory elevation. We used audiovisual combinations of high- or low-frequency bursts of white noise and a visual stimulus comprising a white circle. Auditory and visual stimuli could each occur at high or low elevations. These multisensory stimuli could be congruent or incongruent for three correspondence types: cross-modal featural (auditory pitch/visual elevation), within-modal featural (auditory pitch/auditory elevation) and cross-modal spatial (auditory and visual elevation). Participants performed a 2AFC speeded classification (high or low) task while attending to auditory pitch, auditory elevation, or visual elevation. We tested for modulatory interactions between the three correspondence types. Modulatory interactions were absent when discriminating visual elevation. However, the within-modal featural correspondence affected the cross-modal featural correspondence during discrimination of auditory elevation and pitch, while the reverse modulation was observed only during discrimination of auditory pitch. The cross-modal spatial correspondence modulated the other two correspondences only when auditory elevation was being attended, was modulated by the cross-modal featural correspondence only during attention to auditory pitch, and was modulated by the within-modal featural correspondence while performing discrimination of either auditory elevation or pitch. We conclude that the cross-modal correspondence between auditory pitch and visual elevation interacts strongly with auditory elevation.

The present study tested the hypothesis that even the simplest cognitive tasks require the storage of information in working memory (WM), distorting any information that was previously stored in WM. Experiment 1 tested this hypothesis by requiring observers to perform a simple letter discrimination task while they were holding a single orientation in WM. We predicted that performing the task on the interposed letter stimulus would cause the orientation memory to become less precise and more categorical compared to when the letter was absent or when it was present but could be ignored. This prediction was confirmed. Experiment 2 tested the modality specificity of this effect by replacing the visual letter discrimination task with an auditory pitch discrimination task. Unlike the interposed visual stimulus, the interposed auditory stimulus produced little or no disruption of WM, consistent with the use of modality-specific representations. Thus, performing a simple visual discrimination task, but not a simple auditory discrimination task, distorts information about a single feature being maintained in visual WM. We suggest that the interposed task eliminates information stored within the focus of attention, leaving behind a WM representation outside the focus of attention that is relatively imprecise and categorical.

The crossmodal correspondence between auditory pitch and visuospatial elevation (in which high- and low-pitched tones are associated with high and low spatial elevation respectively) has been proposed as the basis for Western musical notation. One implication of this is that music perception engages visuospatial processes and may not be exclusively auditory. Here, we investigated how music perception is influenced by concurrent visual stimuli. Participants listened to unfamiliar five-note musical phrases with four kinds of pitch contour (rising, falling, rising–falling, or falling–rising), accompanied by incidental visual contours that were either congruent (e.g., auditory rising/visual rising) or incongruent (e.g., auditory rising/visual falling) and judged whether the final note of the musical phrase was higher or lower in pitch than the first. Response times for the auditory judgment were significantly slower for incongruent compared to congruent trials, i.e., there was a congruency effect, even though the visual contours were incidental to the auditory task. These results suggest that music perception, although generally regarded as an auditory experience, may actually be multisensory in nature.

Adult humans spontaneously associate visual features, such as size and direction of movement, with phonetic properties like vowel quality and auditory pitch. A number of recent studies have claimed that looking time in preverbal infants reveals the same associations, which would indicate that some cross-modal correspondences are the result of perceptual biases. Here we tested 30 infants of age 7-13 months, who were exposed to pairs of audiovisual stimuli presented first sequentially and then side by side. The stimuli consisted of a visual object (computer-animated ball or filmed human hand) moving sinusoidally, vertically, or in a U-shape and accompanied by a sliding voice-like tone. Sequential presentation revealed no preference for either audiovisual synchrony or synesthetic congruency, while side-by-side presentation revealed a small preference for incongruent stimuli. The effect of congruency was similar for the animated ball and filmed human hand. These findings extend the results of previous research on pitch-motion synesthesia in preverbal infants, which used animations and sliding whistles, to more ecologically relevant stimuli such as voice and gestures. If infants and adults share the same preferences for non-arbitrary mappings between manual gestures and intonation, this could indicate that cross-modal correspondences facilitate language acquisition. On the other hand, a critical survey of the field revealed that previous studies of audiovisual cross-modal correspondences in infants suffer from replication failures due to poor robustness of the reported effect with respect to experimental stimuli and testing procedure. We therefore argue that the research on cross-modal correspondences in infants would profit from using alternative testing methods in addition to preferential looking and call for replication of previously reported congruency effects. (Less)

A combination of signals across perceptual modalities can facilitate sensory perception. Congruent visual information is generally considered to complement insufficient speech information for hearing-impaired and normal listeners in noisy conditions. In order to develop effective audiovisual strategies for speech perception, it is important to understand the mechanism accounting for the audiovisual integration ability. The present work investigated the effect of simultaneous visual stimulation on detecting auditory frequency changes with random stimulus sequences. Event-related potential responses with normal-hearing listeners showed that simultaneous visual and auditory stimuli elicited larger N1/P2 amplitude and shorter P2 peak latency than auditory-only stimuli did. The present study provided neurophysiological evidence that the cross-modal correspondence between the size of visual stimulus and the level of auditory pitch favors the detection of auditory frequency changes presented in random stimulus sequences.

This study examined whether auditory pitch and loudness affect the perception of object’s weight. Two series experiments showed that the object with High-Pitch sound was perceived as being lighter than the object with Low-Pitch sound and that the perceived weight was not affected by loudness. Because auditory pitch has a relationship to the weight of an object while loudness has a relationship to the distance of a placed object, the perceived weight was affected by auditory pitch not loudness. Given these results, perhaps sound effects may make it easier to carry heavy luggage?

Research has shown participants associate high pitch tones with small objects, and low pitch tones with large objects. Yet it remains unclear when these associations emerge in neural signals, and whether or not they are likely the result of predictive coding mechanisms being influenced by multisensory priors. Here we investigated these questions using a modified version of the implicit association task, 128-channel human EEG, and two approaches to single-trial analysis (linear discriminant and mutual information). During two interlaced discrimination tasks (auditory high/low tone and visual small/large circle), one stimulus was presented per trial and the auditory stimulus-response assignment was manipulated. On congruent trials preferred pairings (high tone, small circle) were assigned to the same response key, and on incongruent trials non-preferred pairings were (low tone, small circle). The results showed participants (male and female) responded faster during auditory congruent than incongruent trials. The EEG results showed that acoustic pitch and visual size were represented early in the trial (~100 ms and ~220 ms), over temporal and frontal regions. Neural signals were also modulated by congruency early in the trial for auditory (

Although research on synaesthesia has blossomed in the last decades, few things are known about inherited sound-colour synaesthesia and processes underlying synaesthetic perception. In this study we investigated how the auditory pitch, the frequency range, the complexity, as well as the structure of the sound affected the synaesthetic visual attributes of chroma, colour brightness, colour plurality and density. A distinct mapping was observed between sound features and visual characteristics, which suggests that visual stimulation requires sensory auditory processing. We propose that congenital synaesthetic perception involves two stages of processing: initially the sensory auditory information is processed by the primary auditory cortex, and then it is distributed to the secondary and extrastriate visual brain areas, generating visual stimulation. This process does not necessarily involve the participation of brain areas responsible for associative thinking.

Linear trend (slope) is important information conveyed by graphs. We investigated how sounds influenced slope detection in a visual search paradigm. Four bar graphs or scatter plots were presented on each trial. Participants looked for a positive-slope or a negative-slope target (in blocked trials), and responded to targets in a go or no-go fashion. For example, in a positive-slope-target block, the target graph displayed a positive slope while other graphs displayed negative slopes (a go trial), or all graphs displayed negative slopes (a no-go trial). When an ascending or descending sound was presented concurrently, ascending sounds slowed detection of negative-slope targets whereas descending sounds slowed detection of positive-slope targets. The sounds had no effect when they immediately preceded the visual search displays, suggesting that the results were due to crossmodal interaction rather than priming. The sounds also had no effect when targets were words describing slopes, such as “positive,” “negative,” “increasing,” or “decreasing,” suggesting that the results were unlikely due to semantic-level interactions. Manipulations of spatiotemporal similarity between sounds and graphs had little effect. These results suggest that ascending and descending sounds influence visual search for slope based on a general association between the direction of auditory pitch-change and visual linear trend.

Experiments using diverse paradigms, including speeded discrimination, indicate that pitch and visually-perceived size interact perceptually, and that higher pitch is congruent with smaller size. While nearly all of these studies used static stimuli, here we examine the interaction of dynamic pitch and dynamic size, using Garner’s speeded discrimination paradigm. Experiment 1 examined the interaction of continuous rise/fall in pitch and increase/decrease in object size. Experiment 2 examined the interaction of static pitch and size (steady high/low pitches and large/small visual objects), using an identical procedure. Results indicate that static and dynamic auditory and visual stimuli interact in opposite ways. While for static stimuli (Experiment 2), higher pitch is congruent with smaller size (as suggested by earlier work), for dynamic stimuli (Experiment 1), ascending pitch is congruent with growing size, and descending pitch with shrinking size. In addition, while static stimuli (Experiment 2) exhibit both congruence and Garner effects, dynamic stimuli (Experiment 1) present congruence effects without Garner interference, a pattern that is not consistent with prevalent interpretations of Garner’s paradigm. Our interpretation of these results focuses on effects of within-trial changes on processing in dynamic tasks and on the association of changes in apparent size with implied changes in distance. Results suggest that static and dynamic stimuli can differ substantially in their cross-modal mappings, and may rely on different processing mechanisms.

Crossmodal correspondences have often been demonstrated using congruency effects between pairs of stimuli in different sensory modalities that vary along separate dimensions. To date, however, it is still unclear the extent to which these correspondences are relative versus absolute in nature: that is, whether they result from pre-defined values that rigidly link the two dimensions or rather result from flexible values related to the previous occurrence of the crossmodal stimuli. Here, we investigated this issue in a speeded classification task featuring the correspondence between auditory pitch and visual size (e.g., congruent correspondence between high pitch/small disc and low pitch/large disc). Participants classified the size of the visual stimuli (large vs. small) while hearing concurrent high- or low-pitched task-irrelevant sounds. On some trials, visual stimuli were paired instead with “intermediate” pitch, that could be interpreted differently according to the auditory stimulus on the preceding trial (i.e., as “lower” following the presentation of a high pitch tone, but as “higher” following the presentation of a low pitch tone). Performance on sequence-congruent trials (e.g., when a small disc paired with the intermediate-pitched tone was preceded by a low pitch tone) was compared to sequence-incongruent trials (e.g., when a small disc paired with the intermediate-pitch tone was by a high-pitched tone). The results revealed faster classification responses on sequence-congruent than on sequence-incongruent trials. This demonstrates that the effect of the pitch/size correspondence is relative in nature, and subjected to trial-by-trial interpretation of the stimulus pair.

Amodal (redundant) and arbitrary cross-sensory feature associations involve the context-insensitive mapping of absolute feature values across sensory domains. Cross-sensory associations of a different kind, known as correspondences, involve the context-sensitive mapping of relative feature values. Are such correspondences in place at birth (like amodal associations), or are they learned from subsequently experiencing relevant feature co-occurrences in the world (like arbitrary associations)? To decide between these two possibilities, human newborns (median age = 44 hr) watched animations in which two balls alternately rose and fell together in space. The pitch of an accompanying sound rose and fell either congruently with this visual change (pitch rising and falling as the balls moved up and down), or incongruently (pitch rising and falling as the balls moved down and up). Newborns' looking behavior was sensitive to this congruence, providing the strongest indication to date that cross-sensory correspondences can be in place at birth.

High-pitched sounds generate larger neural responses than low-pitched sounds. We investigated whether this neural difference has implications, at cognitive level, for the “vertical” representation of pitch. Participants performed a speeded detection of visual targets that could appear at one of four different spatial positions. Rising or falling frequency sweeps were randomly presented before the visual target. Faster reaction times to visual targets appearing above (but not below) a central fixation point were observed after the presentation of rising frequencies. No significant effects were found for falling frequency sweeps and visual targets presented below fixation point. These results suggest that the difference in the level of arousal between rising and falling frequencies influences their capacity for generating spatial representations. The fact that no difference was found, in terms of crossmodal effects, between the two upper positions may indicate that this “spatial representation of pitch” is not specific for any particular spatial location but rather has a widespread influence over stimuli appearing in the upper visual field. The present findings are relevant for the study of music performance, the design of musical instruments, and research in areas where visual and auditory stimuli with certain complexity are combined (music in advertisements, movies, etc.).

Studies investigating cross-modal correspondences between auditory pitch and visual shapes have shown children and adults consistently match high pitch to pointy shapes and low pitch to curvy shapes, yet no studies have investigated linguistic-uses of pitch. In the present study, we used a bouba/kiki style task to investigate the sound/shape mappings for Tones of Mandarin Chinese, for three groups of participants with different language backgrounds. We recorded the vowels [i] and [u] articulated in each of the four tones of Mandarin Chinese. In Study 1 a single auditory stimulus was presented with two images (one curvy, one spiky). In Study 2 a single image was presented with two auditory stimuli differing only in tone. Participants were asked to select the best match in an online ‘Quiz.’ Across both studies, we replicated the previously observed ‘u-curvy, i-pointy’ sound/shape cross-modal correspondence in all groups. However, Tones were mapped differently by people with different language backgrounds: speakers of Mandarin Chinese classified as Chinese-dominant systematically matched Tone 1 (high, steady) to the curvy shape and Tone 4 (falling) to the pointy shape, while English speakers with no knowledge of Chinese preferred to match Tone 1 (high, steady) to the pointy shape and Tone 3 (low, dipping) to the curvy shape. These effects were observed most clearly in Study 2 where tone-pairs were contrasted explicitly. These findings are in line with the dominant patterns of linguistic pitch perception for speakers of these languages (pitch-change, and pitch height, respectively). Chinese English balanced bilinguals showed a bivalent pattern, swapping between the Chinese pitch-change pattern and the English pitch-height pattern depending on the task. These findings show for that the supposedly universal pattern of mapping linguistic sounds to shape is modulated by the sensory properties of a speaker’s language system, and that people with high functioning in more than one language can dynamically shift between patterns. Published version

Crossmodal correspondences are intuitively held relationships between non-redundant features of a stimulus, such as auditory pitch and visual illumination. While a number of correspondences have been identified in humans to date (e.g. high pitch is intuitively felt to be luminant, angular and elevated in space), their evolutionary and developmental origins remain unclear. Here, we investigated the existence of audio–visual crossmodal correspondences in domestic dogs, and specifically, the known human correspondence in which high auditory pitch is associated with elevated spatial position. In an audio–visual attention task, we found that dogs engaged more with audio–visual stimuli that were congruent with human intuitions (high auditory pitch paired with a spatially elevated visual stimulus) compared to incongruent (low pitch paired with elevated visual stimulus). This result suggests that crossmodal correspondences are not a uniquely human or primate phenomenon and they cannot easily be dismissed as merely lexical conventions (i.e. matching ‘high’ pitch with ‘high’ elevation).

Our brain constantly integrates signals across different senses. Auditory–visual synaesthesia is an unusual form of cross-modal integration in which sounds evoke involuntary visual experiences. Previous research primarily focuses on synaesthetic colour, but little is known about non-colour synaesthetic visual features. Here we studied a group of synaesthetes for whom sounds elicit consistent visual experiences of coloured ‘geometric objects’ located at specific spatial location. Changes in auditory pitch alter the brightness, size, and spatial height of synaesthetic experiences in a systematic manner resembling the cross-modal correspondences of non-synaesthetes, implying synaesthesia may recruit cognitive/neural mechanisms for ‘normal’ cross-modal processes. To objectively assess the impact of synaesthetic objects on behaviour, we devised a multi-feature cross-modal synaesthetic congruency paradigm and asked participants to perform speeded colour or shape discrimination. We found irrelevant sounds influenced performance, as quantified by congruency effects, demonstrating that synaesthetes were not able to suppress their synaesthetic experiences even when these were irrelevant for the task. Furthermore, we found some evidence for task-specific effects consistent with feature-based attention acting on the constituent features of synaesthetic objects: synaesthetic colours appeared to have a stronger impact on performance than synaesthetic shapes when synaesthetes attended to colour, and vice versa when they attended to shape. We provide the first objective evidence that visual synaesthetic experience can involve multiple features forming object-like percepts and suggest that each feature can be selected by attention despite it being internally generated. These findings suggest theories of the brain mechanisms of synaesthesia need to incorporate a broader neural network underpinning multiple visual features, perceptual knowledge, and feature integration, rather than solely focussing on colour-sensitive areas.

Children as well as adults make certain links between the visual and the acoustic domain. For example, higher pitched sounds are typically associated with visual stimuli that are brighter, smaller, and higher in space than those associated with sounds of lower pitch. The present study explored the existence of such links in infancy. More specifically, we investigated the relation between visuospatial height and auditory pitch in 7- to 12-month-olds, varying systematically the format of stimulus presentation. Presenting dynamic visual and acoustic stimuli led to positive findings relying on anticipatory looking measures: While listening to an ascending tone, infants looked longer where they had previously seen an upwards moving circle. While listening to a descending tone, they looked longer where they had previously seen a downwards moving circle. When static tones were combined with static visual stimuli in the same kind of display, no corresponding findings were obtained. Thus, the present work suggests...

Memory for the initial pitch of an auditory target that increased or decreased in auditory frequency was examined. Memory was displaced forward in the direction of pitch motion, and this is consistent with the Fröhlich effect previously observed for visual targets moving in visual physical space. The Fröhlich effect for pitch increased with faster target velocity and decreased if an auditory cue with the same pitch as the initial pitch of the target was presented before the target was presented. The Fröhlich effect was larger for descending pitch motion than for ascending pitch motion, and this is consistent with an influence of representational gravity. The data suggest that representation of auditory frequency space exhibits some of the same biases as representation of visual physical space, and implications for theories of attention in displacement and for crossmodal and multisensory representation of space are discussed.

We report two experiments designed to assess the key sensory drivers underlying people’s association of a specific auditory pitch with Belgian beer. In particular, we assessed if people would rely mostly on the differences between beers in terms of their relative alcohol strength, or on the contrast between the most salient taste attributes of the different beers. In Experiment 1, the participants rated three bitter beers (differing in alcohol content), using a narrow range of pitch choices (50–500 Hz). The results revealed that the beers were all rated around the same pitch (Mean = 232 Hz, SD = 136 Hz). In Experiment 2, a wider range of pitch choices (50–1500 Hz), along with the addition of a much sweeter beer, revealed that people mostly tend to match beers with bitter-range profiles at significantly lower pitch ranges when compared to the average pitch of a much sweeter beer. These results therefore demonstrate that clear differences in taste attributes lead to distinctly different matches in terms of pitch. Having demonstrated the robustness of the basic crossmodal matching, future research should aim to uncover the basis for such matches and better understand the perceptual effects of matching/non-matching tones on the multisensory drinking experience.

The aim of the present study is to investigate whether the crossmodal correspondence robustly documented between auditory pitch and visual elevation has analogues in the audio-tactile domain. Across 4 experiments, the compatibility effects between intuitively congruent pairs of stimuli (i.e., outward tactile movement, going from the inside of the finger toward the fingertip and increasing pitch, or inward tactile movement and decreasing pitch) and incongruent pairs stimuli (i.e., the reverse associations) were measured. Two methods were compared to assess the behavioral effects of such a correspondence: One where participants have to respond to either the auditory or tactile stimulus presented simultaneously, while ignoring the other (speeded classification task), and the other where the auditory and tactile stimuli are presented sequentially and associated to different response buttons (implicit association test). No significant compatibility effect was observed under the speeded classification task. The implicit association test revealed a significant compatibility effect. This effect was similar in the conditions where the finger was placed vertically and horizontally. However, this implicit association between pitch and tactile movements was not observed in blind participants. These results have methodological implications for the explanation and testing of crossmodal correspondences, and the origin of the widely discussed association between pitch and vertical elevation. (PsycINFO Database Record

Temporal ventriloquism is the shift in perceived timing of a visual stimulus that occurs when an auditory stimulus is presented close in time. This study investigated whether crossmodal correspondence between auditory pitch and visual elevation modulates temporal ventriloquism. Participants were presented two visual stimuli (above and below fixation) across a range of stimulus onset asynchronies and were asked to judge the order of the events. A task-irrelevant auditory click was presented shortly before the first and another shortly after the second visual stimulus. There were two pitches used (low and high) and the congruency between the auditory and visual stimuli was manipulated. The results show that incongruent pairings between pitch and elevation abolish temporal ventriloquism. In contrast, the crossmodal correspondence effect was absent when the direction of the pitch change was fixed within sessions, reducing the saliency of the pitch change. The results support previous studies suggesting that in addition to spatial and temporal factors, crossmodal correspondences can influence binding of information across the senses, although these effects are likely to be dependent on the saliency of the crossmodal mapping.

To overcome limitations in perceptual bandwidth, humans condense various features of the environment into summary statistics. Variance constitutes indices that represent diversity within categories and also the reliability of the information regarding that diversity. Studies have shown that humans can efficiently perceive variance for visual stimuli; however, to enhance perception of environments, information about the external world can be obtained from multisensory modalities and integrated. Consequently, this study investigates, through two experiments, whether the precision of variance perception improves when visual information (size) and corresponding auditory information (pitch) are integrated. In Experiment 1, we measured the correspondence between visual size and auditory pitch for each participant by using adjustment measurements. The results showed a linear relationship between size and pitch—that is, the higher the pitch, the smaller the corresponding circle. In Experiment 2, sequences of visual stimuli were presented both with and without linked auditory tones, and the precision of perceived variance in size was measured. We consequently found that synchronized presentation of audio and visual stimuli that have the same variance improves the precision of perceived variance in size when compared with visual-only presentation. This suggests that audiovisual information may be automatically integrated in variance perception.

Three Garner interference experiments are described in which baseline, filtering, and correlated performance were assessed at both a macrolevel (condition average) and microlevel (intertrial contingency), using the pair-wise combinations of auditory pitch, loudness, and location. Discrepancies between pairs of dimensions were revealed between macro- and microlevel estimates of performance and, also, between filtering costs and correlated benefits, relative to baseline. The examination of the intertrial effects associated with filtering costs suggested that effects of increased stimulus uncertainty were mandatory, whereas effects of irrelevant variation were not. The examination of the intertrial effects associated with correlated benefits suggested that the detection of stimulus repetition took precedence over that of stimulus change. Violations of standard horse race accounts of processing did not appear to stem from differences in the absolute or relative speeds of processing between dimensions but, rather, from the special role that certain dimensions (e.g., pitch) may play in certain modalities (e.g., audition). The utility of examining repetition effects is demonstrated by revealing a level of understanding regarding stimulus processing typically hidden by aggregated measures of performance.

Growing evidence shows that individuals consistently match auditory pitch with visual size. For instance, high-pitched sounds are perceptually associated with smaller visual stimuli, whereas low-pitched sounds with larger ones. The present study explores whether this crossmodal correspondence, reported so far for perceptual processing, also modulates motor planning. To address this issue, we carried out a series of kinematic experiments to verify whether actions implying size processing are affected by auditory pitch. Experiment 1 showed that grasping movements toward small/large objects were initiated faster in response to high/low pitches, respectively, thus extending previous findings in the literature to more complex motor behavior. Importantly, auditory pitch influenced the relative scaling of the hand preshaping, with high pitches associated with smaller grip aperture compared with low pitches. Notably, no effect of auditory pitch was found in case of pointing movements (no grasp implied, Experiment 2), as well as when auditory pitch was irrelevant to the programming of the grip aperture, that is, in case of grasping an object of uniform size (Experiment 3). Finally, auditory pitch influenced also symbolic manual gestures expressing "small" and "large" concepts (Experiment 4). In sum, our results are novel in revealing the impact of auditory pitch on motor planning when size processing is required, and shed light on the role of auditory information in driving actions.