Search

Communication through auditory cues often evokes associations to other sensory modalities. In film music, for instance, a descending pitch contour commonly resembles a falling motion. Such crossmodal associations to physical actions or shapes are here termed sound gestures and can naturally occur in environmental sounds. Little is known about how reliably listeners perceive gestures in such environmental contexts and how salient the gesture-relevant auditory feature needs to be. This article reports on an exploratory study concerning the identification of sound gestures by crossmodal matching using analogous visualizations. The study considered gesture-related factors, such as auditory salience and contour complexity, and explored whether a concurrent evaluation of features related to the environmental sound source or cause would affect gesture identification. Twenty untrained listeners evaluated sound gestures occurring in environmental sounds, e.g., pitch contour when switching a vacuum cleaner on and off, loudness contour of a ball dropping. Participants evaluated 28 environmental sounds in three variants (original, isolated gesture, hybrid) and had to identify the sound gesture among four visualized options while also inferring the underlying environmental source or cause through verbal description and rating their confidence in identifying the source/cause. Based on features describing the macro contour of gestures, participants correctly identified 81-83% of all gestures. Manipulated sounds that emphasized gesture salience yielded only slight improvements of identification accuracy compared to original environmental sounds. Participants were more confident in identifying the source/cause in sounds containing pitch gestures than those containing loudness gestures, while lexical and semantic diversity in describing underlying materials (source) and actions (cause) varied considerably. For both groups, however, measures for gesture identification and the evaluation of underlying materials and actions correlated only weakly, suggesting task independence. Overall, findings suggest that untrained listeners perceive sound gestures in environmental sounds and can reliably use them to form crossmodal associations, while also evaluating properties related to the sound source and cause. For one, the perception of environmental sounds may evoke crossmodal links, while the reliable identification of sound gestures highlights their utility to crossmodal control or search interfaces.

Without conscious thought, listeners link events in the world to sounds they hear. We study one surprising example: Adults can judge the temperature of water simply from hearing it being poured. We test the development of the ability to hear water temperature, with the goal of informing developmental theories regarding the origins and cognitive bases of nuanced sound source judgments. We first confirmed that adults accurately distinguished the sounds of hot and cold water (pre-registered Experiments. 1, 2; total N = 384), even though many were unaware or uncertain of this ability. By contrast, children showed protracted development of this skill over the course of middle childhood (Experiments. 2, 3; total N = 178). In spite of accurately identifying other sounds and hot/cold images, older children (7-11 years) but not younger children (3-6 years) reliably distinguished the sounds of hot and cold water. Accuracy increased with age; 11-year old's performance was similar to adults. Adults also showed individual differences in accuracy that were predicted by their amount of prior relevant experience (Experiment 1). Experience may similarly play a role in children's performance; differences in auditory sensitivity and multimodal integration may also contribute to young children's failures. The ability to hear water temperature develops slowly over childhood, such that nuanced auditory information that is easily and quickly accessible to adults is not available to guide young children's behavior. HIGHLIGHTS: Adults can make nuanced judgments from sound, including accurately judging the temperature of water from the sound of it being poured. Children showed protracted development of this skill over the course of middle childhood, such that 7-11-year-olds reliably succeeded while 3-6-year-olds performed at chance. Developmental changes may be due to experience (adults with greater relevant experience showed higher accuracy) and the development of multimodal integration and auditory sensitivity. Young children may not detect subtle auditory information that adults easily perceive.

Crossmodal correspondences refer to systematic associations between stimulus attributes encountered in different sensory modalities. These correspondences can be probed in the speeded classification task where they tend to produce congruency effects. This study aimed to replicate and extend previous work conducted by Marks (1987, Experiment 3, Journal of Experimental Psychology: Human Perception and Performance, Vol. 13, No. 3, 384–394) which demonstrated a crossmodal correspondence between auditory and visual intensity attributes. Experiment 1 successfully replicates Marks' original finding that performance in a brightness classification task is affected by whether the loudness of a concurrently presented auditory distractor matches the brightness of the visual target. Furthermore, in line with the original study, we found that this effect was absent in a lightness classification task. In Experiment 2, we demonstrate that loudness–brightness correspondence is robust even when the exact stimulus input changes. This finding suggests that there is a context-dependent mapping between loudness and brightness levels, rather than an absolute mapping between any particular intensity levels. Finally, exploratory analysis using the diffusion model for conflict tasks indicated that evidence from the task-irrelevant modality generates a burst of weak, short-lived automatic activation that can bias decision-making in difficult tasks, but not in easy tasks. Our results provide further evidence for the existence of a flexible crossmodal correspondence between brightness and loudness, which might be helpful in determining one's distance to a stimulus source during the early stages of multisensory integration. Public Significance Statement: Cognitive psychology has uncovered fascinating connections between our senses, suggesting they can work together in surprising ways. We present data which confirm and extend an earlier finding that the loudness of sounds and the brightness of lights are linked in our minds. This link might improve our ability to accurately judge how far away something is. [ABSTRACT FROM AUTHOR]

Humans spontaneously and consistently map information coming from different sensory modalities. Surprisingly, the phylogenetic origin of such cross‐modal correspondences has been under‐investigated. A notable exception is the study of Ludwig et al. (Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees [Pan troglodytes] and humans. Proceedings of the National Academy of Sciences, 108(51), 20661–20665) which reports that both humans and chimpanzees spontaneously map high‐pitched sounds with bright objects and low‐pitched sounds with dark objects. Our pre‐registered study aimed to directly replicate this research on both humans and baboons (Papio papio), an old world monkey which is more phylogenetically distant from humans than chimpanzees. Following Ludwig et al. participants were presented with a visual classification task where they had to sort black and white square (low and high luminance), while background sounds (low or high‐pitched tones) were playing. Whereas we replicated the finding that humans' performance on the visual task was affected by congruency between sound and luminance of the target, we did not find any of those effects on baboons' performance. These results question the presence of a shared cross‐modal pitch‐luminance mapping in other nonhuman primates. Research highlights: Humans and chimpanzees spontaneously associate high‐pitched sounds with bright objects and low‐pitched sounds with dark objects.This effect was not seen in baboons (Papio papio).This result questions the presence of a shared cross‐modal pitch‐luminance mapping between humans and all nonhuman primates. [ABSTRACT FROM AUTHOR]

Recent advances in the field of human cognition development have led to increased interest in the study of crossmodal correspondences, which explore compatibility effects between different sensory modalities. While various crossmodal correspondence domains have received extensive attention, there remains a knowledge gap concerning the compatibility between visual and tactile sensory stimuli. This research aims to extend our understanding of how visual and tactile stimuli interrelate and influence each other. In our conducted experiment, participants were tasked with matching specific colors to vibrotactile stimuli and vice versa. Results reveal a robust correlation between high-frequency vibrotactile stimuli 30.00-43.10 Hz and high saturation values 0.6-0.72 within a hue range of 345-30°. Conversely, weak-frequency vibrotactile stimuli 0.00-15.00 Hz exhibited a correlation with low saturation values 0.15-0.36 within a hue range of 100-226°. These findings suggest that crossmodal correspondences are triggered by both tactile and visual stimuli. [ABSTRACT FROM AUTHOR]

Iconic words and signs are characterized by a perceived resemblance between aspects of their form and aspects of their meaning. For example, in English, iconic words include peep and crash, which mimic the sounds they denote, and wiggle and zigzag, which mimic motion. As a semiotic property of words and signs, iconicity has been demonstrated to play a role in word learning, language processing, and language evolution. This paper presents the results of a large-scale norming study for more than 14,000 English words conducted with over 1400 American English speakers. We demonstrate the utility of these ratings by replicating a number of existing findings showing that iconicity ratings are related to age of acquisition, sensory modality, semantic neighborhood density, structural markedness, and playfulness. We discuss possible use cases and limitations of the rating dataset, which is made publicly available. [ABSTRACT FROM AUTHOR]

Most previous audio-visual crossmodal correspondence studies focused on static visual or auditory stimuli. Moreover, some researchers have found that music training can effectively improve the interaction between auditory and visual channels by enhancing neural plasticity. This study focused on whether crossmodal correspondence occurs when people face dynamic visual and auditory stimuli, and whether it is affected by musical training. Participants were asked to judge different changes in pitch (rise and drop) by showing them different patterns of visual circle motion (enlarged, reduced, and unchanged). The results revealed that the audio-visual congruent combinations (the pitch rise when the circle was enlarged and the pitch drop when the circle was reduced) significantly shortened participants' response times, whereas the audio-visual neutral combinations (the unchanged shape) had the highest accuracy. Participants with musical training were faster than participants without musical training to judge pitch changes in conditions where the association between pitch and shape size was incongruent. These findings provide empirical evidence for dynamic audio-visual crossmodal correspondence and shed light on the bright prospect of using congruent audio-visual stimuli in animation. [ABSTRACT FROM AUTHOR]

Sound-based trajectories or sound gestures draw links to spatiokinetic processes. For instance, a gliding, decreasing pitch conveys an analogous downward motion or fall. Whereas the gesture's pitch orientation and range convey its meaning and magnitude, respectively, the way in which pitch changes over time can be conceived of as gesture shape, which to date has rarely been studied in isolation. This article reports on an experiment that studied the perception of shape in uni-directional pitch, loudness, and tempo gestures, each assessed for four physical scalings. Gestures could increase or decrease over time and comprised different frequency and sound level ranges, durations, and different scaling contexts. Using a crossmodal-matching task, participants could reliably distinguish between pitch and loudness gestures and relate them to analogous visual line segments. Scalings based on equivalent-rectangular bandwidth (ERB) rate for pitch and raw signal amplitude for loudness were matched closest to a straight line, whereas other scalings led to perceptions of exponential or logarithmic curvatures. The investigated tempo gestures, by contrast, did not yield reliable differences. The reliable, robust perception of gesture shape for pitch and loudness has implications on various sound-design applications, especially those cases that rely on crossmodal mappings, e.g., visual analysis or control interfaces like audio waveforms or spectrograms. Given its perceptual relevance, auditory shape appears to be an integral part of sound gestures, while illustrating how crossmodal correspondences can underpin auditory perception. [ABSTRACT FROM AUTHOR]

Introduction: Adults possess a natural inclination to associate sensory cues derived from distinct modalities, such as the pairing of sweet with pink. However, studies exploring crossmodal correspondences in children, particularly in the sensory pairing of visual features and tastes, are scant, leaving unanswered questions regarding the developmental trajectory of crossmodal correspondences. The present study investigates whether Japanese preschool children demonstrate specific biases in shape-color, shape-taste, and color-taste associations. Methods: In a series of in-person experiments, 92 children between 3 to 6 years of age completed matching tasks utilizing paper stimuli. Results: Children exhibit crossmodal correspondences in shape-color (circle-red and asymmetrical star-yellow), shape-taste (triangle-salty and circle-sweet), and color-taste (yellow-sour, black-bitter, and pink-sweet) associations. Moreover, children's choices are not influenced by their individual preferences. Discussion: The crossmodal correspondences observed in this study have been observed in previous research on adults from the same (Japanese) culture, although adults showed more crossmodal correspondences than the children in this study (e.g., pink-circle, triangle-sour, and green-bitter). Thus, while some crossmodal correspondences emerge during childhood, others may require additional time to develop, thereby highlighting the importance of understanding the cognitive mechanisms underlying crossmodal correspondences from an ontogenic perspective. [ABSTRACT FROM AUTHOR]

Introduction: This research investigates consumer acceptance of alternative methods for communicating information about wine, focusing on the alignment between sensory attributes and consumer expectations. Methods: A survey was administered to wine enthusiasts to assess their attitudes toward crossmodal communication. Results: The findings reveal significant associations between consumer behaviors and acceptance of alternative communication methods, highlighting the emerging field of crossmodal correspondences. Discussion: These results suggest that leveraging crossmodal correspondences can enhance the match between a product's sensory qualities and consumer expectations, potentially reducing wine wastage resulting from unmet consumer preferences. These findings have implications for improving communication strategies in the wine industry and enhancing consumer experiences. [ABSTRACT FROM AUTHOR]

“Crossmodal correspondences” are the consistent mappings between perceptual dimensions or stimuli from different sensory domains, which have been widely observed in the general population and investigated by experimental psychologists in recent years. At the same time, the emerging field of human movement augmentation (i.e., the enhancement of an individual’s motor abilities by means of artificial devices) has been struggling with the question of how to relay supplementary information concerning the state of the artificial device and its interaction with the environment to the user, which may help the latter to control the device more effectively. To date, this challenge has not been explicitly addressed by capitalizing on our emerging knowledge concerning crossmodal correspondences, despite these being tightly related to multisensory integration. In this perspective paper, we introduce some of the latest research findings on the crossmodal correspondences and their potential role in human augmentation. We then consider three ways in which the former might impact the latter, and the feasibility of this process. First, crossmodal correspondences, given the documented effect on attentional processing, might facilitate the integration of device status information (e.g., concerning position) coming from different sensory modalities (e.g., haptic and visual), thus increasing their usefulness for motor control and embodiment. Second, by capitalizing on their widespread and seemingly spontaneous nature, crossmodal correspondences might be exploited to reduce the cognitive burden caused by additional sensory inputs and the time required for the human brain to adapt the representation of the body to the presence of the artificial device. Third, to accomplish the first two points, the benefits of crossmodal correspondences should be maintained even after sensory substitution, a strategy commonly used when implementing supplementary feedback. [ABSTRACT FROM AUTHOR]

IntroductionThis research investigates consumer acceptance of alternative methods for communicating information about wine, focusing on the alignment between sensory attributes and consumer expectations.MethodsA survey was administered to wine enthusiasts to assess their attitudes toward crossmodal communication.ResultsThe findings reveal significant associations between consumer behaviors and acceptance of alternative communication methods, highlighting the emerging field of crossmodal correspondences.DiscussionThese results suggest that leveraging crossmodal correspondences can enhance the match between a product’s sensory qualities and consumer expectations, potentially reducing wine wastage resulting from unmet consumer preferences. These findings have implications for improving communication strategies in the wine industry and enhancing consumer experiences.

“Crossmodal correspondences” are the consistent mappings between perceptual dimensions or stimuli from different sensory domains, which have been widely observed in the general population and investigated by experimental psychologists in recent years. At the same time, the emerging field of human movement augmentation (i.e., the enhancement of an individual’s motor abilities by means of artificial devices) has been struggling with the question of how to relay supplementary information concerning the state of the artificial device and its interaction with the environment to the user, which may help the latter to control the device more effectively. To date, this challenge has not been explicitly addressed by capitalizing on our emerging knowledge concerning crossmodal correspondences, despite these being tightly related to multisensory integration. In this perspective paper, we introduce some of the latest research findings on the crossmodal correspondences and their potential role in human augmentation. We then consider three ways in which the former might impact the latter, and the feasibility of this process. First, crossmodal correspondences, given the documented effect on attentional processing, might facilitate the integration of device status information (e.g., concerning position) coming from different sensory modalities (e.g., haptic and visual), thus increasing their usefulness for motor control and embodiment. Second, by capitalizing on their widespread and seemingly spontaneous nature, crossmodal correspondences might be exploited to reduce the cognitive burden caused by additional sensory inputs and the time required for the human brain to adapt the representation of the body to the presence of the artificial device. Third, to accomplish the first two points, the benefits of crossmodal correspondences should be maintained even after sensory substitution, a strategy commonly used when implementing supplementary feedback.

While there has long been public concern over the use of artificial/synthetic food colors, it should be remembered that food and drink products (e.g., red wine) have been purposefully colored for millennia. This narrative historical review highlights a number of reasons that food and drink have been colored, including to capture the shopper's visual attention through to signaling the likely taste/flavor. Over the course of the last century, there has, on occasion, also been interest in the playful, or sometimes even deliberately discombobulating, use of food coloring by modernist chefs and others. The coloring (or absence of color) of food and drink can, though, sometimes also take on more of a symbolic meaning, and, in a few cases, specific food colors may acquire a signature, or branded (i.e., semantic) association. That said, with food color being associated with so many different potential "meanings," it is an open question as to which meaning the consumer will associate with any given instance of color in food, and what role context may play in their decision. Laboratory‐based sensory science research may not necessarily successfully capture the full range of meanings that may be associated with food color in the mind of the consumer. Nevertheless, it seems likely that food color will continue to play an important role in dictating consumer behavior in the years to come, even though the visual appearance of food is increasingly being mediated via technological means, including virtual and augmented reality. [ABSTRACT FROM AUTHOR]

Crossmodal correspondences are defined as associations between crossmodal stimuli based on seemingly irrelevant stimulus features (i.e., bright shapes being associated with high-pitched sounds). There is a large body of research describing auditory crossmodal correspondences involving pitch and volume, but not so much involving auditory timbre, the character or quality of a sound. Adeli and colleagues (2014, Front. Hum. Neurosci. 8 , 352) found evidence of correspondences between timbre and visual shape. The present study aimed to replicate Adeli et al.'s findings, as well as identify novel timbre–shape correspondences. Participants were tested using two computerized tasks: an association task, which involved matching shapes to presented sounds based on best perceived fit, and a semantic task, which involved rating shapes and sounds on a number of scales. The analysis of association matches reveals nonrandom selection, with certain stimulus pairs being selected at a much higher frequency. The harsh/jagged and smooth/soft correspondences observed by Adeli et al. were found to be associated with a high level of consistency. Additionally, high matching frequency of sounds with unstudied timbre characteristics suggests the existence of novel correspondences. Finally, the ability of the semantic task to supplement existing crossmodal correspondence assessments was demonstrated. Convergent analysis of the semantic and association data demonstrates that the two datasets are significantly correlated (−0.36) meaning stimulus pairs associated with a high level of consensus were more likely to hold similar perceived meaning. The results of this study are discussed in both theoretical and applied contexts. [ABSTRACT FROM AUTHOR]

People associate information with different senses but the mechanism by which this happens is unclear. Such associations are thought to arise from innate structural associations in the brain, statistical associations in the environment, via shared affective content, or through language. A developmental perspective on crossmodal associations can help determine which explanations are more likely for specific associations. Certain associations with pitch (e.g., pitch–height) have been observed early in infancy, but others may only occur late into childhood (e.g., pitch–size). In contrast, tactile–chroma associations have been observed in children, but not adults. One modality that has received little attention developmentally is olfaction. In the present investigation, we explored crossmodal associations from sound, tactile stimuli, and odor to a range of stimuli by testing a broad range of participants. Across the three modalities, we found little evidence for crossmodal associations in young children. This suggests an account based on innate structures is unlikely. Instead, the number and strength of associations increased over the lifespan. This suggests that experience plays a crucial role in crossmodal associations from sound, touch, and smell to other senses. [ABSTRACT FROM AUTHOR]

We examine whether crossmodal correspondences (CMCs) modulate perceptual disambiguation by considering the influence of lightness/pitch congruency on the perceptual resolution of the Rubin face/vase (RFV). We randomly paired a black-and-white RFV (black faces and white vase, or vice versa) with either a high or low pitch tone and found that CMC congruency biases the dominant visual percept. The perceptual option that was CMC-congruent with the tone (white/high pitch or black/low pitch) was reported significantly more often than the perceptual option CMC-incongruent with the tone (white/low pitch or black/high pitch). However, the effect was only observed for stimuli presented for longer and not shorter durations suggesting a perceptual effect rather than a response bias, and moreover, we infer an effect on perceptual reversals rather than initial percepts. We found that the CMC congruency effect for longer-duration stimuli only occurred after prior exposure to the stimuli of several minutes, suggesting that the CMC congruency develops over time. These findings extend the observed effects of CMCs from relatively low-level feature-based effects to higher-level object-based perceptual effects (specifically, resolving ambiguity) and demonstrate that an entirely new category of crossmodal factors (CMC congruency) influence perceptual disambiguation in bistability. [ABSTRACT FROM AUTHOR]

Crossmodal correspondences between visual features (e.g., color/shape) and tastes have been extensively documented in recent years. Visual colors and shapes have been shown to consensually match to specific tastes. Meanwhile, individuals with autism spectrum disorder are reported to have atypical sensory processing and deficits in multisensory integration. However, the influence of autistic traits on the formation of such correspondences is relatively unknown. Here, we examined whether autistic traits could influence visual–taste associations using an online questionnaire survey among Japanese participants. The results showed that the participants exhibited strong color–taste, shape–taste, and shape–color associations, and the proportions of choosing the consensual color–taste/shape–color associations were significantly associated with autistic traits. The participants with higher autistic quotient scores chose fewer of the consensual color–taste/shape–color associations while there was no difference in choosing shape–taste associations. We interpreted the results as statistical learning with a reduced prior knowledge effect in participants with higher autistic quotient scores.

Crossmodal correspondences between visual features (e.g., color/shape) and tastes have been extensively documented in recent years. Visual colors and shapes have been shown to consensually match to specific tastes. Meanwhile, individuals with autism spectrum disorder are reported to have atypical sensory processing and deficits in multisensory integration. However, the influence of autistic traits on the formation of such correspondences is relatively unknown. Here, we examined whether autistic traits could influence visual–taste associations using an online questionnaire survey among Japanese participants. The results showed that the participants exhibited strong color–taste, shape–taste, and shape–color associations, and the proportions of choosing the consensual color–taste/shape–color associations were significantly associated with autistic traits. The participants with higher autistic quotient scores chose fewer of the consensual color–taste/shape–color associations while there was no difference in choosing shape–taste associations. We interpreted the results as statistical learning with a reduced prior knowledge effect in participants with higher autistic quotient scores. [ABSTRACT FROM AUTHOR]

From an early age, people exhibit strong links between certainvisual (e.g. size) and acoustic (e.g. duration) dimensions. Dopeople instinctively extend these crossmodal correspondencesto vocalization? We examine the ability of congenitally deafChinese children and young adults (age M = 12.4 years, SD =3.7 years) to generate iconic vocalizations to distinguish itemswith contrasting magnitude (e.g., big vs. small ball). Bothdeaf and hearing (M = 10.1 years, SD = 0.83 years)participants produced longer, louder vocalizations for greatermagnitude items. However, only hearing participants usedpitch—higher pitch for greater magnitude – which countersthe hypothesized, innate size “frequency code”, but fits withMandarin language and culture. Thus our results show thatthe translation of visible magnitude into the duration andintensity of vocalization transcends auditory experience,whereas the use of pitch appears more malleable to linguisticand cultural influence.

• Curvature and symmetry are associated with sweetness using taste words and tastants. • People vary substantially in their sensitivity to taste-shape correspondences. • Liking of tastes and shapes influences the association between these sensory stimuli. People generally associate curved and symmetrical shapes with sweetness, while associating angular and asymmetrical shapes with the other basic tastes (e.g., sour, bitter). However, these group-level taste-shape correspondences likely conceal important variation at an individual-level. We examined the extent to which individuals vary in their sensitivity to crossmodal correspondence between curvature and symmetry, on the one hand, and the five basic taste qualities (sweet, bitter, salty, sour, and umami), on the other. In Experiment 1, participants matched shapes (curved vs. angular, symmetrical vs. asymmetrical) and taste words. In Experiment 2, participants performed a similar task, though this time using actual tastants. Given that people differ in their hedonic experience of such shapes and tastes, we also measured participants' liking for each taste and shape separately. The results replicate the general crossmodal correspondences between curved-sweet and symmetrical-sweet stimuli. Furthermore, participants tended to match sour and bitter tastes with angular and asymmetrical stimuli. However, these group-level taste-shape correspondences coexist alongside substantial variation at the level of the individual. While some participants consistently matched specific tastes with curved and symmetrical stimuli, others consistently matched these tastes with angular and asymmetrical stimuli, or else did not show these taste-shape correspondences. Liking for curved and symmetrical stimuli was higher than for angular and asymmetrical stimuli. However, participants also differed considerably in the extent to which these visual features affected their liking. Overall, our findings highlight the substantial individual differences that are associated with the degree to which people associate and like shapes and tastes. [ABSTRACT FROM AUTHOR]

The relationship between music and dance can sometimes be a 'match', a remarkable fit between the audible manifestation that music is and the visual or kinaesthetic manifestation that dance is. A match between two things seems to require a common measure with respect to which the match obtains. What can this be for two so different phenomena as music and dance? I argue that the most promising answer is: movement. This answer will not be satisfactory unless the movement of music and the movement of dance are the same on some level. I suggest that they are: there are qualities of movement that guide both dancers and musicians when producing dance and music as perceptible phenomena. But the match in qualities is elusive, since the qualities are instantiated in different ways in music and dance, respectively. [ABSTRACT FROM AUTHOR]

Sensory studies emerged as a significant influence upon Human Computer Interaction and traditional multimedia. Mulsemedia is an area that extends multimedia addressing issues of multisensorial response through the combination of at least three media, typically a non-traditional media with traditional audio-visual content. In this paper, we explore the concepts of Quality of Experience and crossmodal correspondences through a case study of different types of mulsemedia setups. The content is designed following principles of crossmodal correspondence between different sensory dimensions and delivered through olfactory, auditory and vibrotactile displays. The Quality of Experience is evaluated through both subjective (questionnaire) and objective means (eye gaze and heart rate). Results show that the auditory experience has an influence on the olfactory sensorial responses and lessens the perception of lingering odor. Heat maps of the eye gazes suggest that the crossmodality between olfactory and visual content leads to an increased visual attention on the factors of the employed crossmodal correspondence (e.g., color, brightness, shape). [ABSTRACT FROM AUTHOR]

A great deal of research effort has been put in exploring crossmodal correspondences in the field of cognitive science which refer to the systematic associations frequently made between different sensory modalities (e.g. high pitch is matched with angular shapes). However, the possibilities cross-modality opens in the digital world have been relatively unexplored. Therefore, we consider that studying the plasticity and the effects of crossmodal correspondences in a mulsemedia setup can bring novel insights about improving the human-computer dialogue and experience. Mulsemedia refers to the combination of three or more senses to create immersive experiences. In our experiments, users were shown six video clips associated with certain visual features based on color, brightness, and shape. We examined if the pairing with crossmodal matching sound and the corresponding auto-generated haptic effect, and smell would lead to an enhanced user QoE. For this, we used an eye-tracking device as well as a heart rate monitor wristband to capture users' eye gaze and heart rate whilst they were experiencing mulsemedia. After each video clip, we asked the users to complete an on-screen questionnaire with a set of questions related to smell, sound and haptic effects targeting their enjoyment and perception of the experiment. Accordingly, the eye gaze and heart rate results showed significant influence of the cross-modally mapped multisensorial effects on the users' QoE. Our results highlight that when the olfactory content is crossmodally congruent with the visual content, the visual attention of the users seems shifted towards the correspondent visual feature. Crosmodally matched media is also shown to result in an enhanced QoE compared to a video only condition. [ABSTRACT FROM AUTHOR]

Women might experience modulation in their perception and cognition of colours and odours during the menstrual cycle, but how women's impressions of and correspondence between colours and odours differ according to the cycle changes remains unknown. Here, we experimentally examined women's performance of several tasks, including evaluation of impressions of colours and odours, matching/nonmatching of colours with odours, and identification of odours, comparing two phases: the beginning of menstruation and ovulation. The results showed that participants had similar impressions of colours and odours and made similar colour choices for odours in both the menstrual and ovulation phases, while "pleasant‐unpleasant" impressions of colour and odour might vary according to the menstrual cycle. We found no significant differences in odour identification between the phases. The findings imply that hormonal changes during the menstrual cycle might affect "pleasant‐unpleasant" impressions of colour and odour but not other features regarding impressions or crossmodal correspondence. In future studies, examination with a large number of participants is necessary. [ABSTRACT FROM AUTHOR]

A wide variety of crossmodal correspondences, defined as the often surprising connections that people appear to experience between simple features, attributes, or dimensions of experience, either physically present or else merely imagined, in different sensory modalities, have been demonstrated in recent years. However, a number of crossmodal correspondences have also been documented between more complex (i.e., multi-component) stimuli, such as, for example, pieces of music and paintings. In this review, the extensive evidence supporting the emotional mediation account of the crossmodal correspondences between musical stimuli (mostly pre-recorded short classical music excerpts) and visual stimuli, including colour patches through to, on occasion, paintings, is critically evaluated. According to the emotional mediation account, it is the emotional associations that people have with stimuli that constitutes one of the fundamental bases on which crossmodal associations are established. Taken together, the literature that has been published to date supports emotional mediation as one of the key factors underlying the crossmodal correspondences involving emotionally-valenced stimuli, both simple and complex. [ABSTRACT FROM AUTHOR]

People tend to associate stimuli from different sensory modalities, a phenomenon known as crossmodal correspondences. We conducted two experiments to investigate how Chinese participants associated musical notes produced by four types of Chinese instruments (bowed strings, plucked strings, winds, and percussion instruments) with different colors, taste terms, and fabric textures. Specifically, the participants were asked to select a sound to match each color patch or taste term in Experiment 1 and to match the experience of touching each fabric in Experiment 2. The results demonstrated some associations between pitch and color, taste term, or the smoothness of fabrics. Moreover, certain types of Chinese instruments were preferentially chosen to match some of the colors, taste terms, and the texture of certain fabrics. These findings therefore provided insights about the perception of Chinese music and shed light on how to apply the multisensory features of sounds to enhance the composition, performance, and appreciation of music. [ABSTRACT FROM AUTHOR]

How should we define synesthesia? After more than a century of research, scientists have still not reached an agreement on what synesthesia is (and is not). The author s opinion of this condition is that the disagreement on the definition of synesthesia is essentially a disagreement on determining what sensations are normal and what are unnormaL All different versions of the definition of synesthesia state, either explicitly or implicitly, that synesthesia is an extraordinary sensation, which is caused by the activation of a second sensory or cognitive pathway that is not normally activated by the inducing stimulus. In other words, the boundary between synesthesia and non-synesthesia depends on what sensations are considered normal, and this can only be judged subjectively. In so far as we consider synesthesia as an unnormal/extraordinary phenomenon, it is illogical to hope that there could be some objective criteria to distinguish synesthesia from non-synesthesia. The remedy is to assume continuity between synesthetic and non-synesthetic experiences. [ABSTRACT FROM AUTHOR]

Innovative wine makers and marketers increasingly see the consumers' multisensory experience in the tasting room as a means of differentiating themselves from the competition. With recent research demonstrating the influence of music on the wine-tasting experience, the present study introduced music as a unique aspect of a VIP tasting room experience at a family-owned Finger Lakes winery. A convenience sample of 46 participants tasted four oaked still wines (2 white, 2 red) in silence and with a complementary soundtrack, and rated the fruitiness, spiciness, and smoothness of each wine in both sound conditions. Undisclosed to the participants, the soundtrack had been designed to bring to mind woody/spiced elements of oak ageing. It was hypothesised that listening to the oak soundtrack would alter the evaluation of the wines to be smoother and spicier. The results revealed that the wines tasted while the soundtrack were playing in the background were rated as significantly fruitier and smoother than the same wines when tasted in silence. These results are discussed in terms of the literature on crossmodal correspondences and the emotional mediation account. Moreover, the positive feedback from the attendees showcases music as a viable component of a multisensory experience when visiting a winery. [ABSTRACT FROM AUTHOR]

A common experimental task used to study the accuracy of estimating when a moving object arrives at a designated location is the time-to-contact (TTC) task. The previous studies have shown evidence that sound motion cues influence TTC estimates of a visual moving object. However, the extent to which sound can influence TTC of visual targets still remains unclear. Some studies on the crossmodal correspondence between pitch and speed suggest that descending pitch sounds are associated with faster speeds compared to ascending pitch sounds due to an internal model of gravity. Other studies have shown an opposite pitch-speed mapping (i.e., ascending pitch associated with faster speeds) and no influence of gravity heuristics. Here, we explored whether auditory pitch glides, a continuous pure tone sound either ascending or descending in pitch, influence TTC estimates of a vertically moving visual target and if any observed effects are consistent with a gravity-centered or gravity-unrelated pitch-speed mapping. Subjects estimated when a disc moving either upward or downward at a constant speed reached a visual landmark after the disc disappeared behind an occluder under three conditions: with an accompanying ascending pitch glide, with a descending pitch glide, or with no sound. Overall, subjects underestimated TTC with ascending pitch glides and overestimated TTC with descending pitch glides, compared to the no-sound condition. These biases in TTC were consistent in both disc motion directions. These results suggest that subjects adopted a gravity-unrelated pitch-speed mapping where ascending pitch is associated with faster speeds and descending pitch associated with slower speeds. [ABSTRACT FROM AUTHOR]

We tested the sensory versus decisional origins of two established audiovisual crossmodal correspondences (CMCs; lightness/pitch and elevation/pitch), applying a signal discrimination paradigm to low-level stimulus features and controlling for attentional cueing. An audiovisual stimulus randomly varied along two visual dimensions (lightness: black/white; elevation: high/low) and one auditory dimension (pitch: high/low), and participants discriminated either only lightness, only elevation, or both lightness and elevation. The discrimination task and the stimulus duration varied between subjects. To investigate the influence of crossmodal congruency, we considered the effect of each CMC (lightness/pitch and elevation/pitch) on the sensitivity and criterion of each discrimination as a function of stimulus duration. There were three main findings. First, discrimination sensitivity was significantly higher for visual targets paired congruently (compared with incongruently) with tones while criterion was unaffected. Second, the sensitivity increase occurred for all stimulus durations, ruling out attentional cueing effects. Third, the sensitivity increase was feature specific such that only the CMC that related to the feature being discriminated influenced sensitivity (i.e. lightness congruency only influenced lightness discrimination and elevation congruency only influenced elevation discrimination in the single and dual task conditions). We suggest that these congruency effects reflect low-level sensory processes. [ABSTRACT FROM AUTHOR]

Highlights • The effect of window shape on consumers' product perceptions is evaluated. • Consumers have a general preference for rounded window shapes. • Similarly, triangular window orientation significantly influences product liking. • However, sweet/sour perceptions are not significantly manipulated by shape. • This novel application of prior findings helps lend validity to supporting theory. Abstract Much previous work has demonstrated that the shape of different design elements, when viewed in relation to food, can influence consumers' evaluations, perceptions, and intentions regarding the food viewed. However, relatively few studies have focused specifically on packaging design and, to date, just one has considered the shape of transparent packaging, a prevalent feature of modern packaging designs. We report the results of two within-participants online studies aimed at rapidly exploring the consequences of any crossmodal correspondences elicited by shape in the novel context of transparent windows on packaging design. Across both experiments, 209 participants viewed a subset of 7 window shapes, across 4 'faux' brands in different product categories. Each packaging design was rated for expected overall liking of the product; willingness to purchase, tastiness, sourness, and sweetness, design innovativeness, and design attractiveness. Certain previous findings were replicated, in that a preference for circular (vs. rectangular) windows, and for upwards- (vs. downwards-)pointing triangles, was found. However, crossmodal correspondences between circular shapes and sweetness, and angular shapes and sourness, were not robustly observed. Explanations for this unpredicted finding are suggested. Furthermore, implications and recommendations for commercial practitioners of packaging design are made. [ABSTRACT FROM AUTHOR]

The present study investigates whether obstruent voicing may or may not affect the imagery of different strengths of motor execution. In a modified version of the implicit association test, participants responded to discrimination tasks that include viewing static pictures of athletes in motion and hearing mono-syllabic linguistic sounds. The results suggest that voiced obstruents are compatible with the motion imagery that implies stronger motor executions, whereas voiceless obstruents are compatible with the imagery that implies weaker motor executions. These results provide experimental support for crossmodal associations between the auditory perception of linguistic sounds, namely, the voicing of obstruents, and the visually induced imagery of different levels of strength in motor actions. [ABSTRACT FROM AUTHOR]

Crossmodal correspondences between seemingly independent sensory modalities are often observed in normal participants. For instance, colours commonly map consistently onto pure tones. In this study, we investigated colour-tone mapping in both normal trichromats and in people with congenital blindness. Participants were asked to match tones of differing pitch to named colours. In both cases, the tones selected varied consistently with the colour. The blind responses were similar to those of the trichromats, except in the case of red and green; the blind did not differentiate these colours, whereas the trichromats associated red with a higher tone and green with a lower tone. Otherwise, the results are consistent with a well-established association between pitch and lightness, with lighter colours associated with higher tones, and darker colours with lower tones. Because the blind never had any sensory experience of colour, their matching of colour to pitch is most likely based on semantic understanding. [ABSTRACT FROM AUTHOR]

A growing body of empirical research documents the existence of several interesting crossmodal correspondences between auditory and gustatory/flavor stimuli, demonstrating that people can match specific acoustic and musical parameters with different tastes and flavors. In this context, a number of researchers and musicians arranged their own soundtracks so as to match specific tastes and used them for research purposes, revealing explicit crossmodal effects on judgments of taste comparative intensity or of taste/sound accordance. However, only few studies have examined implicit associations related to taste-sound correspondences. Thus, the present study was conducted in order to assess possible implicit effects associated to the crossmodal congruency/incongruency between auditory cues and food images during the classification of food tastes. To test our hypothesis, we used 'salty' and 'sweet' soundtracks with salty and sweet food images, and asked 88 participants to classify the taste of each food image while listening to the soundtracks. We found that sweet food images were classified faster than salty food images, regardless of which soundtrack was presented. Moreover, we found a congruency effect, demonstrating that such soundtracks are effective in eliciting facilitating effects of taste quality classification with congruent food images. [ABSTRACT FROM AUTHOR]

People show a tendency to map stimulus properties from different sensory modalities onto each other. This phenomenon of crossmodal correspondences has been mostly studied using sensory presentation of stimuli but recently researchers have started using verbal presentation as well. This raised the question of appropriateness of this presentation form for studying crossmodal correspondences and comparability of results obtained using these two presentation forms. To answer this question within the framework of the evaluative similarity hypothesis which posits that crossmodal correspondences of stimuli are due to similarity of their meanings, an experiment was conducted with 96 undergraduate students using 4 tasks. In tasks 1 and 2, pairs of stimuli belonging to different modalities were presented in their sensory (Task 1) and verbal (Task 2) form and participants rated their similarity on a 7-point scale. In Tasks 3 and 4, stimuli from different modalities were presented individually in their sensory (Task 3) and verbal (Task 4) form and participants rated them on three dimensions of connotative or evaluative meaning (arousal, affective and cognitive evaluation). The results suggest the verbal method might not be suitable for studying crossmodal correspondences and that this is due to a problematic status of similarity ratings of verbalized stimuli. Concurrently, the results lend support to the evaluative similarity hypothesis as sensory presented pairs of stimuli were judged to be more similar the smaller was the Euclidean distance between them in a space defined by the three dimensions of evaluative meaning. [ABSTRACT FROM AUTHOR]

Highlights • Pitch classes evoke color sensations in pitch class-color synesthesia. • The role of absolute pitch (AP) on pitch class-color synesthesia remained unknown. • Experiments revealed that AP is not essential for pitch class-color synesthesia. • We propose a two-step hypothesis of colored hearing in pitch class-color synesthetes. Abstract Sounds evoke color sensations in sound-color synesthesia. Recently, we showed that pitch classes (do , re , mi , etc.) have rainbow hues and that colors are linked to the names of pitch classes rather than to their sounds in 15 subjects who had "pitch class-color synesthesia." However, all synesthetes in our previous study had high levels of absolute pitch (AP); therefore the effects of AP on the condition remained unclear. The present study investigated 18 additional pitch class-color synesthetes who had no or lower levels of AP, and confirmed the generality of the above findings. Furthermore, behavioral experiments indicated a two-step process underlying color sensations: pitches are first associated with their pitch class names, and then the pitch class names evoke color sensations. Two separable brain functions underlie pitch-to-color conversion in pitch class-color synesthesia: a musical function of pitch class identification, and the synesthetic association between pitch class and color. [ABSTRACT FROM AUTHOR]

• Crossmodal correspondence effect facilitate the attention of visual targets and decreased inhibition of return. • Crossmodal correspondence congruency increase the amplitude of P2 in the parietal lobe. • Crossmodal correspondence congruency reduce the amplitude difference of P2 between valid and invalid cue conditions. • When audiovisual integration occurred, the suppression in P2 of crossmodal correspondence effect eliminated. Inhibition of return (IOR) has proved to be weakened by audiovisual integration because of the increased perceptual salience of targets. Although other audiovisual interactions, such as crossmodal correspondence, have also been shown to facilitate attentional processes, to the best of our knowledge, no study has investigated the interaction between crossmodal correspondence and IOR. The present study employed Posner's spatial cueing paradigm and manipulated the cue validity, crossmodal correspondence congruency and time interval of auditory and visual stimuli (AV interval) to explore the effect of crossmodal correspondence on the IOR effect. The behavioral results showed a reduced IOR effect under the correspondence congruency condition in contrast to the correspondence incongruency condition at the AV interval of 200 ms, whereas at an AV interval of 80 ms, the decreased IOR effect under crossmodal correspondence congruency was eliminated. The electrophysiological results showed a reduced amplitude difference in P2 between valid and invalid cue conditions when the crossmodal correspondence effect decreased the IOR effect. The present study provided the first evidence of the weakened effect of the crossmodal correspondence effect on the IOR effect, which could be eliminated by audiovisual integration. [ABSTRACT FROM AUTHOR]