Search

When designing multisensorial experiences, robustly predicting the crossmodal perception of olfactory stimuli is a critical factor. We investigate the possibility of predicting olfactory crossmodal correspondences using the underlying physicochemical features. An electronic nose was tuned to the crossmodal perceptual axis of olfaction and was used to foretell people's crossmodal correspondences between odors and the angularity of shapes, smoothness of texture, perceived pleasantness, pitch, and colors. We found that the underlying physicochemical features of odors could be used to predict people's crossmodal correspondences. The human-machine perceptual dimensions that correlated well are the angularity of shapes (r = 0.71), the smoothness of texture (r = 0.82), pitch (r = 0.70), and the lightness of color (r = 0.59). The human-machine perceptual dimensions that did not correlate well (r0.50) are the perceived pleasantness (r = 0.20) and the hue of the color (r = 0.420.44). All perceptual dimensions except for the perceived pleasantness could be robustly predicted (p-values0.0001) including the hue of color. While it is recognized that olfactory perception is strongly shaped by learning and experience, our findings suggest that there is a systematic and predictable link between the physicochemical features of odorous stimuli and crossmodal correspondences. These findings may provide a crucial building block towards the digital transmission of smell and enhancing multisensorial experiences with better designs as well as more engaging, and enriched experiences.

To form a coherent presentation of the world, the brain needs to combine multiple sensory modalities accurately together in the temporal domain. Judgements on the relative timing of audio–visual stimuli are complex, due to the differing propagation speeds of light and sound through the environment and the nervous system, and the dependence of processing latencies on stimulus intensity (Piéron, 1913). Simultaneity judgement (SJ) and temporal order judgement (TOJ) tasks are often used to assess the temporal mechanisms underlying this binding process. However, these tasks consistently produce measures of perceived simultaneity that are uncorrelated with each other, leading to the suggestion that SJ and TOJ tasks could depend on separate neural mechanisms.Parise and Ernst’s (2016) multisensory correlation detector (MCD) model predicts this lack of correlation by assuming two internal processing stages, a lag computation and a correlation. Here we include and empirically evaluate an intensity-dependent processing delay in the MCD model.We estimate the points of subjective simultaneity (PSSs) using both SJ and TOJ tasks for four different visual intensities and a fixed auditory sound level. Evaluation of four variants of the intensity-dependent MCD model shows that the introduction of an early processing delay can predict the different PSS values obtained in the two respective tasks, without the need for later intensity-dependent multisensorial processing stages. Crucially, this early processing delay can be estimated from simple reaction times.

A method for estimating the non-linear gamma transfer function of liquid–crystal displays (LCDs) without the need of a photometric measurement device was described by Xiao et al. (2011) [1]. It relies on observer’s judgments of visual luminance by presenting eight half-tone patterns with luminances from 1/9 to 8/9 of the maximum value of each colour channel. These half-tone patterns were distributed over the screen both over the vertical and horizontal viewing axes. We conducted a series of photometric and psychophysical measurements (consisting in the simultaneous presentation of half-tone patterns in each trial) to evaluate whether the angular dependency of the light generated by three different LCD technologies would bias the results of these gamma transfer function estimations. Our results show that there are significant differences between the gamma transfer functions measured and produced by observers at different viewing angles. We suggest appropriate modifications to the Xiao et al. paradigm to counterbalance these artefacts which also have the advantage of shortening the amount of time spent in collecting the psychophysical measurements.

An extensive network of cortical areas is involved in multisensory object and action recognition. This network draws on inferior frontal, posterior temporal, and parietal areas; activity is modulated by familiarity and the semantic congruency of auditory and visual component signals even if semantic incongruences are created by combining visual and auditory signals representing very different signal categories, such as speech and whole body actions. Here we present results from a high-density ERP study designed to examine the time-course and source location of responses to semantically congruent and incongruent audiovisual speech and body actions to explore whether the network involved in action recognition consists of a hierarchy of sequentially activated processing modules or a network of simultaneously active processing sites. We report two main results:1) There are no significant early differences in the processing of congruent and incongruent audiovisual action sequences. The earliest difference between congruent and incongruent audiovisual stimuli occurs between 240 and 280 ms after stimulus onset in the left temporal region. Between 340 and 420 ms, semantic congruence modulates responses in central and right frontal areas. Late differences (after 460 ms) occur bilaterally in frontal areas.2) Source localisation (dipole modelling and LORETA) reveals that an extended network encompassing inferior frontal, temporal, parasaggital, and superior parietal sites are simultaneously active between 180 and 420 ms to process auditory–visual action sequences. Early activation (before 120 ms) can be explained by activity in mainly sensory cortices. . The simultaneous activation of an extended network between 180 and 420 ms is consistent with models that posit parallel processing of complex action sequences in frontal, temporal and parietal areas rather than models that postulate hierarchical processing in a sequence of brain regions.

An improved method for visual gamma correction is developed for LCD displays to increase the accuracy of digital colour reproduction. Rather than utilising a photometric measurement device, we use observers’ visual luminance judgements for gamma correction. Eight half tone patterns were designed to generate relative luminances from 1/9 to 8/9 for each colour channel. A psychophysical experiment was conducted on an LCD display to find the digital signals corresponding to each relative luminance by visually matching the half-tone background to a uniform colour patch. Both inter- and intra-observer variability for the eight luminance matches in each channel were assessed and the luminance matches proved to be consistent across observers (Δ E 00 E 00 E 00 . We conclude that this observer-based method of visual gamma correction is useful to estimate the OETF for LCD displays. Its major advantage is that no particular functional relationship between digital inputs and luminance outputs has to be assumed.

Many tasks involve the precise estimation of speed and position of moving objects, for instance to catch or avoid objects that cohabit in our environment. Many of these objects are characterised by signal representations in more than one modality, such as hearing and vision. The aim of this study was to investigate the extent to which the simultaneous presentation of auditory and visual signals enhances the estimation of motion speed and instantaneous position. Observers are asked to estimate the instant when a moving object arrives at a target spatial position by pressing a response button. This task requires observers to estimate the speed of the moving object and to calibrate the timing of their manual response such that it coincides with the true arrival time of the moving object. When both visual and auditory motion signals are available, the variability in estimating the arrival time of the moving object is significantly reduced compared to the variability in the unimodal conditions. This reduction in variability is consistent with optimal integration of the auditory and visual speed signals. The average bias in the estimated arrival times depends on the motion speed: for medium speeds (17deg/s) observers' subjective arrival times are earlier than the true arrival times; for high speeds (47deg/s) observers exhibit a (much smaller) bias in the other direction. This speed-dependency suggests that the bias is due to an error in estimating the motion speeds rather than an error in calibrating the timing of the motor response. Finally, in this temporal localization task, the bias and variability show similar patterns for motion defined by vision, audition or both.

Audio–visual speech recognition systems can be categorised into systems that integrate audio–visual features before decisions are made (feature fusion) and those that integrate decisions of separate recognisers for each modality (decision fusion). Decision fusion has been applied at the level of individual analysis time frames, phone segments and for isolated word recognition but in its basic form cannot be used for continuous speech recognition because of the combinatorial explosion of possible word string hypotheses that have to be evaluated. We present a case for decision fusion at the utterance level and propose an algorithm that can be applied efficiently to continuous speech recognition tasks, which we call N -best decision fusion. The system was tested on a single-speaker, continuous digit recognition task where the audio stream was contaminated by additive multi-speaker babble noise. The audio–visual recognition system resulted in lower word error rates for all signal-to-noise conditions tested compared to the audio-alone system. The magnitude of the improvement was dependent on the signal-to-noise ratio.