Search

Abstract For visual estimation of optical flow, which is crucial for various vision analyses, unsupervised learning by view synthesis has emerged as a promising alternative to supervised methods because the ground-truth flow is not readily available in many cases. However, unsupervised learning is likely to be unstable when pixel tracking is lost via occlusion and motion blur, or pixel correspondence is impaired by variations in image content and spatial structure over time. Recognizing that dynamic occlusions and object variations usually exhibit a smooth temporal transition in natural settings, we shifted our focus to model unsupervised learning optical flow from multi-frame sequences of such dynamic scenes. Specifically, we simulated various dynamic scenarios and occlusion phenomena based on Markov property, allowing the model to extract motion laws and thus gain performance in dynamic and occluded areas, which diverges from existing methods without considering temporal dynamics. In addition, we introduced a temporal dynamic model based on a well-designed spatial-temporal dual recurrent block, resulting in a lightweight model structure with fast inference speed. Assuming the temporal smoothness of optical flow, we used the prior motions of adjacent frames to supervise the occluded regions more reliably. Experiments on several optical flow benchmarks demonstrated the effectiveness of our method, as the performance is comparable to several state-of-the-art methods with advantages in memory and computational overhead.

Summary: The neural and computational mechanisms underlying visual motion perception have been extensively investigated over several decades, but little attempt has been made to measure and analyze, how human observers perceive the map of motion vectors, or optical flow, in complex naturalistic scenes. Here, we developed a psychophysical method to assess human-perceived motion flows using local vector matching and a flash probe. The estimated perceived flow for naturalistic movies agreed with the physically correct flow (ground truth) at many points, but also showed consistent deviations from the ground truth (flow illusions) at other points. Comparisons with the predictions of various computational models, including cutting-edge computer vision algorithms and coordinate transformation models, indicated that some flow illusions are attributable to lower-level factors such as spatiotemporal pooling and signal loss, while others reflect higher-level computations, including vector decomposition. Our study demonstrates a promising data-driven psychophysical paradigm for an advanced understanding of visual motion perception.

Stereoscopic 3D displays (S3D), the most popular consumer display devices for 3D presentation, have a few problems that degrade the natural visual experience, such as unnatural relationships between eye vergence and accommodation, and severe image blurring (ghost) for viewers without stereo glasses. To simultaneously solve these problems, we combine gaze-contingent disparity remapping with Hidden Stereo in a manner that mutually compensates for their respective shortcomings. Gaze-contingent disparity remapping can reduce the vergence-accommodation conflict by shifting the disparity distribution around the gaze position to be centered on the display plane. Hidden Stereo can synthesize 2D-compatible 3D stereo images that do not produce any ghosting artifacts when the images for the two eyes are linearly fused. Thus, by using our new gaze-contingent display, while one viewer with glasses enjoys natural 3D content, many other glassless viewers enjoy clear 2D content. To enable real-time synthesis, we accelerate Hidden Stereo conversion by limiting the processing to each horizontal scanline. Through a user study using a variety of 3D scenes, we demonstrate that Hidden Stereo can effectively hide disparity information to glassless viewers despite the dynamic disparity manipulations. Moreover, we show that our method can alleviate the limitation of Hidden Stereo—the narrow reproducible disparity range—by manipulating the disparity so that the depth information around the gaze position is maximally preserved.

It has been suggested that perceiving blurry images in addition to sharp images contributes to the development of robust human visual processing. To computationally investigate the effect of exposure to blurry images, we trained convolutional neural networks (CNNs) on ImageNet object recognition with a variety of combinations of sharp and blurred images. In agreement with recent reports, mixed training on blurred and sharp images (B+S training) brings CNNs closer to humans with respect to robust object recognition against a change in image blur. B+S training also slightly reduces the texture bias of CNNs in recognition of shape-texture cue conflict images, but the effect is not strong enough to achieve human-level shape bias. Other tests also suggest that B+S training cannot produce robust human-like object recognition based on global configuration features. Using representational similarity analysis and zero-shot transfer learning, we also show that B+S-Net does not facilitate blur-robust object recognition through separate specialized sub-networks, one network for sharp images and another for blurry images, but through a single network analyzing image features common across sharp and blurry images. However, blur training alone does not automatically create a mechanism like the human brain in which sub-band information is integrated into a common representation. Our analysis suggests that experience with blurred images may help the human brain recognize objects in blurred images, but that alone does not lead to robust, human-like object recognition.

Android robots capable of emotional interactions with humans have considerable potential for application to research. While several studies developed androids that can exhibit human-like emotional facial expressions, few have empirically validated androids’ facial expressions. To investigate this issue, we developed an android head called Nikola based on human psychology and conducted three studies to test the validity of its facial expressions. In Study 1, Nikola produced single facial actions, which were evaluated in accordance with the Facial Action Coding System. The results showed that 17 action units were appropriately produced. In Study 2, Nikola produced the prototypical facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and naïve participants labeled photographs of the expressions. The recognition accuracy of all emotions was higher than chance level. In Study 3, Nikola produced dynamic facial expressions for six basic emotions at four different speeds, and naïve participants evaluated the naturalness of the speed of each expression. The effect of speed differed across emotions, as in previous studies of human expressions. These data validate the spatial and temporal patterns of Nikola’s emotional facial expressions, and suggest that it may be useful for future psychological studies and real-life applications.

Motion detection is a fundamental sensory function for multiple modalities, including touch, but the mechanisms underlying tactile motion detection are not well understood. While previous findings supported the existence of high-level feature tracking, it remains unclear whether there also exist low-level motion sensing that directly detects a local spatio-temporal correlation in the skin-stimulation pattern. To elucidate this mechanism, we presented, on braille displays, tactile random-dot kinematograms, similar to those widely used in visual motion research, which enables us to independently manipulate feature trackability and various parameters of local motion. We found that a human observer is able to detect the direction of difficult-to-track tactile motions presented to the fingers and palms. In addition, the direction-discrimination performance was better when the stimuli were presented along the fingers than when presented across the fingers. These results indicate that low-level motion sensing, in addition to high-level tracking, contribute to tactile motion perception.

Visually inferring material properties is crucial for many tasks, yet poses significant computational challenges for biological vision. Liquids and gels are particularly challenging due to their extreme variability and complex behaviour. We reasoned that measuring and modelling viscosity perception is a useful case study for identifying general principles of complex visual inferences. In recent years, artificial Deep Neural Networks (DNNs) have yielded breakthroughs in challenging real-world vision tasks. However, to model human vision, the emphasis lies not on best possible performance, but on mimicking the specific pattern of successes and errors humans make. We trained a DNN to estimate the viscosity of liquids using 100.000 simulations depicting liquids with sixteen different viscosities interacting in ten different scenes (stirring, pouring, splashing, etc). We find that a shallow feedforward network trained for only 30 epochs predicts mean observer performance better than most individual observers. This is the first successful image-computable model of human viscosity perception. Further training improved accuracy, but predicted human perception less well. We analysed the network's features using representational similarity analysis (RSA) and a range of image descriptors (e.g. optic flow, colour saturation, GIST). This revealed clusters of units sensitive to specific classes of feature. We also find a distinct population of units that are poorly explained by hand-engineered features, but which are particularly important both for physical viscosity estimation, and for the specific pattern of human responses. The final layers represent many distinct stimulus characteristics-not just viscosity, which the network was trained on. Retraining the fully-connected layer with a reduced number of units achieves practically identical performance, but results in representations focused on viscosity, suggesting that network capacity is a crucial parameter determining whether artificial or biological neural networks use distributed vs. localized representations.

Many studies have investigated various effects of smooth pursuit on visual motion processing, especially the effects related to the additional retinal shifts produced by eye movement. In this article, we show that the perception of apparent motion during smooth pursuit is determined by the interelement proximity in retinal coordinates and also by the proximity in objective world coordinates. In Experiment 1, we investigated the perceived direction of the two-frame apparent motion of a square-wave grating with various displacement sizes under fixation and pursuit viewing conditions. The retinal and objective displacements between the two frames agreed with each other under the fixation condition. However, the displacements differed by 180 degrees in terms of phase shift, under the pursuit condition. The proportions of the reported motion direction between the two viewing conditions did not coincide when they were plotted as a function of either the retinal displacement or of the objective displacement; however, they did coincide when plotted as a function of a mixture of the two. The result from Experiment 2 showed that the perceived jump size of the apparent motion was also dependent on both retinal and objective displacements. Our findings suggest that the detection of the apparent motion during smooth pursuit considers the retinal proximity and also the objective proximity. This mechanism may assist with the selection of a motion path that is more likely to occur in the real world and, therefore, be useful for ensuring perceptual stability during smooth pursuit.

Abstract Signals in one sensory modality can influence perception of another, for example the bias of visual timing by audition: temporal ventriloquism. Strong accounts of temporal ventriloquism hold that the sensory representation of visual signal timing changes to that of the nearby sound. Alternatively, underlying sensory representations do not change. Rather, perceptual grouping processes based on spatial, temporal, and featural information produce best-estimates of global event properties. In support of this interpretation, when feature-based perceptual grouping conflicts with temporal information-based in scenarios that reveal temporal ventriloquism, the effect is abolished. However, previous demonstrations of this disruption used long-range visual apparent-motion stimuli. We investigated whether similar manipulations of feature grouping could also disrupt the classical temporal ventriloquism demonstration, which occurs over a short temporal range. We estimated the precision of participants’ reports of which of two visual bars occurred first. The bars were accompanied by different cross-modal signals that onset synchronously or asynchronously with each bar. Participants’ performance improved with asynchronous presentation relative to synchronous - temporal ventriloquism - however, unlike the long-range apparent motion paradigm, this was unaffected by different combinations of cross-modal feature, suggesting that featural similarity of cross-modal signals may not modulate cross-modal temporal influences in short time scales.

Abstract A wide variety of tactile sensations arise from the activation of several types of mechanoreceptor-afferent channels scattered all over the body, and their projections create a somatotopic map in the somatosensory cortex. Recent findings challenge the traditional view that tactile signals from different mechanoreceptor-channels/locations are independently processed in the brain, though the contribution of signal integration to perception remains obscure. Here we show that vibrotactile frequency perception is functionally enriched by signal integration across different mechanoreceptor channels and separate skin locations. When participants touched two sinusoidal vibrations of far-different frequency, which dominantly activated separate channels with the neighboring fingers or the different hand and judged the frequency of one vibration, the perceived frequency shifted toward the other (assimilation effect). Furthermore, when the participants judged the frequency of the pair as a whole, they consistently reported an intensity-based interpolation of the two vibrations (averaging effect). Both effects were similar in magnitude between the same and different hand conditions and significantly diminished by asynchronous presentation of the vibration pair. These findings indicate that human tactile processing is global and flexible in that it can estimate the ensemble property of a large-scale tactile event sensed by various receptors distributed over the body.

Visual estimation of the material and shape of an object from a single image includes a hard ill-posed computational problem. However, in our daily life we feel we can estimate both reasonably well. The neural computation underlying this ability remains poorly understood. Here we propose that the human visual system uses different aspects of object images to separately estimate the contributions of the material and shape. Specifically, material perception relies mainly on the intensity gradient magnitude information, while shape perception relies mainly on the intensity gradient order information. A clue to this hypothesis was provided by the observation that luminance-histogram manipulation, which changes luminance gradient magnitudes but not the luminance-order map, effectively alters the material appearance but not the shape of an object. In agreement with this observation, we found that the simulated physical material changes do not significantly affect the intensity order information. A series of psychophysical experiments further indicate that human surface shape perception is robust against intensity manipulations provided they do not disturb the intensity order information. In addition, we show that the two types of gradient information can be utilized for the discrimination of albedo changes from highlights. These findings suggest that the visual system relies on these diagnostic image features to estimate physical properties in a distal world.

When two sequential video frames extracted from a single video clip are followed by the negative of the two frames, a viewer often experiences a visual illusion whereby a scene in the frames continuously moves in a single direction (four-stroke apparent motion). To create a four-stroke apparent motion display, the image intensities of the whole of the second pair of images are reversed. However, this intensity reversal creates a strong impression of flicker that can be undesirable for comfortable viewing. This study reports that four-stroke apparent motion can be induced by only reversing the luminance intensities in those spatial areas which contain motion signals in high-pass filtered images. This use of only a partial reversal of image intensities greatly reduces the apparent flicker in the display while retaining motion perception.

An Asian spice, Szechuan pepper (sanshool), is well known for the tingling sensation it induces on the mouth and on the lips. Electrophysiological studies have revealed that its active ingredient can induce firing of mechanoreceptor fibres that typically respond to mechanical vibration. Moreover, a human behavioral study has reported that the perceived frequency of sanshool-induced tingling matches with the preferred frequency range of the tactile rapidly adapting (RA) channel, suggesting the contribution of sanshool-induced RA channel firing to its unique perceptual experience. However, since the RA channel may not be the only channel activated by sanshool, there could be a possibility that the sanshool tingling percept may be caused in whole or in part by other sensory channels. Here, by using a perceptual interference paradigm, we show that the sanshool-induced RA input indeed contributes to the human tactile processing. The absolute detection thresholds for vibrotactile input were measured with and without sanshool application on the fingertip. Sanshool significantly impaired detection of vibrations at 30 Hz (RA channel dominant frequency), but did not impair detection of higher frequency vibrations at 240 Hz (Pacinian-corpuscle (PC) channel dominant frequency) or lower frequency vibrations at 1 Hz (slowly adapting 1 (SA1) channel dominant frequency). These results show that the sanshool induces a peripheral RA channel activation that is relevant for tactile perception. This anomalous activation of RA channels may contribute to the unique tingling experience of sanshool.

The influence of body movements on visual time perception is receiving increased attention. Past studies showed apparent expansion of visual time before and after the execution of hand movements and apparent compression of visual time during the execution of eye movements. Here we examined whether the estimation of sub-second time intervals between visual events is expanded, compressed, or unaffected during the execution of hand movements. The results show that hand movements, at least the fast ones, reduced the apparent time interval between visual events. A control experiment indicated that the apparent time compression was not produced by the participants' involuntary eye movements during the hand movements. These results, together with earlier findings, suggest hand movement can change apparent visual time either in a compressive way or in an expansive way, depending on the relative timing between the hand movement and visual stimulus.

Different features of a stimulus, such as color, motion, or orientation, are to some extent processed independently in the brain. It is not clear, however, whether a change within a feature is perceived with different delays for different features because different methods —temporal order, synchrony, and pairing judgments—show conflicting results. Previous methods provide estimates of perceived timing that often are quite variable across trials and observers suggesting the involvement of high-level strategies. Here, we report a new method to measure the perceived timing of feature changes that shows low variability. When four pac-men are presented properly aligned above a background defined by some feature, the illusory surface that pac-men produce appears to be filled in with the feature. We asked observers to report the feature value inside a briefly presented illusory surface when the feature in the background changed over time (Motoyoshi, 2007). We found that observers' estimates are very precise for both continuous and discrete feature changes. For example, when a background changes motion repetitively between two discrete values (60 ms leftwards and 60 ms rightwards), observers can precisely report the motion direction filling in an illusory surface presented for 60 ms. We propose that our method measures the perceived timing for different features at the level of surface formation.

Local motion signals are extracted in parallel by a bank of motion detectors, and their spatiotemporal interactions are processed in subsequent stages. In this talk, I will review our recent studies on spatial interactions in visual motion processing. First, we found two types of spatial pooling of local motion signals. Directionally ambiguous 1D local motion signals are pooled across orientation and space for solution of the aperture problem, while 2D local motion signals are pooled for estimation of global vector average (e.g., Amano et al., 2009 Journal of Vision 9 (3:4) 1–25). Second, when stimulus presentation is brief, coherent motion detection of dynamic random-dot kinematogram is not efficient. Nevertheless, it is significantly improved by transient and synchronous presentation of a stationary surround pattern. This suggests that centre-surround spatial interaction may help rapid perception of motion (Linares et al., submitted). Third, to know how the visual system encodes pairwise relationships between remote motion signals, we measured the temporal rate limit for perceiving the relationship of two motion directions presented at the same time at different spatial locations. Compared with similar tasks with luminance or orientation signals, motion comparison was more rapid and hence efficient. This high performance was affected little by inter-element separation even when it was increased up to 100 deg. These findings indicate the existence of specialized processes to encode long-range relationships between motion signals for quick appreciation of global dynamic scene structure (Maruya et al., in preparation).

Agents have to determine which external events their action has causally produced. A sensation of causal relation between action and outcome is called subjective causality. Subjective causality has been linked to the comparator model. This model assumes that the brain compares an internal prediction for action outcome with an actual sensory outcome, distinguishing between self and externally produced outcomes depending on spatiotemporal congruency. However, recent studies have expressed some doubt about the idea that subjective causality arises depending solely on the spatiotemporal congruency, suggesting instead that other perceptual/cognitive factors play a critical role in determining subjective causality. We hypothesized that crossmodal grouping between action and outcome contributed to subjective causality. Crossmodal temporal grouping is an essential factor for crossmodal simultaneity judgments with ungrouped crossmodal signals likely to be judged as non-simultaneous. We predicted that subjective causality would decrease when an agent's action was not temporally grouped with action outcome. In the experiment, observers were asked to press a key in order to trigger a display change with some temporal delay. To disrupt temporal grouping between action and outcome, a task-irrelevant visual flash or tone was sometimes presented synchronously with the button press and/or the display change. Subjective causality was decreased when the flash or the tone was coincided with the button press. This demonstrates that perceptual grouping has a key role in determination of subjective causality, a result that is not accounted for by the standard comparator model.

The cross-modal double flash illusion (DFI) refers to an interaction of multisensory signals wherein the presentation of a single visual flash accompanied by two cross-modal inducers, auditory or tactile pulses, results in observers reporting the presentation of two flashes. This phenomenon has become an exemplary case of the strong modulatory effect possible between cross-modal signals. However, it remains unclear precisely what about the different signals is interacting. The prevailing interpretation typically invokes interactions between different singular sensory representations—an explanation consistent with what would be expected in detection tasks. Here we investigated whether such a simple interaction could in fact account for the DFI. Using a paradigm similar to the original, we manipulated the apparent similarity of the cross-modal inducers such that they differed in attribute (1000 Hz pure tone or Gaussian noise) or modality (audio or tactile). When the inducer pair was the same, a DFI was found. However, if the inducer pair differed, the DFI was strongly mitigated, if not abolished. These results demonstrate that the DFI depends critically on the apparent similarity of the cross-modal inducers, something that wouldn't matter if the illusion were based on direct interactions of discrete sensory representations. We propose that the DFI is the result of interactions between multi-event temporal structures, rather than discrete sensory events. These structures are determined through within-mode grouping and compared and combined supra-modally to generate event representations. This type of interaction likely underlies other multisensory timing phenomena such as temporal ventriloquism.

The onset position of a visual moving object is mislocalized in the direction of subsequent motion. Although recent studies have proved that visual and auditory motion processing is closely related to each other, the interaction of motion and position coding in a cross-modal situation was largely unknown. The present study explored whether the position of a visual flash was mislocalized in the direction of subsequent auditory motion. Observers first memorized the position of a probe and then compared this position with the position of a visual flash, which acted as a target; the target temporally lagged the probe by 600 ms. The auditory motion was defined on the basis of the interaural level difference, and lasted for 100 ms. The onset of the target was synchronized with the onset of the auditory motion. As a result, the target location was significantly mislocalized in the direction of subsequent auditory motion. The results indicate that the interaction of motion and position coding occurs beyond the sensory modality.

An informative performance measure of the brain's integration across different sensory attributes/modalities is the critical temporal rate of feature alternation (between, eg, red and green) beyond which observers could not identify the feature value specified by a timing signal from another attribute (eg, a pitch change). Interestingly, this limit, which we called the critical crowding frequency (CCF), is fairly low and nearly constant (∼2.5 Hz) regardless of the combination of attributes and modalities (Fujisaki & Nishida, 2010, IMRF). One may consider that the CCF reflects the processing time required for the brain to identify the specified feature value on the fly. According to this idea, the similarity in CCF could be ascribed to the similarity in identification time for the attributes we used (luminance, color, orientation, pitch, vibration). To test this idea, we estimated the identification time of each attribute from [Go/ No-Go choice reaction time – simple reaction time]. In disagreement with the prediction, we found significant differences among attributes (eg, ∼160 ms for orientation, ∼70 ms for pitch). The results are more consistent with our proposal (Fujisaki & Nishida, Proc Roy Soc B) that the CCF reflects the common rate limit of specifying what happens when (timing-content binding) by a central, presumably postdictive, mechanism.

Events encoded in separate sensory modalities, such as audition and vision, can seem to be synchronous across a relatively broad range of physical timing differences. This may suggest that the precision of audio-visual timing judgments is inherently poor. Here we show that this is not necessarily true. We contrast timing sensitivity for isolated streams of audio and visual speech, and for streams of audio and visual speech accompanied by additional, temporally offset, visual speech streams. We find that the precision with which synchronous streams of audio and visual speech are identified is enhanced by the presence of additional streams of asynchronous visual speech. Our data suggest that timing perception is shaped by selective grouping processes, which can result in enhanced precision in temporally cluttered environments. The imprecision suggested by previous studies might therefore be a consequence of examining isolated pairs of audio and visual events. We argue that when an isolated pair of cross-modal events is presented, they tend to group perceptually and to seem synchronous as a consequence. We have revealed greater precision by providing multiple visual signals, possibly allowing a single auditory speech stream to group selectively with the most synchronous visual candidate. The grouping processes we have identified might be important in daily life, such as when we attempt to follow a conversation in a crowded room.

Human observers see a single mixed color (yellow) when different colors (red and green) rapidly alternate. Accumulating evidence suggests that the critical temporal frequency beyond which chromatic fusion occurs does not simply reflect the temporal limit of peripheral encoding. However, it remains poorly understood how the central processing controls the fusion frequency. Here we show that the fusion frequency can be elevated by extra-retinal signals during smooth pursuit. This eye movement can keep the image of a moving target in the fovea, but it also introduces a backward retinal sweep of the stationary background pattern. We found that the fusion frequency was higher when retinal color changes were generated by pursuit-induced background motions than when the same retinal color changes were generated by object motions during eye fixation. This temporal improvement cannot be ascribed to a general increase in contrast gain of specific neural mechanisms during pursuit, since the improvement was not observed with a pattern flickering without changing position on the retina or with a pattern moving in the direction opposite to the background motion during pursuit. Our findings indicate that chromatic fusion is controlled by a cortical mechanism that suppresses motion blur. A plausible mechanism is that eye-movement signals change spatiotemporal trajectories along which color signals are integrated so as to reduce chromatic integration at the same locations (i.e., along stationary trajectories) on the retina that normally causes retinal blur during fixation.