Search

The development of assistive technologies is improving the independent access of blind and visually impaired people to visual artworks through non-visual channels. Current single modality tactile and auditory approaches to communicate color contents must compromise between conveying a broad color palette, ease of learning, and suffer from limited expressiveness. In this work, we propose a multi-sensory color code system that uses sound and scent to represent colors. Melodies express each color’s hue and scents the saturated, light, and dark color dimensions for each hue. In collaboration with eighteen participants, we evaluated the color identification rate achieved when using the multi-sensory approach. Seven (39%) of the participants improved their identification rate, five (28%) remained the same, and six (33%) performed worse when compared to an audio-only color code alternative. The participants then evaluated and compared a color content exploration prototype that uses the proposed color code with a tactile graphic equivalent using the System Usability Scale. For a visual artwork color exploration task, the multi-sensory color code integrated prototype received a score of 78.61, while the tactile graphics equivalent received 61.53. User feedback indicates that the multi-sensory color code system improved the convenience and confidence of the participants.

For years the HCI community’s research has been focused on the hearing and sight senses. However, in recent times, there has been an increased interest in using other types of senses, such as smell or touch. Moreover, this has been accompanied with growing research related to sensory substitution techniques and multi-sensory systems. Similarly, contemporary art has also been influenced by this trend and the number of artists interested in creating novel multi-sensory works of art has increased substantially. As a result, the opportunities for visually impaired people to experience artworks in different ways are also expanding. In spite of all this, the research focusing on multimodal systems for experiencing visual arts is not large and user tests comparing different modalities and senses, particularly in the field of art, are insufficient. This paper attempts to design a multi-sensory mapping to convey color to visually impaired people employing musical sounds and temperature cues. Through user tests and surveys with a total of 18 participants, we show that this multi-sensory system is properly designed to allow the user to distinguish and experience a total of 24 colors. The tests consist of several semantic correlational adjective-based surveys for comparing the different modalities to find out the best way to express colors through musical sounds and temperature cues based on previously well-established sound-color and temperature-color coding algorithms. In addition, the resulting final algorithm is also tested with 12 more users.

The objective of this study was to improve user experience when appreciating visual artworks with soundscape music chosen by a deep neural network based on weakly supervised learning. We also propose a multi-faceted approach to measuring ambiguous concepts, such as the subjective fitness, implicit senses, immersion, and availability. We showed improvements in appreciation experience, such as the metaphorical and psychological transferability, time distortion, and cognitive absorption, with in-depth experiments involving 70 participants. Our test results were similar to those of “Bunker de Lumières: van Gogh”, which is an immersive media artwork directed by Gianfranco lannuzzi, the fitness scores of our system and “Bunker de Lumières: van Gogh” were 3.68/5 and 3.81/5, respectively. Moreover, the concordance of implicit senses between artworks and classical music was measured to be 0.88%, and the time distortion and cognitive absorption improved during the immersion. Finally, the proposed method obtained a subjective satisfaction score of 3.53/5 in the evaluation of its usability. Our proposed method can also help spread soundscape-based media art by supporting traditional soundscape design. Furthermore, we hope that our proposed method will help people with visual impairments to appreciate artworks through its application to a multi-modal media art guide platform.

Good performance is important element not only in workplace but also in daily activities. Performance of the human depends on the mental capacity and mental workload. Especially, children in concrete operational stage is critical for further learning ability that they develop their ability to distinguish between quality and quantity. However, the reason that mental workload is difficult to quantify through physiological measures, makes it more complicated to demonstrate the mental workload. When it comes to children’s development, physical change is visible and easy to identify but mental change is not. HRV is relatively easy to measure but has limitation because it is indirect way of measuring brain signal. Above all things, many researches of real-time indicator measuring physiological data such as heart rate variability (HRV) have been done sporadically but not integrated. Therefore, In this study we tried to demonstrate if we can predict the mental capacity not mental workload with the EEG. Attention ability was measured with Stroop task, and memory ability was measured with digit span task. The main outcome of this study is that building predictive models for cognitive functions using physiological measures is feasible and that its predictive models for cognitive functions using physiological measures is feasible and that its predictive power is further improved when EEG is used along with HRV data. It is implied form the outcome of study that combining physiological measures may improve its predictive power by improving the signal relative to noises and that future studies may focus on discovery of further biomarkers for prediction of cognitive functions.

The earth has abundant water resources, and efficient harvesting of energy from waves is always a research hotspot. In this paper, we fabricate a box-like triboelectric-electromagnetic hybrid nanogenerator for harvesting water wave energy, which is composed of the triboelectric nanogenerator (TENG) and the electromagnetic generator (EMG). In the case of TENG, the contact and separation between the aluminum film-coated PMMA cylinder and the silicone film covered interdigitated Ag electrodes is formed by ocean waves. In the case of EMG, a rolling magnetic ball is placed on the four coils to generate magnetic flux changes, thus generating electricity. Several experiments are conducted to validate the availability. Experimental results show that the instantaneous maximum output power of TENG is 0.08 mW at a load of 100 MΩ, and the instantaneous maximum output power of EMG is 14.9 mW at a load of 1 kΩ. The hybrid nanogenerator can light up 60 LEDs at the same time. After storing energy in the capacitor, the output DC signal can provide a continuous energy supply with the temperature sensor. More important, the nanogenerator can power the wireless acoustic sensing system.