Search

Artificial intelligence research in natural language processing in the context of poetry struggles with the recognition of holistic content such as poetic symbolism, metaphor, and other fine-grained attributes. Given these challenges, multi-modal image–poetry reasoning and retrieval remain largely unexplored. Our recent accessibility study indicates that poetry is an effective medium to convey visual artwork attributes for improved artwork appreciation of people with visual impairments. We, therefore, introduce a deep learning approach for the automatic retrieval of poetry suitable to the input images. The recent state-of-the-art CLIP provides a way for multi-modal visual and text features matched using cosine similarity. However, it lacks shared cross-modality attention features to model fine-grained relationships. The proposed approach in this work takes advantage of strong pre-training of the CLIP model and overcomes its limitations by introducing shared attention parameters to better model the fine-grained relationship between both modalities. We test and compare our proposed approach using the expertly annotated MiltiM-Poem dataset, which is considered the largest public image–poetry pair dataset for English poetry. The proposed approach aims to solve the problems of image-based attribute recognition and automatic retrieval for fine-grained poetic verses. The test results reflect that the shared attention parameters alleviate fine-grained attribute recognition, and the proposed approach is a significant step towards automatic multi-modal retrieval for improved artwork appreciation of people with visual impairments.

Tactile perception enables people with visual impairments (PVI) to engage with artworks and real-life objects at a deeper abstraction level. The development of tactile and multi-sensory assistive technologies has expanded their opportunities to appreciate visual arts. We have developed a tactile interface based on the proposed concept design under considerations of PVI tactile actuation, color perception, and learnability. The proposed interface automatically translates reference colors into spatial tactile patterns. A range of achromatic colors and six prominent basic colors with three levels of chroma and values are considered for the cross-modular association. In addition, an analog tactile color watch design has been proposed. This scheme enables PVI to explore artwork or real-life object color by identifying the reference colors through a color sensor and translating them to the tactile interface. The color identification tests using this scheme on the developed prototype exhibit good recognition accuracy. The workload assessment and usability evaluation for PVI demonstrate promising results. This suggest that the proposed scheme is appropriate for tactile color exploration.

User Experience Design (UX Design) comes from focusing on how products, in reality, affect the user's experience. In particular, the design of multi-modal interfaces for blind people facilitates the flexible and natural product or service capacity and improves blind people's interaction by overcoming the various existing constraints associated with any particular interaction. There have been various attempts to help visually impaired people appreciation of visual artwork, including multi-modal associations. However, these methods can only provide general information in terms of edge and pattern recognition by the sense of touch and restrained by the availability and number of specially developed artworks. We propose a novel method explaining visual artworks through image caption generation using artificial intelligence (AI) to improve artwork accessibility. This method can objectively describe any impressionism artwork used as a standalone description of art interpretation for blind people or can aide tactile-based methods. Based on end-to-end learning with a deep neural network, an encoder-decoder architecture model is adopted, and comprehensive experiments perform to confirm the stability of generated image captioning for stylized MS-COCO datasets with impressionism.

Pedestrian dead reckoning (PDR) for indoor localization has the privilege of utility without prior environment knowledge and adjunct infrastructure. It can be a potentially scalable solution for various indoor positioning, visually impaired people navigation aide, and commercial applications. In this study, we develop a multi-stage deep learning-based approach to detect and estimate the stride and heading of a user. This approach takes advantage of classifying user action units from inertial sensors of smartphone and relevant action units and is then separately processed to estimate user displacement in terms of displacement distance and direction, respectively, with automatic feature extraction. The proposed system provides improved performance over the preceding deep learning model. It also exhibits two-dimensional finer resolution maneuvering of the user in contrast to only the left and right turn. Experiments were conducted to train and evaluate the proposed system's performance, the results of which validate the improved utility of our deep learning-based system.

This study aims toward the development of an automatic wristwatch(ColorWatch) tactile system which may assist People with Visual Impairments (PVI) to recognize object colors in their daily life. This work identified perpetual colors in daily lives and associated them with cross-modular tactile patterns. The designed system considers the fundamentals of tactile actuation, color perception, and the intuitive learnability of humans. Achromatic and monochromatic colors as well as the tone of these for each color in two levels are considered and a one-to-one cross-modular mapping with the tactile pattern has been proposed. Furthermore, the system considers artwork as well as a variety of general-purpose object color translation. Focus group experiments indicate good recognition results and usability of tactile patterns for the tactile actuation and their color associations. Based on the proposed concept and experimental evaluation, further ColorWatch prototype and user devices for translation of color into an automatic tactile system may be developed.

Multi-sensory interaction aids learning, inclusion, and collaboration because it accommodates the diverse cognitive and perceptual needs [...]

Recent developments in the field of preventive healthcare have received considerable attention due to the effective management of various chronic diseases including diabetes, heart stroke, obesity, and cancer. Various automated systems are being used for activity and food recognition in preventive healthcare. The automated systems lack sophisticated segmentation techniques and contain multiple sensors, which are inconvenient to be worn in real-life settings. To monitor activity and food together, our work presents a novel wearable system that employs the motion sensors in a smartwatch together with a piezoelectric sensor embedded in a necklace. The motion sensor generates distinct patterns for eight different physical activities including eating activity. The piezoelectric sensor generates different signal patterns for six different food types as the ingestion of each food is different from the others owing to their different characteristics: hardness, crunchiness, and tackiness. For effective representation of the signal patterns of the activities and foods, we employ dynamic segmentation. A novel algorithm called event similarity search (ESS) is developed to choose a segment with dynamic length, which represents signal patterns with different complexities equally well. Amplitude-based features and spectrogram-generated images from the segments of activity and food are fed to convolutional neural network (CNN)-based activity and food recognition networks, respectively. Extensive experimentation showed that the proposed system performs better than the state of the art methods for recognizing eight activity types and six food categories with an accuracy of 94.3% and 91.9% using support vector machine (SVM) and CNN, respectively.

The number of studies on the development of indoor positioning systems has increased recently due to the growing demands of the various location-based services. Inertial sensors available in commercial smartphones play an important role in indoor localization and navigation owing to their highly accurate localization performance. In this study, the inertial sensors of a smartphone, which generate distinct patterns for physical activities and action units (AUs), are employed to localize a target in an indoor environment. These AUs, (such as a left turn, right turn, normal step, short step, or long step), help to accurately estimate the indoor location of a target. By taking advantage of sophisticated deep learning algorithms, we propose a novel approach for indoor navigation based on long short-term memory (LSTM). The LSTM accurately recognizes physical activities and related AUs by automatically extracting the efficient features from the distinct patterns of the input data. Experiment results show that LSTM provides a significant improvement in the indoor positioning performance through the recognition task. The proposed system achieves a better localization performance than the trivial fingerprinting method, with an average error of 0.782 m in an indoor area of 128.6 m2. Additionally, the proposed system exhibited robust performance by excluding the abnormal activity from the pedestrian activities.