Search

Audio Description (AD) or Video Description is a vital accessibility concept in blind and visually impaired people's life. Automating this task is not easy and involves many problems, such as describing the scenario, actions, emotions, and characters. This paper presents an approach to automatically describe characters --- in a video or image --- combining Deep Learning (DL), Face detection, Facial Expression detection techniques, and audio synthesizers. Our proposal uses the detection tools, applies some DL models to the analyzed data, and generates an audio description. To evaluate the feasibility of our proposal, we have developed a proof of concept of the solution and performed some computational experiments to evaluate it.

We present Slidecho, a system that enables non-visual access of the slide content in a presentation video on-demand. Slidecho automatically extracts slides and their text and image elements from the presentation video and aligns these elements to the presenter’s speech. When listening to the video, Slidecho provides learners with audio notifications about slide changes and slide elements that are not described by the presenter. The learner can pause the video and browse the entire slide, or only the undescribed slide elements, to gain information. A technical evaluation with presentation videos in-the-wild shows that compared to the presenter’s speech alone, Slidecho provides access to an additional 20% of total text elements and 30% of total image elements that were previously not described. Blind and visually impaired participants in our user study reported that it was easier to locate undescribed slide elements with Slidecho’s synchronized interface than when browsing the video and extracted slides separately, and using Slidecho they read fewer slides that were fully redundant with the speech.

Video accessibility is essential for people with visual impairments. Audio descriptions describe what is happening on-screen, e.g., physical actions, facial expressions, and scene changes. Generating high-quality audio descriptions requires a lot of manual description generation [50]. To address this accessibility obstacle, we built a system that analyzes the audiovisual contents of a video and generates the audio descriptions. The system consisted of three modules: AD insertion time prediction, AD generation, and AD optimization. We evaluated the quality of our system on five types of videos by conducting qualitative studies with 20 sighted users and 12 users who were blind or visually impaired. Our findings revealed how audio description preferences varied with user types and video types. Based on our study’s analysis, we provided recommendations for the development of future audio description generation technologies.

Social media platforms feature short animations known as GIFs, but they are inaccessible to people with vision impairments. Unlike static images, GIFs contain action and visual indications of sound, which can be challenging to describe in alternative text descriptions. We examine a large sample of inaccessible GIFs on Twitter to document how they are used and what visual elements they contain. In interviews with 10 blind Twitter users, we discuss what elements of GIF content should be described and their experiences with GIFs online. The participants compared alternative text descriptions with two other alternative audio formats: (i) the original audio from the GIF source video and (ii) a spoken audio description. We recommend that social media platforms automatically include alt text descriptions for popular GIFs (as Twitter has begun to do), and content producers create audio descriptions to ensure everyone has a rich and emotive experience with GIFs online.

Audio descriptions make videos accessible to those who cannot see them by describing visual content in audio. Producing audio descriptions is challenging due to the synchronous nature of the audio description that must fit into gaps of other video content. An experienced audio description author will produce content that fits narration necessary to understand, enjoy, or experience the video content into the time available. This can be especially tricky for novices to do well. In this paper, we introduce a tool, Rescribe, that helps authors create and refine their audio descriptions. Using Rescribe, authors first create a draft of all the content they would like to include in the audio description. Rescribe then uses a dynamic programming approach to optimize between the length of the audio description, available automatic shortening approaches, and source track lengthening approaches. Authors can iteratively visualize and refine the audio descriptions produced by Rescribe, working in concert with the tool. We evaluate the effectiveness of Rescribe through interviews with blind and visually impaired audio description users who preferred Rescribe-edited descriptions to extended descriptions. In addition, we invite novice users to create audio descriptions with Rescribe and another tool, finding that users produce audio descriptions with fewer placement errors using Rescribe.

This paper reports on the work being done towards achieving an immersive and accessible reconstruction of Historical and Cultural Heritage, focusing on Theaters of the Early Modern as use case. In particular, the paper presents and discusses potential possibilities to enable: 1) the acoustical reconstruction of the virtual (lost) environments – beyond the graphical reconstruction for the buildings, elements and performances; 2) effective interaction features and navigation within the virtual environment (e.g. by means of adaptive interfaces, guiding methods, insertion of Point of Interest); and 3) accessible experiences, by means of an innovative and personalized presentation modes for access services, like subtitles and audio description. For most of these aspects and features, proof of concept implementation are provided, and opportunities for future work are outlined. With an effective combination of all these contributions, the goal is to bring back valuable (both tangible and intangible) Cultural Heritage from the past, providing high benefits in relevant sectors like Culture, Tourism and Education.

While board games are a popular social activity, their reliance on visual information can create accessibility problems for blind and visually impaired players. Because some players cannot easily read cards or locate pieces, they may be at a disadvantage or may be unable to play a game without sighted help. We present Game Changer, an augmented workspace that provides both audio descriptions and tactile additions to make the state of the board game accessible to blind and visually impaired players. In this paper, we describe the design of Game Changer and present findings from a user study in which seven blind participants used Game Changer to play against a sighted partner. Most players stated the game was more accessible with the additions from Game Changer and felt that Game Changer could be used to augment other games.

Audio description (AD) is an accessibility resource designed to improve access for blind or low vision individuals by describing images, narrating actions and visual elements, such as scene details, some aspects of the character (eg, age, gender, clothing), among others. However, in general, an AD is only generated in sections of the video that do not contain dialogue. This is done to prevent any overlap with the dialogue in the video, which may hinder the user's understanding rather than helping it. Thus, one of the first steps in the AD generation process is to identify the speechless intervals, which are candidates to receive AD. In this work, we present a solution for automatic identification of speechless intervals in digital videos using Convolutional Neural Networks (CNNs). Our proposal is to automate this step in the AD generation process, reducing the time and effort involved for generating AD. Another alternative would be to integrate it into an automatic or semi-automatic audio description generation system. The results shows that, considering a minimum confidence level of 0.5 for the output of the classification model, the solution obtained a balanced average accuracy of 93% to identify speechless segments considering all the videos tested.

Audio description, an accessibility service used by blind or visually impaired individuals, provides spoken descriptions of visual content. This accommodation allows those with low or no vision the ability to access information that sighted people obtain visually. A method for deploying pre-recorded audio description in a live musical theater environment is presented here. This method uses a reference audio recording and an online time warping algorithm to automatically align audio description with repeated live performances of a theatrical production. This system is used in two evaluation experiments that show the method successfully aligns multiple recordings of works of musical theater in order to automatically trigger pre-recorded, descriptive audio in real time.

Audio description is an accessibility service used by blind or visually impaired individuals. Often accompanying movies, television shows, and other visual art forms, the service provides spoken descriptions of visual content, allowing people with vision loss the ability to access information that sighted people obtain visually. At live theatrical events, audio description provides spoken descriptions of scenes, characters, props, and other visual elements that those with vision loss may otherwise find inaccessible. In this paper we present a method for deploying pre-recorded audio description in a live musical theater environment. This method uses a reference recording and an online time warping algorithm to align audio description with live performances, including a process for handling unexpected interruptions. A software implementation that is integrated into an existing theatrical workflow is also described. This system is used in two evaluation experiments that show the method successfully aligns multiple recordings of works of musical theater in order to automatically trigger pre-recorded, descriptive audio in real time.

A lot of work has been focused around enabling accurately synchronised companion screen experiences. The challenge has been to ensure that the delays between the presentation of programme content to the TV and the delivery of the relevant companion screen content to a mobile device are kept to a minimum. This is mainly driven by the need to ensure that the integrity of the editorial design of companion screen experiences can be maintained at the users' end. This paper presents a 32-participant study which sought to explore the impact of delays between the presentation of programmes on a TV and the presentation of companion content on a Tablet. Three types of experiences: 1) video-to-slideshow using Factual content, 2) video-to-alt-video using Sports content, and 3) video-to-AD (audio description) using Drama content; were tested across eight levels of delays. Participant responses suggest different factors influence their evaluation of the different types of experiences tested.

The paper investigates the potential benefits of bringing together Internet of Things and deep learning techniques toward the development of assistive technologies for users with visual disabilities. We propose a computer vision system designed to classify objects in the user's surroundings and to provide its user with an audio description of the detected things. The solution exploits a wearable vision sensor, which is mounted on the user's glasses and controlled by a single board computer running Google TensorFlow framework. With this software layer, the aid may support users in specific environments, e.g., museums. Finally, experiments show promising results in the context of image classification.

"Gamification" research has evolved and grown dramatically in recent years, gaining popularity across disciplines. While such efforts have generated headway in many respects, and in various directions, from conceptual understandings to user studies, the field could benefit from more work focused upon use in research methodologies at the nexus of practice and theory. This paper, in turn, reflects upon such an experiment aimed at the design and application of gamification techniques within a typical technical-communication context. In this case, subject matter experts within the National Park Service were being asked to improve accessibility of their site brochures by audio describing them. During this training, they were given an overview of audio description, as a process, as well as introduced to a prototype web tool and then asked to use that tool to create the description for their site brochure. Unlike previous training exercises with other parks in this project, though, this group also was organized by sites into a tournament bracket, in which pairs of parks competed against each other in exercises designed to create comparable audio description. The winner of each round, as determined by an independent panel of judges, advanced to the next round, spurred by the promise of fun Hawaiian-themed prizes at the end. This gamification strategy appeared to generate more data, and more research-focused data, than the previous training exercises we have offered, per user. It also apparently engaged many, in various evident ways. But it also seemed to disenfranchise some as well, who dropped out of this voluntary training, creating a mix of results, which will be outlined in this paper.

Digital and mobile technologies have become increasingly popular to support and improve the quality of experience during cultural visits. The portability of the device, the daily adaptation of most people to its usage, the easy access to information and the opportunity of interactive augmented reality have been key factors of this popularity. We believe that computer vision may help to improve such quality of experience, by making the mobile device smarter and capable of inferring the visitor interests directly from his/her behavior, so triggering the delivery of the appropriate information at the right time without any specific user actions. At MICC University of Florence, we have developed two prototypes of smart audio guides, respectively for indoor and outdoor cultural visits, that exploit the availability of multi-core CPUs and GPUs on mobile devices and computer vision to feed information according to the interests of the visitor, in a non intrusive and natural way. In the first one [Seidenari et al. 2017], the YOLO network [Redmon et al. 2016] is used to distinguish between artworks and people in the camera view. If an artwork is detected, it predicts a specific artwork label. The artwork's description is hence given in audio in the visitor's language. In the second one, the GPS coordinates are used to search Google Places and obtain the interest points closeby. To determine what landmark the visitor is actually looking at, the actual view of the camera is matched against the Google Street Map database using SIFT features. Matched views are classified as either artwork or background and for artworks, descriptions are obtained from Wikipedia. Both prototypes were conceived as a smart mate for visits in museums and outdoor sites or cities of art, respectively. In both prototypes, voice activity detection provides hints about what is happening in the surrounding context of the visitor and triggers the audio description only when the visitor is not talking with the accompanying persons. They were developed on NVIDIA Jetson TK1 and deployed on a NVIDIA Shield K1 Tablet, run in real time and were tested in real contexts in a musum and the city of Florence.

Audiovisual translation covers several techniques such as subtitling, dubbing, voice-over or audio description (AD), which can be applied to various multimedia contexts as is the case of Television. These translation techniques have an important role as enablers of the access to television content by people with special needs, namely persons with visual impairments, illiterate people and persons who do not comprehend another spoken language besides the native one. Currently, the production methods of audiovisual translation techniques are expensive and complex, involving several professionals and procedures. However, the current technological convergence paradigm inherent to Television provides great opportunities to enhance these techniques, leveraging the number of translated TV content and bringing relevant socio-economic impacts. In this context, this paper aims to address and discuss two inclusive approaches for audiovisual translation production (collaborative/spontaneous AD and automatic reading of subtitles) in an Interactive Television (iTV) context, benefiting a wide range of viewers' profiles. Based on a focus group with experts on the iTV field, we found that the proposed approaches are relevant and applicable in functional and technological terms.

Technological advances in digital cinema have allowed people to encounter experiences that awaken their imagination and expose them to other realities. Experiencing these realties can be more difficult for the blind or visually impaired, however. In our cinema rooms, visual impairments create barriers that can restrict a person's access to critical information. Therefore, we propose a solution that attempts to eliminate these barriers by using a computational system that is able to automatically generate and distribute accessible audio tracks that describe the digital cinema experience. Using mobile devices to provide the content, visually impaired participants were given the opportunity to partake in an experiment to confirm or reject the viability of the solution presented in this article. The results of the experiment demonstrated that our computational system may be a feasible solution.

This paper describes the process we undertook to address a significant technical communication issue embedded within traditional information-distribution procedures in the U.S. National Park Service: The bureau's nationally circulated brochures, as print artifacts, are inaccessible to people who are blind, visually impaired and print dyslexic and also not optimized for people who prefer audio communication. This report includes findings from the second year of a three-year project, funded by a grant from the National Park Service. Our first year on this project was spent getting to know the frameworks and foundations of the 342 brochures currently circulating in the country, through a deep reading of samples and a larger content analysis of the brochure structures and content genres. The second year of this project, which is the focus of this paper and poster presentation at SIGDOC 2016, revolves around the design-based research prototyping experiments we conducted as a way to create and refine a web tool to support and produce audio description, particularly in relation to the remediation of NPS brochures to audio-based mobile apps. This process led to prototyping experiments within the NPS that revealed a significant gap between practitioner knowledge and best practices. Those best practices also were found to be lacking in many areas, such as guidance for the practitioners to audio describe a NPS staple, the site map. While audio description, like open/closed captioning, might be considered a relatively straightforward process to those uninitiated in the activity, these practitioner experiments revealed many of the nuances and complexities of audio description that make it a difficult translation exercise. They also led to the generation of important scholarly research questions, such as "In what ways could one audio describe a map?"

Unigrid" design specifications created by Massimo Vignelli have provided the standards for the layout of paper brochures at U.S. National Park Service sites for more than three decades. These brochures offer visitors a familiar analog presentation of visual information, blending text, photographs, maps, and illustrations. These materials, however, are not accessible to people who are blind, have low vision, or a print disability. The National Park Service for decades has been challenged -- by requirements and principle -- to offer alternate formats that provide equivalent experiences and information of these print materials. In other words, people who are blind or visually impaired should have access to a "brochure" experience, too. This exploratory study, funded by the National Park Service, takes a new approach to this long-term problem by conducting a content analysis of current Unigrid brochures to determine their fundamental components, found in practice. This components-based approach is intended to provide clear pathways for cross-modal translation of the printed material into audio-described media, which then, can be efficiently distributed via mobile apps, as an extension of these original components.

The Assistive Technology Centers of Instituto Federal de Educacao, Ciencia e Tecnologia of Rio Grande do Sul and Ceara in Brazil have been developing a range of low cost assistive technology (AT) products. These products are developed to have exactly the same functionality and performance as commercial ones, but with much lower price. Some examples of low cost AT products developed include: Braille display, thermoforming tool for production of tactile materials, cane tip that detects puddles, digital game that helps the visually impaired in rehabilitation involving mobility and orientation using 3D sound effects, Android application for blind people to access Twitter, different types of switches that use balls to perform computer mouse functions (small ball to be used by hand and big ball to be used by foot), sip-and-puff switch, capacitive touch switch, a type of mouse that uses a set of buttons to perform regular mouse functions and movements, another type of mouse that uses buttons and rolling bars for moving around the screen, computer keyboard key-guard, home automation system with web interface, augmentative and alternative communication devices, and a proposition for digitally creating video audio description. The range of technologies developed continues to grow due to increasing demand.

This paper discusses how human emotion could be quantified using contextual and physiological information that has been gathered from a range of sensors, and how this data could then be used to automatically generate music playlists. The work is very much in progress and this paper details what has been done so far and plans for experiments and feature mapping to validate the concept in real-world scenarios. We begin by discussing existing affective systems that automatically generate playlists based on human emotion. We then consider the current work in audio description analysis. A system is proposed that measures human emotion based on contextual and physiological data using a range of sensors. The sensors discussed to invoke such contextual characteristics range from temperature and light to EDA (electro dermal activity) and ECG (electrocardiogram). The concluding section describes the progress achieved so far, which includes defining datasets using a conceptual design, microprocessor electronics and data acquisition using Matlab. Lastly, there is brief discussion of future plans to develop this research.

In this paper we compare levels of presence between sighted audiences and those with a visual impairment accessing audiovisual media via Audio Description (AD). AD is a verbal commentary conveying visual information. Anecdotal evidence suggests AD is at its best when the user is unaware of it. The strength of this 'perceptual illusion of non-mediation' [8] may therefore be a useful measure of effective AD. This paper discusses the results of two studies comparing the impact of sound effects and verbal pictures on dimensions of presence (spatial presence, ecological validity, engagement and negative effects) in people with varying levels of sight for audio-only and audiovisual media. Findings suggest AD influences the various dimensions of presence in different ways. Spatial presence and ecological validity are enhanced when media form is reflected in the AD script, while engagement is affected by content comprehension. Most surprisingly, levels of presence for those with visual impairments can exceed those of the sighted audience when an audiovisual stimulus is accessed via AD of an appropriate style. Possible reasons for this are discussed.

Our DVD Player was designed based on the needs identified in our prior in situ study with blind and visually impaired individuals. In this study we identified three major issues that would significantly improve the audio-visual content with audio description «AD). Beside the regular accessible functions found in existing video player, we added three specialized functions to enhance the user experience. Thus, the player provides context information on the content, basic and an augmented quantity of AD and recall functions on key information (scene identification, actions in the scene and the actors in the scene). We propose to demonstrate the CRIM DVDPlayer and to discuss the results of our seven months usability study.

The paper explores the effects of adding audio description to an educational film on children's learning behaviour, manifested by a visual recognition task. We hypothesize that the multimodal educational setting, consisting of both verbal (film dialogue and audio description) and non-verbal (motion pictures) representations of knowledge, fosters knowledge acquisition as it provides information via multiple channels, which in turn strengthens memory retrieval. In the study we employ eye tracking methodology to examine the recognition of previously seen film material, testing whether audio description promotes recognition- rather than elimination-based decision-making in the visual recognition task. The analysis of first fixation duration and first run fixation count measures in the experimental and control groups partially confirmed our hypotheses. Children in the experimental group generally looked longer at the scenes they had seen, which supports the hypothesis that their decision was based on recognition, whereas children in the control group had longer fixations on scenes they were unfamiliar with, suggesting a decision based on elimination.

Audio description (AD) has become a cultural revolution for the visually impaired; however, the range of AD beneficiaries can be much broader. We claim that AD is useful for guiding children's attention. The paper presents an eye-tracking study testing the usefulness of AD in selective attention to described elements of a video scene. Forty-four children watched 2 clips from an educational animation series while their eye movements were recorded. Average fixation duration, fixation count, and saccade amplitude served as primary dependent variables. The results confirmed that AD guides children's attention towards described objects resulting e. g., in more fixations on specific regions of interest. We also evaluated eye movement patterns in terms of switching between focal and ambient processing. We postulate that audio description could complement regular teaching tools for guiding and focusing children's attention, especially when new concepts are introduced.

Microsoft's Kinect 3-D motion sensor is a low cost 3D camera that provides color and depth information of indoor environments. In this demonstration, the functionality of this fun-only camera accompanied by an iPad's tangible interface is targeted to the benefit of the visually impaired. A computer-vision-based framework for real time objects localization and for their audio description is introduced. Firstly, objects are extracted from the scene and recognized using feature descriptors and machine-learning. Secondly, the recognized objects are labeled by instruments sounds, whereas their position in 3D space is described by virtual space sources of sound. As a result, the scene can be heard and explored while finger-triggering the sounds within the iPad, on which a top-view of the objects is mapped. This enables blindfolded users to build a mental occupancy grid of the environment. The approach presented here brings the promise of efficient assistance and could be adapted as an electronic travel aid for the visually-impaired in the near future.

The Digital Terrestrial Television (DTT) switchover may leverage the implementation of a variety of features and services that will allow users to take on an interactive role. However, among the future clients of DTT are a significant number of elderly and visually impaired people who can not entirely benefit from the TV content or possible interactive services. In this way, and considering that interactive television (iTV) services can contribute towards improving life quality of this type of users, it is important to apply accessibility, usability and design-oriented guidelines for the development of accessible iTV systems.In this context, this paper introduces a research related to universal design issues on the iTV scope. Beginning with the identification of the difficulties and needs of visually impaired users as consumers of television contents, this research aims to propose a set of requirements to develop an interactive audio description service that meets the needs of visually impaired users, following a universal design approach.

We conducted a series of experiments to assess the feasibility of synthesized narrations to describe online videos. To reduce the cultural bias, we included adult blind or low-vision participants from Japan and the U.S. in the main study. Our research also includes a follow-up study we conducted in Japan to assess the effectiveness of synthesized video descriptions in realistic situations. The results showed that synthesized video descriptions were generally accepted in both countries. We also found that appropriate technology support allowed a novice describer to make effective video descriptions. Based on these results, we discuss the implications for developing a technology platform for describing online videos.

Videodescription (VD) or audio description is added to the sound track of audio-visual contents to make media such as film and television accessible to individuals with visual impairment. VD translates the relevant visual information into auditory information. In our previous users' testing, we found that the need of VD could be quite different depending on the visual disabilities of the participants. In order to better identify those differences, we conducted a study with ten legally blind individuals (with and without residual vision) to observe the type, quantity and frequency of the information needed by them. We learned that the degree of residual vision and the complexity of the content have a significant impact of the required level of VD. This suggests that a tool to render VD should offer a basic level of information, allow enough flexibility to provide more VD if needed, and answer on the fly demands for specific information. These specifications were implanted into an accessible video player.

Most online videos are inaccessible for blind and visually impaired people. Those who are illiterate or barely literate also face difficulties with many videos. An audio description (AD) can help, but is rarely provided for actual online videos because of the time and trouble required. The technology platform we are developing uses a speech synthesis technology to add ADs to online videos at minimal cost. We are developing an authoring tool for AD scripts, a Web browser for playing the described videos, and a text-based format to handle the AD scripts.

Providing blind and visually impaired people with the descriptions of key visual elements can greatly improve the accessibility of video, film and television. This project presents a Website platform for rendering videodescription (VD) using an adapted player. Our goal is to test the usability of an accessible player that provides end-users with various levels of VD, on-demand. This paper summarizes the user evaluations covering 1) the usability of the player and its controls, and 2) the quality and quantity of the VD selected. The complete results of these evaluations, including the accessibility of the Website, will be presented in the poster. Final results show that 90% of the participants agreed on the relevancy of a multi-level VD player. All of them rated the player easy to use. Some improvements were also identified. We found that there is a great need to provide blind and visually impaired people with more flexible tool to access rich media content.

The work presented here combines the analysis of a film's audiovisual features with the analysis of an accompanying audio description. Specifically, we describe a technique for semantic-based indexing of feature films that associates character names with meaningful events. The technique fuses the results of event detection based on audiovisual features with the inferred on-screen presence of characters, based on an analysis of an audio description script. In an evaluation with 215 events from 11 films, the technique performed the character detection task with Precision = 93% and Recall = 71%. We then go on to show how novel access modes to film content are enabled by our analysis. The specific examples illustrated include video retrieval via a combination of event-type and character name and our first steps towards visualization of narrative and character interplay based on characters occurrence and co-occurrence in events.