An augmented reality sign-reading assistant for users with reduced vision

1Dartmouth College 2Cardiff University

In PLOS ONE, 2019

Teaser
Left column: Original view of a user looking at two signs with normal vision (top row) and simulated visual impairment via Gaussian blur (bottom row). A white circle indicates the user's virtual cursor. Note that the signs are no longer readable with visual impairment. Middle column: Green AR signs are shown placed over the physical signs, indicating that the application identified two signs with high certainty. In real-time, these signs would be flashing at 3 Hz. Right column: Text is displayed with enhanced visibility after a user selects an AR sign.
Abstract Downloads Cite Related

Abstract

People typically rely heavily on visual information when finding their way to unfamiliar locations. For individuals with reduced vision, there are a variety of navigational tools available to assist with this task if needed. However, for wayfinding in unfamiliar indoor environments the applicability of existing tools is limited. One potential approach to assist with this task is to enhance visual information about the location and content of existing signage in the environment. With this aim, we developed a prototype software application, which runs on a consumer head-mounted augmented reality (AR) device, to assist visually impaired users with sign-reading. The sign-reading assistant identifies real-world text (e.g., signs and room numbers) on command, highlights the text location, converts it to high contrast AR lettering, and optionally reads the content aloud via text-to-speech. We assessed the usability of this application in a behavioral experiment. Participants with simulated visual impairment were asked to locate a particular office within a hallway, either with or without AR assistance (referred to as the AR group and control group, respectively). Subjective assessments indicated that participants in the AR group found the application helpful for this task, and an analysis of walking paths indicated that these participants took more direct routes compared to the control group. However, participants in the AR group also walked more slowly and took more time to complete the task than the control group. The results point to several specific future goals for usability and system performance in AR-based assistive tools.

Downloads

Cite

Jonathan Huang, Max Kinateder, Matt J. Dunn, Wojciech Jarosz, Xing-Dong Yang, Emily A. Cooper. An augmented reality sign-reading assistant for users with reduced vision. PLOS ONE, 14(1):1–9, January 2019. 
@article{huang19augmented,
    author    = {Huang, Jonathan and Kinateder, Max and Dunn, Matt J. and Jarosz, Wojciech and Yang, Xing-Dong and
                 Cooper, Emily A.},
    title     = {An augmented reality sign-reading assistant for users with reduced vision},
    journal   = {PLOS ONE},
    publisher = {Public Library of Science},
    year      = {2019},
    month     = jan,
    volume    = {14},
    number    = {1},
    pages     = {1--9},
    doi       = {10/gfznd2}
}