Blind in a virtual world -color, spatial perception and navigation for the blind using auditory sensory substitution in virtual environments

Abstract

 

Purpose

 

Graphical virtual environments are becoming a central part of our lives, yet are far from accessible to over 280 million blind and visually-impaired worldwide as most of the input they offer is visual. This is especially unfortunate as such virtual environments hold great potential for them, e.g., for safely learning an environment virtually before visiting it in the real world, for gaming and for learning. Visual-to-audio Sensory Substitution Devices (SSDs) can potentially increase their accessibility in a generic fashion by sonifying the on-screen content. However, whether blind users will be able to use this information to perceive and interact successfully within virtual environments, if they could feel immersed within them, and the extent of training required are currently unclear.

 

Methods

 

We tested this using the EyeCane SSD, which conveys single-point depth, and the EyeMusic SSD, which conveys whole-scene shape and color information, to perform virtual tasks otherwise not possible without vision such as navigating virtually down corridors and through rooms, locating doors and differentiating between them based on their features (shape, color) and surroundings (trees, houses) and finally navigating to them. These tasks were performed by a group of blind users who were proficient with these SSDs and a group of blind and blindfolded-sighted users which was naïve to their use.

 

Results

 

We show that these tasks can indeed be accomplished by blind users with previous experience using these SSDs (success in all tasks >90%) and show similar success by naïve users who received only 20 minutes of training with the devices (success in all tasks >85%). We additionally show the users felt immersed within these environments (1-5 scale: 3.8+-0.6) and enjoyed the experience (1-5 scale: 3.6+-0.6).

 

Conclusions

 

These results provide a proof-of-concept for the use of SSDs to increase the accessibility of virtual environments, and additionally show that congenitally blind users can indeed perceive virtual environments and interact with them.

 

 

Graphical screenshot of the virtual environment from 3rd (left) and 1st (right) person views. Users experienced the environments only via the substituted auditory cues which included the visual information, and received no direct visual input. In this task users had to find a door which was between two trees.

 

Keywords: 717 space and scene perception • 713 shape, form, contour, object perception • 471 color vision