Hearing Through Your Fingers: Device that Converts Speech to Touch Enables the Hearing-Impaired to Hear Through Their Fingertips

A novel study published in Restorative Neurology and Neuroscience provides the first evidence that a simple and inexpensive non-invasive speech-to-touch sensory substitution device has the potential to improve hearing in hearing-impaired cochlear implant patients, as well as individuals with normal hearing, to better discern speech in various situations like learning a second language or trying to deal with the “cocktail party effect.” The device can provide immediate multisensory enhancement without any training.

“Despite recent advancements in hearing aid and cochlear implants, the most widespread surgical implant to restore audition, hearing-impaired users still encounter significant practical and social challenges with or without aids,” explained lead investigator Amir Amedi, PhD, Department of Medical Neurobiology, Institute for Medical Research Israel-Canada, Faculty of Medicine, Hebrew University of Jerusalem, Hadassah Ein-Kerem, and The Cognitive Science Program, The Hebrew University of Jerusalem, Jerusalem, Israel. “In particular, they all struggle with understanding speech in challenging, noisy acoustic environments, especially in presence of a competing speaker.”

The number of sensory deprived patients and auditory deprived patients is expected to rise so it is crucial to develop efficient techniques for auditory recovery designed to convey the missing information to patients. “We live in a world that is becoming steadily more multisensory and we really need to understand the mechanisms underlying multisensory perception and integration. Providing relevant information using the sense of touch can significantly help hearing,” commented Dr. Amedi.

In this current proof-of-concept study investigators hypothesized that they would be able to improve speech understanding under challenging conditions by exploiting the ability of the brain to integrate information coming simultaneously from different senses. They designed a minimalistic auditory-to-tactile sensory substitution device (SSD) that transforms low-frequency speech signals into tactile vibrations delivered on two fingertips. They asked a group of non-native English speakers to repeat a series of sentences, which was degraded by embedding them in speech-like noise.

As expected, when participants could rely only on audition, their understanding of such sentences was poor. Crucially, however, their sentence understanding significantly improved when they paired the degraded speech signal with complementary vibration delivered on the participants fingertips. The vibration conveyed a specific set of frequencies known as fundamental frequencies that characterize speech signals.

Caption: A) Vibrating interface of the Vibrating Auditory Stimulator; B) Matlab GUI for stimuli presentation and control; C) Speech Reception Threshold (SRT) values obtained for auditory and auditory-tactile speech in noise stimulation at the group level; and D) in individual subjects [subject 6 showed an improvement from 0.3 to –3.0 SRT(dB)] 

Credit: IOS Press

The reported improvement at the group level was 6 dB – a major difference considering that an increase of 10 dB represents a doubling of the perceived loudness. These results are especially relevant when compared to earlier SSD studies showing effects in behavior only after demanding cognitive training.

“Our results carry important implications for further research, as well as possible clinical and practical solutions,” noted co-author Tomasz Wolak, PhD Eng, Head of the Bioimaging Research Center, Institute of Physiology and Pathology of Hearing, World Hearing Center, Warsaw, Poland. “The ability to ‘hear through one’s fingers can significantly help hearing. Our approach suggests that multisensory stimulations providing the same type of information (in this case spoken language conveyed through touch in addition to hearing) should be processed in the same brain region (in this case spoken language centers), ultimately then predicting that multisensory stimulations (both sounds and touch) should enhance perception.”

According to lead author Katarzyna Cieśla, PhD, World Hearing Center, Warsaw, and Hebrew University of Jerusalem, “The most compelling aspect of our study is the fact that learning to use such speech-to-touch sensory substitution device did not require any training. We believe it can also serve as an aid for the elderly population, which finds it challenging to follow extensive training regimes. This might be the first study showing such immediate relevant enhancement of a sensory substitution device and suggests the brain is much more multisensory than the common wisdom.”

“This study is a major step forward to introduce multisensory plasticity of the brain as an innovative paradigm to maximize the potential of patients to compensate for their sensory loss,” commented Bernhard Sabel, PhD, Editor-in-Chief of Restorative Neurology and Neuroscience.

Next the team plans to further improve the device and training regimes in order to reach the goal of 10 dB enhancement and to test for human brain mechanisms using an MRI-compatible version of the device in various populations (both hearing and hearing-impaired people).

Contacts: