Hearing aids amplify everything – “blindly” – a group of scientists working on a system that could make brain-controlled hearing aids possible have written.
Their lead author Vishal Chaudhary* said, “Current hearing aids are good at amplifying sounds and voices, but they struggle with the classic ‘cocktail party problem’ – deciding which sounds matter to the listener.” Study published in Nature Neuroscience.
Focusing your attention on a single sound in a crowded room can take a lot of effort. “Listening is not just about whether the words are understood correctly,” Choudhary told DW. “Two people can both understand [what they’re saying]But a person may require far more mental effort to follow a conversation. “That can get tiring over time.”
As a result, many people stop using hearing aids when they need them most – in restaurants, cafeterias, parties, or busy social settings.
So, Chaudhary and his colleagues are trying to develop smart technology that can sense what a hearing aid user is hearing. They want to amplify that one sound or voice, while also reducing the volume of any other sound, voice or background noise.
And to do this, they designed a system that reads brain waves and uses artificial intelligence to interpret what the listener is hearing.
“Many hearing aids use beamforming, which amplifies sounds coming from a certain direction, usually in front of the listener. But real conversations are dynamic,” Choudhury said. “People turn their heads, change attention, or listen to someone even without looking directly at them.”
Brain-controlled listening from theory to practice
Under Nima Mesgarani, a professor and principal investigator at Columbia’s Zuckerman Institute, Choudhary and the team developed a real-time machine learning algorithm that could examine brain waves and identify conversations that four normal-hearing test participants were listening to.
Researchers call this the closed-loop auditory attention decoding (AAD) system. They wanted to find out whether AAD could be accurate and fast enough to selectively amplify the sound of an individual speaker while suppressing background sounds.
Although the idea sounds amazing, it is not ready for general use. For now, this concept relies on electrodes attached to the brain in a clinical setting.
In the study, four test patients were being brain monitored for epilepsy – so, they already had intracranial electrodes and it was convenient for the researchers.
Participants were presented with recordings of two competing sound sources, coming from small speakers located on the left and right.
The recording shows people of a different gender mix chatting about food, travel and exercise. Their words were described by researchers as “multi-talker babbling” and pedestrian noise.
Mesgarani and his colleagues discovered in 2012 that brain waves increase and decrease depending on a person’s attention. They exhibit peaks and troughs, the timing of which corresponds to sounds and silences in conversation.
In the new study, “the intensity of competitive interactions was adjusted dynamically in real time based on decoded brain signals,” Chowdhury explained. “Attended interactions became louder while competing interactions became quieter.”
According to the researchers, the system showed good results whether they directed participants to listen to a certain conversation and then asked them to switch their attention, as well as when participants chose the conversation.
Challenges and promises ahead for brain-controlled listening
Other experts in the field acknowledge the progress brought about by Choudhary and Mesgarani’s work.
Volker Hohmann, Professor of Auditory Signal Processing at the Cluster of Excellence Hearing4All at the University of Oldenburg, highlighted “the impressive accuracy with which auditory attention can be decoded from brain signals, especially when intracranial electrodes are used.”
However, in an email to DW, Hohmann said the system is not yet useful in everyday situations — a point the researchers also noted.
The acoustic conditions were “completely stable, the audience was not moving […] Acoustic communication in daily life is much more dynamic,” Hohmann said.
“The issue is that if one source is boosted, it becomes harder to hear the other, making it harder to focus on the quieter source and discern the change,” Bernhard Sieber, professor of audio information processing at the Technical University of Munich, said in an email to DW.
Sieber credited the US team for showing that the system can respond in real time when a listener engages attention, but said that more research is needed “to achieve reliable real-time attention decoding from skin-electrode signals” – that is, a less invasive way to monitor brain signals than intracranial electrodes.
This is where Choudhary sees promise for the system, and ultimately, it being incorporated into smart, wearable technology: “Imagine smart glasses or earbuds that work with your brain signals to know what conversation you’re listening to, help summarize important information, or even aid with memory and note taking in noisy environments,” he said.
*Vishal Chaudhary conceived the research as a PhD candidate under Nima Mesgarani at the Zuckerman Mind Brain Behavior Institute at Columbia University. Chaudhary is now a founding research scientist at an AI company in Seattle, US.
Edited by: Richard Connor