How AI helps some people with disabilities communicate
Aubrie Lee can often tell by the look on someone’s face if they’re pretending to understand her.
A furrowed brow stuck in concentration purgatory. Polite laughter that doesn’t really match how funny her punchline was.
Apparently I had an obvious tell:
Lee was born with a rare form of muscular dystrophy. She uses a wheelchair, has a speech impairment and can’t really smile.
The 31-year-old Lee is a brand manager at Google, where she helped name Project Relate, an artificial intelligence-enabled app that make conversations like these easier:
It may have mistaken “test” for “chest,” but Relate’s computerized voice translated the word I was hung up on: “Riddle.”
Project Relate is one piece of a suite of artificial intelligence tools in what’s known among people with nonstandard speech as AAC — augmentative and alternative communication.
At the core of these breakthroughs is “deep learning.” In this case, it involves algorithms built on thousands of recordings of people with speech disabilities, then trained on an individual user’s voice pattern. It’s a different, older strain of AI than the large language models that power ChatGPT. But in the past few years, it’s become very good at decoding nonstandard speech.
“So, what the machines are doing is it becomes a professional listener,” said Jordan Green, a speech pathologist at Massachusetts General Hospital’s Institute of Health Professions who works with Google. “And then you can take somebody who is highly unintelligible, like maybe you or I can only understand 10% of what the person is saying. But the machine can learn to recognize with accuracy of 80% to 90%.”
Since being released in 2021, Project Relate’s beta app for Android phones has been downloaded by thousands of users. A host of smaller companies also offer similar products specifically tailored to people with serious speech impairments, like those resulting from cerebral palsy or ALS, also known as Lou Gehrig’s disease.
“About a year ago, we were asking individuals to record 1,000 phrases in order to acquaint our technology with their unique way of speaking,” said Sara Smolley, cofounder of the AAC AI startup Voiceitt. “Fast-forward to now, in a very short amount of time, we have decreased that initial training time for most users to about 200 phrases, and for some, almost none at all.”
The recent explosion in generative AI tools like ChatGPT and Dall-E have also provided a new avenue for people with major mobility disabilities to express themselves.
Amie Thornburg was diagnosed with ALS 25 years ago. It’s sapped much of her voice, and she can’t really move any muscles below her head.
Over Zoom, she showed me a digital image that looks like an eerie oil painting. In the foreground is a bright red poppy flower, with an ominous eyeball in the middle of the pistil.
Thornburg used the AI art tool Fotor to create the image with software that followed her eye movements to type the prompts. She calls the image “Always Watching.”
“You tend to get a lot of looks when you go out in a wheelchair,” Thornburg said.
Thornburg has printed a few of these images on canvas and sells them on Etsy. She said AI helps scratch her artistic itch. Before the diagnosis, she would create big merchandise displays in malls across the Midwest.
Her husband, Pat, said he’s been impressed with how technology has evolved to help Amie communicate. But if and when Amie loses her voice for good, he’s developed a pretty efficient way of understanding her without machines.
“If I do something out of sync, if I forget something, there’s just certain looks that tell me, ‘Hey, what are you doing?'”
He said that after 38 years of marriage, he doesn’t need AI to translate everything.