Think AI Can Perceive Emotion? Think Again.

Numerous MindBlog posts have presented the work and writing of Elizabeth Feldman Barrett (enter Barrett in the search box in the right column of this web page). Her book, "How Emotions Are Made," is the one I recommend when anyone asks me what I think is the best popular book on how our brains work.  Here I want to pass on her piece on AI and emotions in the Sat. May 18 Wall Street Journal. It collects together the various reasons that AI can not, and should not, be used for detecting our emotional state from our facial expressions or other body language.  Here is her text: 

Imagine that you are interviewing for a job. The interviewer asks a question that makes you think. While concentrating, you furrow your brow and your face forms a scowl. A camera in the room feeds your scowling face to an AI model, which determines that you’ve become angry. The interview team decides not to hire you because, in their view, you are too quick to anger. Well, if you weren’t angry during the interview, you probably would be now.

This scenario is less hypothetical than you might realize. So-called emotion AI systems already exist, and some are specifically designed for job interviews. Other emotion AI products try to create more empathic chatbots, build more precise medical treatment plans and detect confused students in classrooms. But there’s a catch: The best available scientific evidence indicates that there are no universal expressions of emotion.

In real life, angry people don’t commonly scowl. Studies show that in Western cultures, they scowl about 35% of the time, which is more than chance but not enough to be a universal expression of anger. The other 65% of the time, they move their faces in other meaningful ways. They might pout or frown. They might cry. They might laugh. They might sit quietly and plot their enemy’s demise. Even when Westerners do scowl, half the time it isn’t in anger. They scowl when they concentrate, when they enjoy a bad pun or when they have gas.

Similar findings hold true for every so-called universal facial expression of emotion. Frowning in sadness, smiling in happiness, widening your eyes in fear, wrinkling your nose in disgust and yes, scowling in anger, are stereotypes—common but oversimplified notions about emotional expressions.

Where did these stereotypes come from? You may be surprised to learn that they were not discovered by observing how people move their faces during episodes of emotion in real life. They originated in a book by Charles Darwin, “The Expression of the Emotions in Man and Animals,” which proposed that humans evolved certain facial movements from ancient animals. But Darwin didn’t conduct careful observations for these ideas as he had for his masterwork, “On the Origin of Species.” Instead, he came up with them by studying photographs of people whose faces were stimulated with electricity, then asked his colleagues if they agreed.

In 2019, the journal Psycho--logical Science in the Public Interest engaged five senior scientists, including me, to examine the scientific evidence for the idea that people express anger, sadness, fear, happiness, disgust and surprise in universal ways. We came from different fields—psychology, neuroscience, engineering and computer science—and began with opposing views. Yet, after reviewing more than a thousand papers during almost a hundred videoconferences, we reached a consensus: In the real world, an emotion like anger or sadness is a broad category full of variety. People express different emotions with the same facial movements and the same emotion with different facial movements. The variation is meaningfully tied to a person’s situation.

In short, we can’t train AI on stereotypes and expect the results to work in real life, no matter how big the data set or sophisticated the algorithm. Shortly after the paper was published, Microsoft retired the emotion AI features of their facial recognition software.

Other scientists have also demonstrated that faces are a poor indicator of a person’s emotional state. In a study published in the journal Psychological Science in 2008, scientists combined photographs of stereotypical but mismatched facial expressions and body poses, such as a scowling face attached to a body that’s holding a dirty diaper. Viewers asked to identify the emotion in each image typically chose what was implied by the body, not the face— in this case disgust, not anger. In a study published in the journal Science in 2012, the same lead scientist showed that winning and losing athletes, in the midst of their glory or defeat, make facial movements that are indistinguishable.

Nevertheless, these stereotypes are still widely assumed to be universal expressions of emotion. They’re in posters in U.S. preschools, spread through the media, designed into emojis and now enshrined in AI code. I recently asked two popular AIbased image generators, Midjourney and OpenAI’s DALL-E, to depict “an angry person.” I also asked two AI chatbots, OpenAI’s ChatGPT and Google’s Gemini, how to tell if a person is angry. The results were filled with scowls, furrowed brows, tense jaws and clenched teeth.

Even AI systems that appear to sidestep emotion stereotypes may still apply them in stealth. A 2021 study in the journal Nature trained an AI model with thousands of video clips from the internet and tested it on millions more. The authors concluded that 16 facial expressions are made worldwide in certain social contexts. Yet the trainers who labeled the clips with emotion words were all English--speakers from a single country, India, so they effectively transmitted cultural stereotypes to a machine. Plus, there was no way to objectively confirm what the strangers in the videos were actually feeling at the time.

Clearly, large data sets alone cannot protect an AI system from applying preconceived assumptions about emotion. The European Union’s AI Act, passed in 2023, recognizes this reality by barring the use of emotion AI in policing, schools and workplaces.

So what is the path forward? If you encounter an emotion AI --product that purports to hire skilled job candidates, diagnose anxiety and depression, assess guilt or innocence in court, detect terrorists in airports or analyze a person’s emotional state for any other purpose, it pays to be skeptical. Here are three questions you can ask about any emotion AI product to probe the scientific approach behind it.

Is the AI model trained to account for the huge variation of real-world emotional life? Any individual may express an emotion like anger differently at different times and in different situations, depending on context. People also use the same movements to express different states, even nonemotional ones. AI models must be trained to reflect this variety.

Does the AI model distinguish between observing facial movements and inferring meaning from these movements? Muscle movements are measurable; inferences are guesses. If a system or its designers confuse description with inference, like considering a scowl to be an “anger expression” or even calling a facial movement a “facial expression,” that’s a red flag.

Given that faces by themselves don’t reveal emotion, does the AI model include abundant context? I don’t mean just a couple of signals, such as a person’s voice and heart rate. In real life, when you perceive someone else as emotional, your brain combines signals from your eyes, ears, nose, mouth, skin, and the internal systems of your body and draws on a lifetime of experience. An AI model would need much more of this information to make reasonable guesses about a person’s emotional state.