Hundreds of human faces flashed on the screen one by one, some of them staring, some with flat mouths, and some with closed eyes, raised corners of their mouths and wide mouths. Seeing these faces, you have to answer a simple question: is this person experiencing orgasm or labor?

In 2018, psychologist Rachael Jack and her colleagues recruited 80 people to do this test 1. The team from the University of Glasgow in the UK recruited these participants from the West and East Asia to study a long-standing hot question: can facial expressions really convey emotions?

Researchers have been asking subjects to read emotions from their faces for decades, including adults and children in different countries and even Aboriginal people in remote areas. In the 1960s and 1970s, the famous observational study conducted by American psychologist Paul Ekman found that people all over the world can accurately infer the emotion behind the facial expression, which shows that the expression of emotion is interlinked. 2,3.

This view has been largely unchallenged in a generation. However, a new generation of psychologists and cognitive scientists have raised questions after reviewing these data. Many researchers now believe that the actual situation is much more complex, and facial expressions have different meanings in different situations and cultures. For example, Jack’s research found that although Westerners and East Asians have similar understanding of facial expressions that express pain, they have different views on which expression expresses pleasure.

For Ekman’s conclusion that the face is a window for emotional expression, there are growing differences among researchers. However, this does not prevent commercial companies and the government from “paying” for his statement and applying it in a way that will change people’s destiny. For example, in many western judicial systems, reading the defendant’s emotions is part of a fair trial. Anthony Kennedy, chief justice of the U.S. Supreme Court, wrote in 1992 that it is necessary to “understand the hearts and minds of criminals”.

Ekman once designed a controversial training program for the Transportation Security Administration (TSA). The program was launched in 2007, and its core is to interpret emotions. The project is called “screening passengers by observation techniques (spot). Its main purpose is to train TSA personnel to monitor dozens of suspicious signs on passengers, which may reflect their anxiety, deception and fear. The project has been widely questioned by scientists, U.S. congressmen, the American Civil Liberties Union and other non-governmental organizations, who accuse this practice of being inaccurate and bringing racial prejudice.

The constant questioning has not stopped the top technology companies, which believe that emotions are easy to detect. Some of them have developed emotion recognition software. At present, these software are being tested or promoted. Their applications include assessing the matching degree between job seekers and positions, lie detection, making advertising more attractive, and detecting a range of diseases from dementia to depression.

The industry is valued at tens of billions of dollars. Technology giants such as Microsoft, IBM and Amazon, as well as some more professional enterprises (such as affectiva in Boston and neurodata lab in Miami) have launched algorithms to detect emotions through faces.

Researchers are still debating whether the human face can faithfully express and perceive emotions. Many experts also believe that it is too early to use computers to automate them, especially this technology has potential destructive power. AI now Institute, a research center at New York University, calls for banning the use of emotion recognition technology in sensitive situations, such as recruitment and law enforcement. 4.

Aleix Martinez, a researcher at Ohio State University who is engaged in relevant research, said that human facial expressions are difficult to interpret, even for humans themselves. He said that with this in mind and the current trend that everything can be automated, “we should be worried.”

Superficial view

The human face has 43 muscles, which can stretch, lift and twist to express dozens of different expressions. Although facial muscles can do many movements, scientists have always believed that specific expressions correspond to specific emotions.

Those who hold this view include Darwin. In 1859, he published his great book “origin of species” on field investigation, which can be regarded as a textbook on observation. His other less influential work, the expression of human and animal emotions (1872), is quite dogmatic.

Darwin noted that primate facial movements are somewhat similar to human expressions of emotions such as disgust and fear. He suggested that these expressions must have some adaptive function. For example, the expression related to disgust, such as pouting, wrinkling nose and squeezing eyes, may initially be to resist harmful pathogens. Only with the emergence of social behavior, these facial expressions began to play a role in communication.


Darwin’s discussion of emotion includes a large number of gestures, such as those of subjects who try to imitate pain. Source: Alamy

Ekman’s first cross-cultural field studies in the 1960s supported Darwin’s hypothesis. He has studied human expressions and perceptions of six key emotions (happiness, sadness, anger, fear, surprise and disgust) around the world, including even a remote tribe in New Guinea. 2,3.

Ekman told nature that he chose these six emotions out of practical considerations. He said that some emotions, such as shame and guilt, have no explicit expression. “The six emotions I am concerned about are expressive, that is, they can be used as research objects.”.

Ekman believes that these early studies support the expression universalism derived from Darwin’s theory of evolution. Later studies have proved that some facial expressions have adaptive advantages 5.

Lisa Feldman Barrett, a psychologist at Northeastern University in Boston, said: “for a long time, people thought facial expression was a compulsive action.” In other words, our faces cannot hide our emotions. However, an obvious flaw in this hypothesis is that people can actually fake emotions, or they can let emotions not be written on their faces. Scholars of Ekman school also admit that there is no so-called “gold standard” for the expression of each emotion.

More and more researchers have proposed that the range of expression corresponding to emotion is so large that the concept of the gold standard almost falls apart. They supported this idea with a large review 6. A few years ago, the editors of the journal Psychological Science in the public interest invited some mutually exclusive authors to form an expert group to complete this review.

Barrett, who led the cooperation, said: “we try our best to discard prejudices.” They did not set up assumptions in advance, but directly started from the data. “When there is disagreement, we look for new evidence,” she said Finally, they read about 1000 papers. After two and a half years of research, they came to an obvious conclusion: there is no evidence, or very little evidence, that people can infer someone’s emotional state from various facial movements.



The emotion that the face can reflect is very limited. The complete picture is shown below. Source: Lance king/hector vivas/ronaldo schemidt/kevin winter/getty

The researchers even cited studies that showed that facial movements had nothing to do with internal emotions. Carlos crivelli, a psychologist at the University of de Montfort in the United Kingdom, once studied the residents of the trobrien islands in Papua New Guinea. He found no evidence to support Ekman’s view. Crivelli’s conclusion is that inferring the internal psychological state from the external performance is like weighing with a ruler.

Another reason for the lack of evidence to prove the universality of expression is that faces only provide part of the information. Other information, such as body movements, personality, tone of voice and facial changes, also play an important role in our identification and expression of emotions. Just as emotional changes will affect blood flow, blood flow will affect the face. Martinez and colleagues found that people were able to detect the relationship between facial changes and mood7. Visual signals such as background can also provide clues to identify emotional states 8.



Clockwise from the top left: basketball player Zion Williamson celebrates the successful dunk; Mexican fans celebrate the World Cup group stage promotion; Singer Adele won the 2012 Grammy Award; Justin Bieber’s fans wept at a concert in Mexico City.

Complex emotions

Other researchers pointed out that the counterattack to Ekman’s conclusion was overdone. Ekman himself thought so. In 2014, in his response to Barrett’s criticism, he pointed out that there were a large number of studies supporting his previous conclusions, including studies that proved that the face would spontaneously make expressions. Other studies have found links between facial expressions and brain and body states. In response, he said that these studies showed that facial expressions not only reflected people’s emotions, but also reflected the patterns of neurophysiological activities (see He said his views had not changed.

According to Jessica Tracy, a psychologist at the University of British Columbia in Canada, the evidence given by those who think Ekman’s expression universality theory is wrong is just a small cluster of counterexamples, and they are exaggerating.

She believes that even if different groups or cultures have a slight deviation in their understanding of anger expression, the whole theory cannot be overthrown. Most people knew it was an angry face at first sight, and she cited an analysis of 100 Studies9. “There is a lot of other evidence that most people in most cultures around the world think this expression is universal,” she said

Tracy and three other psychologists think that 10. Barrett said in the literature review that they rigidly correspond six emotions to facial movements one by one. This interpretation is a bit exaggerated. One of the authors, disa Sauter of the University of Amsterdam, said: “I don’t think there are other researchers in the field of emotional science who agree with her.”

Sauter and Tracy argue that interpreting facial expressions requires a more complex classification of emotions. Researchers should not regard happiness as a single emotion, but should continue to subdivide it; Happiness also includes happiness, pleasure, pity, pride and so on. The expressions of these emotions may differ or overlap.

Some studies use computers to generate random facial expressions. In a study conducted by Rachael Jack in 2018, participants need to point out the degree to which each face conforms to their definition of pain or orgasm. Source: C. Chen et al./pnas (CC by 4.0)

At the heart of this controversy is the definition of salience. In one study, participants were asked to choose one of six emotion tags to describe the face they saw. Some researchers may think that if the probability of an expression being selected is greater than 20%, it indicates that the expression is more universal.

Others think the 20% standard is too loose. Jack thinks Ekman’s threshold is too low. She read Ekman’s early papers when she was a doctoral student. She said, “I always go to my tutor and show him these charts in the 1960s and 1970s. Each chart has a huge difference in cultural understanding. Up to now, there is still no data to prove that the recognition of emotion is universal.”

Even without considering significance, researchers have to face the problem of Subjectivity: many studies need to label emotions in advance so that they can be compared after the experiment. Therefore, Barrett, Jack and other scholars want to study emotions in a more objective way. Barrett is studying physiological indicators, which she hopes to use to describe anger, fear and pleasure.

Jack uses the expression randomly generated by the computer to replace the facial photos, avoiding being limited to the six most common emotions. Other researchers asked participants to classify faces themselves, or asked participants from different cultures to mark photos in their native languages.

Silicon based emotion

Software companies avoid free association of algorithms. Generally speaking, the artificial intelligence algorithm for emotion recognition needs to learn millions of face images and hundreds of hours of video – each emotion is labeled, and then learn patterns from these materials. Affectiva said that the company has trained the software with more than 7million faces from 87 countries. At present, its emotion recognition accuracy has reached 90%.

The company declined to disclose the scientific basis behind the algorithm. Neurodata lab is aware of the differences in facial emotional expression, but points out: “if someone is experiencing a certain emotion, the probability of some facial expressions is higher than the random probability.” The company’s algorithm uses this rule. Researchers with different opinions, regardless of which side they stand, are skeptical of such software, whether they are worried about the data used in the training algorithm or think that there is no final conclusion in this field.

Ekman said he had directly challenged these companies’ claims. He wrote to several companies, but refused to disclose their names, saying only that “they are all the largest software companies in the world”, and asked them for evidence to prove the effectiveness of their automation technology, but received no reply. “In my opinion, there is no evidence to support their theory,” he said

Martinez compromised that automated emotion recognition might represent the average emotional response of a group. Affectiva has sold software to marketing organizations and certain brands to help them predict the response of specific consumers to a product or marketing method.

Even if the software goes wrong, it will not have much impact. At most, the effect of advertising is not as good as expected. However, the application of some algorithms may change people’s fate, such as interview and border inspection. Last year, Hungary, Latvia and Greece tried a pre screening system for passengers to detect lies by analyzing facial microexpressions.

Different research methods are needed to quell the emotion expression debate. Barrett is often invited to present her research to technology companies and recently went to Microsoft. She believes that researchers should follow the practice of Darwin when he wrote the origin of species: “observe, observe, and observe again.” Observe how people convey information through their faces and bodies in real life, not just in the laboratory. Then use the machine to record and analyze the images from real life.

Barrett believes that more data and analysis techniques, rather than reviewing old data and experiments, can help researchers acquire new knowledge. Many technology companies are eager to try this science, which seems untenable to her and other researchers. She challenged these enterprises: “we have reached the cliff. Should AI enterprises continue to use flawed research assumptions or do what should be done?”

Responsible editor; zl

Leave a Reply

Your email address will not be published. Required fields are marked *