How do we recognize other people's emotions?

A person’s facial expression provides much of the crucial information for us to recognize their emotions. However, according to some new research, there’s more to it than that.

A multi-university team of researchers describes emotion recognition not as a separate module, but as part of a comprehensive process that helps us form a general impression of another person. This process of person impression formation also includes physical and cultural characteristics as well as background information. The paper was published in the journal Philosophy and Phenomenological Research.

In the 1970s, the theory was put forward that the face is the window to our emotions. Researcher Paul Ekman described basic emotions such as fear, anger, disgust, joy and sadness using typical facial expressions, which were found to be similar across all cultures.

What the researchers say: “However, in recent years it’s become increasingly obvious that there are many situations in life where a typical facial expression is not necessarily the key piece of information that guides our assessment of other people’s feelings,” the lead researcher points out and cites the following example: “People almost universally rate a typical facial expression of fear as anger when they have the background knowledge that the assessed person’s been turned away by a waiter even though they’d demonstrably reserved a table.” In such a situation, people expect the person to be angry, and this expectation determines the perception of their emotion, even if their facial expression would typically be attributed to a different emotion.

“In addition, we can sometimes recognize emotions even without seeing the face; for example, the fear experienced by a person who’s being attacked by a snarling dog, even though we only see them from behind in a stance of freeze or fright.” Recognizing an emotion is therefore part of our overall impression of a person.

The researchers propose that recognizing emotions is a sub-process of our ability to form an overall impression of a person. In doing so, people are guided by certain characteristics of the other person, for example physical appearance characteristics such as skin color, age and gender, cultural characteristics such as clothing and attractiveness as well as situational characteristics such as facial expression, gestures and posture.

Based on such characteristics, people tend to quickly assess others and immediately associate social status and even certain personality traits with them. These associations dictate how we perceive other people’s emotions. “If we perceive a person as a woman and they show a negative emotion, we’re more likely to attribute the emotion to fear, whereas with a man it’s more likely to be read as anger,” the lead author points out. This is because background information is included in the assessment.

In addition to the perception of characteristics and initial associations, we also hold detailed person images that we use as background information for individuals in our social circle – family members, friends and colleagues. “If a family member suffers from Parkinson’s, we learn to assess the typical facial expression of this person, which seems to indicate anger, as neutral, because we are aware that a rigid facial expression is part of the disease,” he said.

The background information also includes person models of typical occupational groups. “We hold stereotypical assumptions about the social roles and responsibilities of, for example, doctors, students and workmen,” the authors explained. “We generally perceive doctors as less emotional, for example, which changes the way we assess their emotions.”

In other words, people make use of the wealth of characteristics and background knowledge to assess the emotion of another person. Only in rare cases do they read the emotion from a person’s facial expression alone. “All this has implications for emotion recognition using artificial intelligence (AI): It will only be a reliable option when AI doesn’t rely solely on facial expressions, which is what most systems currently do,” the authors cautioned.

So, what? Often good research merely confirms, what I call “the blinding flash of the obvious.” So it is with this piece.