Getting to Know You: New Insights in Facial Recognition

The COVID-19 global pandemic is opening new questions for exploring how we get to know and recognize people in our lives. From the rise of digital representations of people — whether through avatars or teleconferences — to the use of masks for protection, we are taking in lots of different types of information beyond mere facial recognition. 

New types of analyses, including multivariate pattern analysis techniques, are giving cognitive neuroscientists new insights into the neural mechanisms of recognizing objects and faces in a way that was previously not possible. In a new review in the Journal of Cognitive Neuroscience, Gyula Kovács of Friedrich Schiller University Jena lays out a model for how people become familiar to others. 

“My major aim was  to integrate unfamiliar and familiar face processing into a single, unified framework. I feel that this is entirely missing from the literature,” Kovács says. “ We have a lot of data generally on faces, and specifically on familiar faces, but the gradual transformation of one into the other is not represented in any current model.”

CNS spoke with Kovács about his model and its implications for recognizing everyone from celebrities to people in masks, what his review says about brain areas involved in getting to know a person, and next steps for this line of research. 

CNS: How did you become personally interested in this research area?

Kovács: I worked for several years on the neural basis of face perception. Recently, it became more and more obvious that the face of a familiar person — let that be Angelina Jolie or your own mother — is represented very differently in the human brain than that of an unknown person. This finding was very puzzling to many face-researchers, like myself, and made us ask what the exact difference is between the processing of familiar and unfamiliar faces. As a result, several studies came out that pointed to the most important differences being in relatively advanced, “high-level” processing steps. This made me wonder: What exactly does it take to get to know a person? In other words, what are the steps of this cognitive process? How can we get to a level that (we feel) we know Angelina Jolie very well? And how is it different from the representation of my wife? Above all, what happens during this process in the human brain and how can we “catch” the representations of a given individual through experiments in the laboratory? 

CNS: Can you walk us through how the model would show how we get to know someone with an example from the real world?

Kovács: In the paper I describe the process with the following example. 

“Imagine the following situation. You are commuting every morning the same time with the same train to your workplace. After a while, you start to realize that other people around you do the same and you recognize a few faces as familiar. A blond man always reads his newspaper, or a brunette woman always wears colourful scarfs. Once you initiate a quick chat with that blond man, you learn that he lives in the same city and has the same profession as you. One evening, as your train is cancelled, you have to wait another hour at the railway station. You decide to invite the blond man for a beer to the corner pub. He is called Jules and has two daughters, the same age as your own ones. You like the conversation with the man, and you decide to invite him over to your house next week for a dinner with his family. From this point onward, you slowly develop a friendship with him, and in a year or two, he becomes a very good friend of yours. During this process, an originally unfamiliar person (and his face) first becomes somewhat familiar and is eventually very well known to you.”

I think this example illustrates well the three most important stages of the process of how our brain collects information about another person as we get to know someone. The first stage is basically just the repeated visual presentation of a face: We see someone regularly, but we don’t have any other information about the person. After a while we would feel that they are “familiar”, even if we don’t know anything about them. This level of “perceptual” familiarization already leads to differential activations for familiar and unfamiliar faces in quite a few cortical and subcortical areas. At the same time, identity-specific activations are either very basic or non-existent at this level. The second stage would be the gradual accumulation of associated knowledge, such as context information, biographical details or episodic memories associated with the person’s faces. For example, we would learn to associate the railway-station or the newspaper with the blond man, helping to identify him.  This level is probably the best modelled by our knowledge of faces of famous people whom we only got to know through various media sources. Finally, this representation gradually turns into the third, final stage where our knowledge about a person gets unique and detailed, enabling effortless person recognition and identification. Eventually the man from the railway-station became a good friend. 

People are more than their faces, and person identification involves the integration of other visual features and cues, multimodal sensory information, and a range of non-sensory features, such as biographical knowledge, emotional aspects, and personality traits.

CNS: Does your paper offer any insights on how that might change with someone wearing a mask?

Kovács: During the current COVID-19 pandemic we are all expected to cover the lower half of our face in public with surgical masks or bandanas. But, as we all share this experience on a daily basis, wearing a mask makes many aspects of social interactions more difficult and awkward. Interestingly, previous research showed that the lower face-half is less informative and useful for identifying individuals, yet a mask still impairs our identification during our daily life. As of today, I am aware of only a handful of systematic studies about the effect of deliberate disguise on face perception and recognition. Indeed, the few available behavioral studies of this question suggest impaired performances in recognition memory and perceptual tasks under such challenging conditions and I have already read reports of crimes committed with the help of surgical masks as well.

Luckily, people are more than their faces, and person identification involves the integration of other visual features and cues (e.g. jewelry, hair color, cloths, gestures, gaits), multimodal sensory information (e.g. voices), and a range of non-sensory features, such as biographical knowledge, emotional aspects, and personality traits. As the review shows, these aspects are encoded in an extensive network, named as person identity network (PIN) and face processing areas represent only a part, albeit an important one, of this network. Thus, I would assume, that through the lack of proper facial information humans will learn to rely more on other (sensory and associated) features and will rapidly overcome those difficulties that are caused through masks. It is, however, an interesting, open question how we actually identify someone whose lower face half is disguised by a mask and if we could be specifically trained for such tasks. 

CNS: How does the model change for recognizing a celebrity?

Kovács: Celebrities are only one subtype of familiar persons. In cognitive neuroscience, familiarity is traditionally divided into three distinct qualities: Perceptual and personal familiarity and famous persons. Celebrities we know through media consumption and usually without any bi-directional interactions. I would argue that the neural representations of our favorite celebrities, whom we know for a long time, whose personal life we follow intensively in the media and towards whom we develop emotions, are very similar to that of personally familiar individuals. In a previous experiment of my laboratory (Ambrus et al, 2019), we could easily decode identity specific information from the electrophysiological signals for four well-known celebrities. At the same time, my current research suggests that this is not possible for experimentally familiarized faces. So, if you know and like her, you definitely have a robust representation of Angelina Jolie in your brain. However, for less known persons, whom we know just because we recently watched a TV series, whom we neither like nor find particularly interesting the neural representations are at an intermediate level between unfamiliar and personally familiar persons. For example, I know that Matthew Rhys is an actor, and I remember his face from the TV series I watched, but I don’t really “know” him.

CNS: What were you overall most excited to find? 

Kovács: To me personally the most surprising result of the survey of the literature was that it is not the face specific subgroup of areas that encode person identity differentially for various familiar faces. Rather it seems that some high-level areas, involved in long-term memory, such as the medial temporal cortex and parietal areas as well as emotion processing areas, such as the amygdala change their activity as we get to know someone. Previous studies concentrated on the differential encoding of unfamiliar and familiar faces, mostly in the so-called “core” face network, situated over occipito-temporal locations. But it seems that these areas encode familiarity and identities from a very early stage of familiarization onwards. Although there are differences in the nature of this representation (for example it becomes more and more independent of local stimulus features, as one gets more and more familiar with a face), it seems that the activation of the entire PIN network, and especially that of the memory and emotion related areas, is necessary for a robust and efficient person identification. 

This review shows that a large portion, including areas from every corner of your brain, is necessarily active, even for such a simple task as identifying your spouse or your own children.

CNS: What do you most want people to understand about this work?

Kovács: For the general public the most important message is that identification and recognition are the results of the activation of a large network. It is a common myth that we only use 10 or 15 or whatever percent of our brain capacity. I usually ask my students at my first lecture what do they think about this statement by presenting the trailer of the film Lucy (Luc Besson) where Morgan Freeman’s character says, “It is estimated human beings only use 10 percent of the brain’s capacity.” The students usually consider it true. This review shows that a large portion, including areas from every corner of your brain, is necessarily active, even for such a simple task as identifying your spouse or your own children. And this is true for every single cognitive task. Our brain always processes information across the activations in a network. So, we always rely on the entirety of our whole brain.

For the cognitive neuroscientist colleagues, perhaps the most important message is that we need far longer and comprehensive familiarization training periods in the future than what we typically use (for example, in recognition memory experiments) if we study the neural correlates of recognition and identification. In a typical experiment, the repeated presentation of stimuli is followed by a subsequent retrieval phase and the participants’ task is to decide if they have seen a given item or not. Such paradigms have only minutes or maximum hours long study phases. The current review suggests that the resulting representations are very different from those of famous or personally familiar faces where we had weeks-months or even years-long experiences. One explanation of this difference is that the creation of a robust and reliable identity representation across the entire PIN requires the long-term and intensive training we only have for famous and personally familiar identities. 

CNS: What’s next for this line of work?

Kovács: We pointed to several open questions, gaps in our knowledge and those require specifically targeted experiments. The review also raised several testable hypotheses as to the factors influencing the various stages of the process when we get to know someone. At the end of our article, we listed a few potentially important directions. For example, we asked about the exact role of spatial or conceptual context in identification. We know that context can facilitate visual perception for nonface objects, but how does it modulate identity representation? Another, exciting question, that became especially important during the COVID-19 pandemic, concerns the similarities and differences of representations for persons we encountered in real-life and those we “met” only digitally. The comparison of these real-life interactions and the digital, parasocial relationships is entirely missing from cognitive sciences. The simultaneous acquisition of neuroimaging and electrophysiological data, together with advanced multivariate pattern analysis methods in the future might refine the proposed model in these directions and advance our general knowledge about the neural representations of items, objects and above all, persons.

-Lisa M.P. Munoz