Computers Generate Faces Based on Mental Maps

Cognitive neuroscientists have long debated about whether people have visual-like “pictures in the brain” that we activate when we think of them, or whether representations are more semantically organized in sets of features. So, for example, if asked to think of a blond person, will someone conjure in their “mind’s eye,” a typical blond person such as Taylor Swift, or will they just target some sort of blondness feature? 

To help tease out this process, some cognitive neuroscientists are now collaborating with computer scientists, using machine learning paired with EEG measurements to map mental representations. In a new paper in Science Reports, Lauri Kangassalo and colleagues at the University of Helsinki used such a technique to computer-generate faces based on the intentions of people thinking about certain facial features. 

“Faces, in particular, seem to work great, maybe because it is a very constrained part of vision that humans seem to be incredibly good at; only very recently, new methods in machine learning have been able to surpass our ability to recognize thousands of distinct faces,” says Michiel Spapé, a co-author on the new paper. “As such models very much started by borrowing from research on human physiology and perceptual processes, I think it’d be a beautiful irony if now we could use them to solve old questions in cognitive psychology as well.”

CNS spoke with Spapé about the research area, the paper’s significance, and new directions for the interface of cognitive neuroscience and computer science. 

CNS: How did you become personally interested in this research area?

Spapé: Back in 2014 or so, I was working as a postdoc at the Helsinki Institute for Information Technology, a research institute established to foster collaboration across the two universities of Helsinki – the University of Helsinki and Aalto University – with an interdisciplinary technology focus, perhaps a bit like MIT. That maybe explains how I as a psychologist and EEG researcher got to talk to two fellow postdocs, one an expert in computer science and information retrieval, Tuukka Ruotsalo, and the other in mathematics and machine learning. Their idea was to get ‘search intentions’ directly from the brain by doing some machine learning wizardry and automatically classifying brain waves to know what someone might be looking for if, for example, they’re trying to find the right paper in a science search engine like Google Scholar. 

This idea of directly getting something from the brain has always been a particular beef of mine, and then, as now, I argued that you cannot treat an EEG amplifier like some sort of plug-and-play device like a computer mouse. Instead, you require both a good idea of what EEG does well and doesn’t do well, of sound experimental design, and of the myriad things that can go wrong with data processing. 

CNS: So what was the result of that collaboration? 

Spapé: We got together and eventually designed a nifty machine to detect from the brain whatever a participant is trying to search for. It classified words during reading of Wikipedia information as either relevant or irrelevant to a target topic and then fed relevant words as a search query, which would retrieve the ‘intention’ and based on this, suggest more Wikipedia articles the participant should be interested in. I sometimes call this, tongue in cheek, the ultimate Brain-Wikibinge-machine, hinting at its practical applicability. Theoretically, however, it’s a powerful new way of approaching BCI that is at the core of our ongoing studies: classify relevant information, then use such partial detections to adaptively search within a representational space. 

As I left Finland for tenure in England, Tuukka continued to explore the topic and eventually struck upon the idea of using generative adversarial neural networks (GANs) in this context, which is a super-hot topic in computer science [generating new data from a set of training data]. We discussed the topic and I rather liked the idea of using EEG to link a hypothetical mental representational space to a neural network space. So as Tuukka got a bit of funding, he managed to lure me back to Finland to work on this as the cognitive neuroscientist in the team.

CNS: How does your neuroadaptive modeling approach differ from those done in the past?

Spapé: Brain-computer interfaces (BCIs) generally use classification of usually quite simple mental states, like left vs right imaginary hand movements, in order to control a computer, for example moving a cursor left or right. This sort of closed control-loop — between EEG, training, online classification, and output towards the left or right — is not exactly what we are doing. Instead, in our work we show fake faces retrieved from the GAN while participants have a mental target (like “anyone who is smiling”) in mind, with single-trial EEG classified as suggesting a face was seen that was either relevant or irrelevant to the task. Now, as each face is in itself a point in the GAN, we can use the information of relevant spaces to triangulate a point that is optimal towards the task, converging – as they say in this field – on a target that is perhaps the ultimate smiler. That means we don’t immediately show with every face some sort of BCI output like cursor movements. Instead while the subject is doing their thing, we gradually learn what relevance means to them, and infer the target implicitly, while we explicitly show the subjects the target only  a few months later. In that sense, the paper reviewers were quite right in that control was exercised in quite a ‘blind’ manner.

CNS: What were your most excited to find? 

Spapé: I still have the email exchange with the first produced results, and it was a real ‘wow’ moment when the produced, completely novel images seemed to match the intended category pretty much every single time. In fact, coming from a background that seems plagued by failed replications of small effects and samples, I took tons of precautions to make sure the effect was measurable and stable, but as it is we report near ceiling levels in our blind evaluations.

While there are many bombastic claims out there that we will soon be able to “decode the brain,” gaining magic insight into what people are “really thinking,” our research doesn’t show this, nor does it seem likely that it is soon possible. It is not, necessarily, that our tools are incapable as such, but rather that we don’t even seem capable of explaining what ‘it’ is that it is being decoded. It is more like interactive modelling than directly decoding. 

CNS: What is the takeaway message you most want people to understand about this work? 

Spapé: The takeaway is quite simply this: We have a new kind of machine that we think can be an invaluable, new tool in mapping cognitive processes, such as perhaps mental imagery and the nature of visual mental representations. It may be risky to claim  that the GAN presents a good model for mental representations, but it is an exciting empirical question at that. In particular, since the GAN is generative, this would allow us to visualize ongoing mental processing by producing images. So we need to approach this question from different angles, and if anyone reads this and wants to collaborate, just shoot me an email.

For non-scientists, I would like to reiterate that while there are many bombastic claims out there that we will soon be able to “decode the brain,” gaining magic insight into what people are “really thinking,” our research doesn’t show this, nor does it seem likely that it is soon possible. It is not, necessarily, that our tools are incapable as such, but rather that we don’t even seem capable of explaining what ‘it’ is that it is being decoded. It is more like interactive modelling than directly decoding. 

It would be amazing if we could make a contribution to … visualizing implicit associations in a way that avoids the problems in using reaction times.

CNS: What’s next for this line of work? 

Spapé: We are exploring avenues of more theoretical, cognitive interest at the moment. It is always a bit of a balance with interdisciplinary work, and sometimes you have to give a bit and take a bit, but after working together for so long we have reached a pretty solid understanding of what works. So, this Scientific Reports work had a reasonable balance between computer science and cognitive neuroscience, but in the same project we published one article in CHI on the use of BCIs in crowdsourcing, and at least two articles are submitted on the more engineering-related parts of this neuroadaptive modelling approach.

Now my own interest is more in the fundamental question whether this system can tap into representations and say something meaningful about cognition. We did a proof of concept study that was very similar to the categories one, but instead of having objective categories, used a purely subjective one, attractiveness. In the task we called BrainTinder, instead of asking people to focus on blond or dark-haired people, we had them to concentrate on anyone they found attractive. The question then was whether the EEG-predicted relevant – attractive – people could be used as vectors in the GAN-space to produce a mental image of a face that was expected to be attractive specifically to the person generating it. And like the currently published study, we got people back in the lab after a little while and tested whether these completely new, personalized images were ‘swiped right’ by them. It’s still under review. The funny thing is that if you take a look at all the generated faces, our male (self-reported) subjects generated remarkably similar faces across the board, while women showed a lot more variance. And this maps onto the literature, which means the neuroadaptive modelling approach can provide a unique, visual insight into subjective preferences and individual differences.

CNS: Is there anything I didn’t ask you about that you’d like to add?

Spapé: Yes, well, with regards to social psychological research, social neuroscience, and pervasive issues of systemic bias in society, it is perhaps good to mention that we are aware of certain bias in the model, as we sometimes get questions about this. For example, there are very few people of color in the produced images. Similarly, the neuroadaptively generated images visualize stereotypes in a very interesting way: generated images for the old category, for example, rarely look female. So, it is a reasonable question whether this is due to the GAN’s training data or the representational space of the participants. It is important to keep in mind that the GAN space itself was generated with celebrities, so you can expect certain overrepresentations that mimic the #OscarsSoWhite issue. On the other hand, it may well be possible that the exposure to such norms have an impact on mental representations themselves, and in that way, it may well be that the neuroadaptive model works so well because the lack of diversity of the GAN arguably matches our own implicit associations. At this moment, however, we do not know for sure whether the inequality is due to the GAN or the human representation, but it would be amazing if we could make a contribution to that field, visualizing implicit associations in a way that avoids the problems in using reaction times.

 -Lisa M.P. Munoz