Schedule of Events | Symposia

Sketchpad Series

Human-like social and emotional perception with GPT-4V

Poster Session D - Monday, March 31, 2025, 8:00 – 10:00 am EDT, Back Bay Ballroom/Republic Ballroom

Severi Santavirta^1,2 (svtsan@utu.fi), Yuhang Wu^1,3, Lauri Suominen¹, Lauri Nummenmaa^1,2,4; ¹Turku PET Centre, University of Turku, Turku, Finland, ²Turku University Hospital, Turku, Finland, ³Department of Psychology, Renmin University of China, Beijing, China, ⁴Department of Psychology, University of Turku, Turku, Finland

Humans navigate the social world by rapidly perceiving social features from others and social interactions. Social interactions evoke emotions that subsequently influence social interactions highlighting the tight link between social and emotional processes. Recently, artificial intelligence (AI) and large-language models (LLMs), have achieved high-level visual capabilities for object and scene content recognition and description. This raises the question whether LLMs can perceive nuanced and tacit social and emotional information from visual stimuli, or simulate the emotions that humans experience in their everyday life. To answer these questions, we collected rich social (136 features) and emotional (48 features) human annotations from hundreds of images and videos and compared those to the similar annotations produced by a prominent visual LLM, GPT-4V. Additionally, we recreated the neural representations for social perception based on GPT-4V social perceptual evaluations in a fMRI study where 97 subjects viewed socioemotional movie clips to investigate the practical reliability of simulated annotations within cognitive neuroscience. The preliminary results revealed that GPT-4V can produce human-like social perceptual evaluations (r_image = 0.61, r_video = 0.53) and their neural representations (PPV = 0.76) demonstrating the practical potential of LLM derived socioemotional responses. Reliable simulation of social-emotional processes would have wide range of real-life applications ranging from health care to business and would open exciting new avenues for cognitive research through automated high-dimensional stimulus annotations.

Topic Area: EMOTION & SOCIAL: Person perception