Schedule of Events | Symposia

Graduate Student Award Winner

Thunder and Lightning: Vision Language Model Representations Predict EEG Response Differences to Visual vs Auditory Attributes in Property Verification

Poster Session E - Monday, March 31, 2025, 2:30 – 4:30 pm EDT, Back Bay Ballroom/Republic Ballroom

Harshada Vinaya¹ (hyadav@ucsd.edu), Sean Trott¹, Seana Coulson¹; ¹University of California San Diego

A key criticism of using LLM vectors as proxies for human semantic representations is the evidence that humans also rely on sensorimotor experiences in the world. However, vision language models (such as CLIP) operationalize semantic representations that integrate textual and visual information. Here, using representations acquired from CLIP, we ask whether these LLM representations informed with vision can predict differences in human EEG response to words for visual (“red”) versus auditory (“loud”) attributes presented in a property verification task. EEG was recorded as participants (n=18) indicated whether properties (“red”) were typical for the preceding concepts (“apple”). Our dependent variables were the mean amplitude measurements of single-trial EEG responses to properties during the early (300-400ms) and late (400-500ms) phases of the N400. We then modeled the EEG using mixed effects models with random intercepts for subjects, items, and electrodes, and fixed effects of word frequency, modality of the property word (auditory versus visual), and semantic distance measurements using GloVe and CLIP embeddings calculated as cosine between the vectors for concept and property words. The early phase of the N400 was significantly predicted by word frequency (β=0.4), CLIP (0.4), modality (-0.6), and interaction between modality and CLIP (-0.3) showing differential effects predicted by CLIP based on the sensory attributes. The late phase, however, was predicted only by CLIP (0.2). Findings suggest human semantic representations accessed early in the N400 during this task may incorporate visual information.

Topic Area: LANGUAGE: Semantic