Schedule of Events | Symposia

Graduate Student Award Winner

Comparing the Multidimensional Mental Representations of Object Images and Object Nouns

Poster Session F - Tuesday, April 1, 2025, 8:00 – 10:00 am EDT, Back Bay Ballroom/Republic Ballroom

Laura M. Stoinski^1,2,3 (stoinski@cbs.mpg.de), Tongue Zhuang^1,4, Chris I. Baker⁵, Martin N. Hebart^1,4,6; ¹Max Planck Institute for Human Cognitive & Brain Sciences, Leipzig 04103, Germany, ²University of Leipzig, Leipzig 04103, Germany, ³International Max Planck Research School on Cognitive NeuroImaging (IMPRS CoNI), ⁴Department of Medicine, Justus Liebig University, Giessen 35390, Germany, ⁵Section on Learning and Plasticity, Laboratory of Brain & Cognition, National Institute of Mental Health, Bethesda, MD 20814, USA, ⁶Center for Mind, Brain and Behavior, Universities of Marburg, Giessen and Darmstadt

Objects can be distinguished along various properties, from visual-perceptual to higher-level semantic dimensions. Visual and semantic dimensions are often intertwined, making it challenging to separate the influences of visual input and semantic knowledge. We can address this by comparing the dimensions underlying object images with those of their corresponding nouns, since words, unlike images, contain no image features. This allows us to distill the dimensions dominating object representations when visual features are present versus absent. Here, we aimed to identify core dimensions underlying mental representations of object words and systematically compare them with 49 dimensions previously identified from similarity judgments of object images (Hebart et al., 2020). We gathered over 1.3 million odd-one-out judgments of 1,388 diverse object nouns, revealing 50 interpretable dimensions using a computational model trained to capture these similarity patterns. The 50 dimensions captured known semantic relationships and higher-level categories and predicted cross-validated similarity choices close to intersubject consistency (94% of explainable variance). Correlating the representational similarities of objects based on word-derived dimensions with those from image-derived dimensions showed that words captured much of the similarity structure of images (r = 0.83). While images and words shared many semantic dimensions, words were only represented by a few visual dimensions related to shape, without capturing other perceptual aspects related to color or texture. Together, our findings underscore the central role of semantics in mental object representations and highlight the importance of image presentations for evoking visual-conceptual structure in similarity judgments.

Topic Area: PERCEPTION & ACTION: Vision