Schedule of Events | Symposia

Aligning behavioral expressions of memory with convolutional neural network representations

Poster Session C - Sunday, March 30, 2025, 5:00 – 7:00 pm EDT, Back Bay Ballroom/Republic Ballroom

Julian Gamez¹, Anisha S. Babu¹, Brice A. Kuhl¹; ¹University of Oregon

Deep convolutional neural networks have become an increasingly popular tool in cognitive neuroscience. A primary appeal of these models is that they approximate human judgments of similarity for complex stimuli. However, less is known about how these models align with behavioral expressions of memory—in part because memory measures typically lack representational structure. Here, we used Natural Language Processing to capture the representational structure of verbal recall of naturalistic scenes and tested for alignment with different layers of a convolutional neural network (VGG-16). Subjects (N=38) first studied and practiced recalling associations between faces and scenes. Scenes included 6 exemplars from 6 visual categories (e.g., libraries, pools, etc.). After extensive training, subjects completed a final recall task where they were shown each face and typed a detailed description (memory) of the associated scene. MPNet was applied to these descriptions, yielding a unique semantic embedding for each memory. We then calculated the cosine similarity between (a) the semantic embeddings and (b) features extracted by VGG-16 across different layers of its architecture. We found that representational structure of recall was well explained by VGG-16. Interestingly, however, whereas relationships between scenes from different visual categories (e.g., libraries vs. pools) were much better explained by higher vs. lower model layers, fine-grained relationships between scenes from the same category (e.g., library 1 vs. library 2) were equally-well explained by intermediate and higher layers. Together, these results establish an important link between the behavioral expressions of memory and the representational structure of convolutional neural networks.

Topic Area: LONG-TERM MEMORY: Episodic