cognitive tools lab

reverse engineering the human cognitive toolkit

our research


How does the human mind transform a cascade of sensory information into meaningful knowledge?

While traditional approaches to learning focus on how people process the data provided to them by the world, this approach leaves aside all of the powerful tools people have to actively reformat their experiences and generate new ones. For example, we choose what to look at, bring certain memories to mind, produce pictures to share, and compose stories to tell. The goal of our research is to “reverse engineer” the core mechanisms by which employing such cognitive tools affects how we perceive and understand the world.

As a case study, our recent work seeks to understand human visual communication. This encompasses behaviors from informal sketching to formal scientific visualization, and its applications to education, user interface design, and assistive technologies.

Overview of some of our recent projects

Common object representations for visual production and visual recognition

From ancient etchings on cave walls to modern digital displays, the ability to externalize our thoughts in visual form lies at the heart of key human innovations (e.g., painting, cartography, data visualization). Perhaps the most basic and versatile visualization technique is drawing, the earliest examples of which date to at least 40,000 years ago, and which can yield images ranging from photorealistic renderings to schematic diagrams.

How do we convey concepts in visual form, and how does refining this skill, in turn, affect recognition?


We developed an online platform for collecting large amounts of drawing and recognition data, and applied a deep convolutional neural network model of visual cortex trained only on natural images to explore the hypothesis that drawing recruits the same abstract feature representations that support natural visual object recognition (Panel A, above). Consistent with this hypothesis, higher layers of this model captured the abstract features of both drawings and natural images most important for recognition (Panel B), and people learning to produce more recognizable drawings of objects exhibited enhanced recognition of those objects (Panel C). These findings could explain why drawing is so effective for communicating visual concepts, they suggest novel approaches for evaluating and refining conceptual knowledge, and they highlight the potential of deep networks for understanding human learning.

Fan, J., Yamins, D., and Turk-Browne, N. (2018). Common object representations for visual production and recognition. Cognitive Science.

Visual content and social context jointly determine pictorial meaning


Drawing is a versatile technique for visual communication, ranging from photorealistic renderings to schematic diagrams consisting entirely of symbols. How does a medium spanning such a broad range of appearances reliably convey meaning?

On the one hand, recent work in computational vision has found that the identity of an object depicted in a drawing can be derived from its visual properties alone. On the other hand, classic work in aesthetics has emphasized the role of cultural and social context in determining how drawings denote objects.

Towards reconciling these perspectives, we recently conducted two studies exploring the hypothesis that visual information and social context jointly determine the correspondence between a drawing and the object it depicts. Both studies used a graphical reference game paradigm where participants were paired in an online environment and communicated with their partner only via a drawing canvas. The sketcher’s goal on each trial was to draw a particular object — the target — so that the viewer could select it from an array of distractors visible to both participants.

Our findings show that both visual information and social context are crucial for determining how drawings derive referential meaning during visual communication, and suggest that a critical factor affecting the balance between the two may be the type and amount of shared knowledge between communicators. More broadly, elucidating the cognitive mechanisms that underlie our ability to produce and interpret images across a wide range of visual abstraction may shed light on the nature of our ability to apprehend meaning from visual artifacts, and on the sources of cultural variation in pictorial style.


Fan, J., Hawkins, R., Wu, M., and Goodman, N. (submitted). Pragmatic inference and visual abstraction enable contextual flexibility in visual communication.)

Hawkins, R., Sano, M., Goodman, N., and Fan, J. (under review). Graphical convention formation during visual communication.)