Research Interests

LLM-ology: probing the “black box” of Large Language Models.

Large Language Models (LLMs) like ChatGPT are seeing widespread use. But how exactly do LLMs work?

Part of my research focuses on LLM-ology, the study of how LLMs work “under the hood”. So far, this has focused on applying methods from Cognitive Science to probe the capabilities and limitations of LLMs through the lens of cognitive psychology and neuroscience. Ultimately, I believe that research on LLMs can also inform our understanding of human cognition.

Relevant papers and projects:

  • Trott, S., Jones, C., Chang, T., Michaelov, J., & Bergen, B. (2023). Do Large Language Models know what humans know?. Cognitive Science [Link to paper]

  • Jones, C. R., Chang, T. A., Coulson, S., Michaelov, J. A., Trott, S., & Bergen, B. (2022). Distributional Semantics Still Can’t Account for Affordances. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 44, No. 44). [Link to paper]

Why are languages so ambiguous?

Human lexica are rife with ambiguity––words with the same form, but different meanings. Sometimes these meanings are entirely unrelated, as is the case for homophony (e.g., the bark of a dog vs. the bark of a tree); sometimes they are closely related, as in polysemy (e.g., the chicken in the yard vs. the chicken on the plate).

Why would a communication system ostensibly evolved for efficient communication allow the reuse of the same signal for multiple meanings? Do homophones offer a benefit to languages and language users, or are they simply a (mostly harmless) “bug” in the system? And is polysemy subject to distinct selection pressures?

Relevant papers and projects:

How are ambiguous words represented?

The prevalence of lexical ambiguity also raises the question of how human minds process and represent the meanings of ambiguous words.

Traditionally, words and their meanings as conceived as discrete entries in a mental dictionary. But meaning is often dynamically modulated in different contexts. We’ve been exploring an alternative account, in which word meanings are viewed as attractors in a continuous state-space—and then asking whether there is evidence for category boundaries atop this continuous space. Recent work suggests word meaning may be both categorical and continuous.

Relevant papers and projects:

  • Trott, S., & Bergen, B. (2023). Word meaning is both categorical and continuous. Psychological Review. [Link]

  • DeLong, K. A., Trott, S., & Kutas, M. (2022). Offline dominance and zeugmatic similarity normings of variably ambiguous words assessed against a neural language model (BERT). Behavior Research Methods, 1-21.

  • Trott, S., Bergen, B. (2022). Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in context. [Link to arXiv][Link to dataset]

  • Trott, S., & Bergen, B. (2021). RAW-C: Relatedness of Ambiguous Words, in Context (A New Lexical Resource for English). ACL-IJCNLP-2021. [Link to paper] [Link to dataset and code]

Pragmatic inference

People often speak indirectly. For example, the sentence “My car isn’t starting” could be intended not only as a statement of fact, but also as a request for a ride. Similarly, the sentence “Can you open that window?” can function as a request to open the window, a question about the hearer’s ability to do so, or both.

How do comprehenders determine whether a speaker is making a request? Specifically: which linguistic and non-linguistic cues to an utterance’s meaning do comprehenders exploit to enrich the meaning of an under-specified utterance like “My car isn’t starting”?

Relevant papers and projects:

  • Trott, S., Reed, S., Kaliblotzky, D., Ferreira, V., & Bergen, B. (2022). The role of prosody in disambiguating English indirect requests. [Link to paper][Data and code for analysis]
  • Trott, S., & Bergen, B. (2020). When do comprehenders mentalize for pragmatic inference? Discourse Processes. [Link to preprint][Data and code for analysis]
  • Trott, S., Reed, S., Ferreira, V., & Bergen, B. (2019) Prosodic cues signal the intent of potential indirect requests. Proceedings of the 41st Annual Meeting of the Cognitive Science Society. [Link] [Data and code for analysis]
  • Trott, S., & Bergen, B. (2018). Individual Differences in Mentalizing Capacity Predict Indirect Request Comprehension. Discourse Processes. [Link] [Link to experimental materials]
  • Trott, S., & Bergen, B. (2017, October). A theoretical model of indirect request comprehension. In 2017 AAAI Fall Symposium Series. [Link]