Generative Misinterpretation
Monday, March 30, 2026 | 10:00 – 11:50 a.m. (PT) | Room 134
Author: Benjamin Sobel, University of Wisconsin
Commentator: Hon. Kevin Newsom, 11th Cir.
Abstract
“Generative Misinterpretation”
By James Grimmelmann, Benjamin L.W. Sobel, & David Stein
In this Article, we show that LLMs are not yet fit for use in judicial chambers. Generative interpretation, like all empirical methods, must bridge two gaps to be useful and legitimate. The first is a reliability gap: are its methods consistent and reproducible enough to be trusted in high-stakes, real-world settings? Unfortunately, as we show, LLM proponents’ experimental results are brittle and frequently arbitrary. The second is an epistemic gap: do these methods measure what they purport to? Here, LLM proponents have pointed to (1) LLMs’ training processes on large datasets, (2) empirical measures of LLM outputs, (3) the rhetorical persuasiveness of those outputs, and (4) the assumed predictability of algorithmic methods. We show, however, that all of these justifications rest on unstated and faulty premises about the nature of LLMs and the nature of judging.
Colloquium Description
Many scholars write about the courts, about judicial process, and about the practice of judging. But what do judges think of this scholarship? Is it correct? Is it helpful? How could it be better? This colloquium on courts and judicial process brings scholars, judges, students, and faculty together to discuss current research projects about courts, judging, and procedure, among other topics. Over the course of the semester, we will discuss six projects. During a typical workshop, an invited scholar will present their work, and a judge of a federal, state, or foreign court will offer commentary on the research. Students and faculty will join in the open discussion that follows.