Computational Methods Guided by Expert Judgment

I am a computer scientist working across disciplines to develop methods that allow experts to guide, evaluate, and revise computational analysis. I study how instructions, examples, and feedback can preserve expert judgment when language models are used to analyze large document collections. My research develops these methods through collaborations in law and education, where errors can change a scholarly conclusion or a decision about learning.

I am a Researcher Associate in the Computer Science Department at Carnegie Mellon University and a member of the leadership team at the Technology for Effective and Efficient Learning (TEEL) Lab. I am responsible for the lab’s research activities, setting strategic priorities and advising researchers. My training in computer science, intelligent systems, and law supports collaborations that connect technical method development with the questions experts need to answer.

Research

My research focuses on computational methods for analyzing large collections of documents. In empirical legal and social science research, scholars use these collections to study institutions, professional practices, and human behavior. I develop methods that allow researchers to bring their expertise into this analysis and evaluate whether the results support their conclusions.

Researchers identifying substantive but often nuanced patterns across large document collections must interpret individual passages and relate them to broader concepts. My work spans methods for retrieving relevant evidence, annotating documents, and supporting interpretive analysis. I investigate how expert knowledge can guide these methods through instructions, examples, and feedback, and how evaluation can reveal errors that affect research conclusions. This includes developing tools and interpretable natural language specifications, such as annotation instructions and evaluation criteria, that help researchers inspect computational analyses and revise them as their understanding develops.

Selected contributions

  • Retrieving evidence for legal interpretation. I led the development and evaluation of methods for retrieving judicial passages that explain statutory terms, with supervision and advice from Kevin Ashley. The methods identify passages that help explain a term’s meaning, giving researchers evidence for interpreting it. Research paper.
  • Expert-guided analysis of case law. In collaborative work with Jakub Drapal and Hannes Westermann, I developed the computational approach and designed the study and evaluation for thematic analysis of 785 theft-case descriptions. Expert feedback improved the initial labels assigned to passages, while the resulting themes still required expert supervision. The study identifies both opportunities and limits for scaling interpretive research. Research paper.
  • Tools for expert annotation. I developed Gloss, a software environment for annotating documents, used by research groups at more than ten universities. For example, it supported annotation of a corpus of 42 European Court of Human Rights decisions. Example corpus and study.

Further work evaluates language models for assigning meaning-based labels to legal texts and generated assessment questions in programming education. Google Scholar lists my publications.

Teaching

  • Large Language Models: Methods and Applications, Carnegie Mellon University, 2025 - Present (11-667 in 2025; 11-967 in 2026). I teach graduate students the methods underlying large language models and their evaluation and application. 2025 course materials.
  • Applied Legal Analytics and AI, University of Pittsburgh, 2020 - Present. I co-teach a project-based course in which students from law, social science, and technical backgrounds formulate research questions, analyze legal documents, and evaluate computational methods.

Funded Projects

  • Teaching Programmers When to Rely on Artificial Intelligence: Building and Measuring Automation Judgment - Co-principal investigator, National Science Foundation award 2551554. Awarded for 2026 - 2030, starting October 2026. Total project funding: US$800,000.
  • Adapting a College AI Literacy Curriculum for High School Classrooms Through a District-Wide Research-Practice Partnership - Co-principal investigator, National Science Foundation award 2619806. September 2026 - 2029. Total project funding: US$749,261.
  • Carnegie Mellon Accenture Center of Excellence for AI (ACE-AI) - A collaboration between Accenture and Carnegie Mellon University focused on addressing critical challenges in workforce development. The center is built on three pillars: AI for coaching and tutoring, AI for training content development, and AI for learning analytics. I am a co-principal investigator.
  • AI Technicians - A collaboration between the U.S. Army’s Artificial Intelligence Integration Center (AI2C) and Carnegie Mellon University to design, implement and evaluate novel rapid occupational training methods to create a competitive AI workforce at the technicians level. I am a co-principal investigator.
  • AI Institute for Societal Decision Making (AI-SDM) - The institute brings together AI and social sciences researchers to develop human-centric AI for societal good that harnesses the power of data and an improved understanding of human decisions to create better and more trusted choices. The TEEL Lab’s role in AI-SDM is to develop and deliver an AI Literacy curriculum targeted towards learners with a high school education and without a background in science, technology, engineering, or mathematics.

Background

  • Post-doctoral Researcher, Computer Science Department at Carnegie Mellon University, 2020 - 2022
  • Ph.D., Intelligent Systems Program at University of Pittsburgh, 2013 - 2020
  • Earlier degrees in Computer Science (B.Sc., 2013) and Law and Legal Science (Mgr., 2009), Masaryk University. I also worked as a data scientist at Reed Smith LLP (2017 - 2020), developing legal AI for e-discovery and due diligence.

Service