“Evaluating, Understanding, and Building Autonomous Agents For Real-World Software Engineering and Analysis”

Programming agents are becoming increasingly prevalent. Yet, how well do we understand their behavior in practice? Do they operate as intended, what factors influence their performance, and how can we improve them? In the first half of this talk, I will present my lab’s work on designing abstract representations of agentic trajectories and developing algorithms to evaluate the behavior of programming agents. I will then discuss our recent efforts that build on these techniques to better understand whether and how agents follow prescribed plans or expected behaviors. When deviations occur, we explore runtime interventions to guide agents back on track. These studies have provided key insights that inform the build of our own multi-agent systems, which autonomously perform language-agnostic, repository-level code translation and validation, as well as automated resolution of GitHub issues.

Reyhan Jabbarvand is an Assistant Professor of Computer Science at the University of Illinois Urbana-Champaign. Her research interests lie at the intersection of software engineering, programming languages, and artificial intelligence, with a focus on enabling and evaluating intelligent systems in complex software environments. She received her Ph.D. from the University of California, Irvine, where she was advised by Sam Malek. She was awarded the Google PhD Fellowship in Programming Technology and Software Engineering for her work on advancing energy testing of Android applications and was recognized as a Rising Star in EECS by the University of Illinois Urbana-Champaign. She received the NSF CAREER Award in 2023, and her research has been supported by generous funding from the National Science Foundation, IBM Research, Amazon, and C3.ai.