GitHub AI Researcher Automates Own Intellectual Toil, Unleashes Self-Service Coding Agents for Team

Breaking: GitHub Copilot Applied Science Team Researcher Builds 'Eval-Agents' to Automate Benchmark Analysis

A lead AI researcher at GitHub's Copilot Applied Science team has developed a tool that automates the intellectually demanding task of analyzing coding agent performance, effectively outsourcing the analysis to AI agents themselves. The tool, called eval-agents, emerged from the researcher's repeated use of GitHub Copilot to sift through thousands of lines of agent trajectory data.

"I may have just automated myself into a completely different job," the researcher said. The tool allows team members to generate and share custom agents that analyze benchmark runs, reducing analysis time from hours to minutes.

Background

Evaluating coding agents requires poring over trajectories—JSON files containing hundreds of lines detailing an agent's thought processes and actions during benchmark tasks like TerminalBench2 or SWEBench-Pro. A single benchmark run can generate hundreds of thousands of lines of such data.

Previously, the researcher used GitHub Copilot to surface patterns, manually investigating the most promising leads. "I kept repeating the same loop," they said. "The engineer in me said, 'I want to automate that.'" That realization sparked the creation of eval-agents.

What This Means

The eval-agents system enables scientists and engineers to author new analysis agents without writing boilerplate, share them across the team, and make coding agents the primary vehicle for contributions. This shifts the researcher's role from manual analyst to maintainer of an automated pipeline.

"Engineering and science teams work better together," the researcher emphasized. The project's design priorities—make agents easy to share and use, easy to author, and the primary contribution vehicle—reflect values the researcher honed as a maintainer of the GitHub CLI open-source project. The full implications for AI evaluation workflows are still unfolding, but early adopters report dramatic speedups in benchmark analysis.

This development comes as the industry races to evaluate increasingly complex AI coding agents. Standardized benchmarks are multiplying, and the ability to rapidly analyze agent performance could accelerate progress. The researcher expects the tool to be open-sourced in the future, pending internal reviews.

This is a breaking story. More details to follow.