Measuring Proactive Coding Agent Insights

Key points

• 1

  Ground Truth from Real Bug Fixes: To measure proactive agents, we analyze actual bug-fixing history using two heuristics: temporal proximity (bugs filed close together) and semantic similarity (related bug symptoms). For example, a cluster of bugs with 'sandbox timeout errors,' 'broker config failures,' and 'network isolation flaky tests' all point to the goal of 'Strengthening sandbox execution reliability.' This approach creates a 'ground truth' by treating the team's real bug fixes as the target for evaluating how well agents diagnose underlying engineering objectives. The key is that individual bugs are too narrow to be goals, but together they reveal the higher-level objective the developers were actually working toward.

• 2

  Exploration Budget Determines Accuracy: The agent's performance depends on how much time it gets to explore the codebase. Our tests showed that with just one exploration round, agents consistently identify highly relevant insights (4.5/5 accuracy). But for complex problems, increasing the exploration budget from two to three rounds boosts the agent's ability to capture secondary signals—improving 'Hit@5 accuracy' from 33% to 57%. This means developers should give proactive agents more time to investigate when tackling intricate issues, as the extra rounds help uncover hidden patterns that single-pass analysis misses. For instance, a bug cluster requiring network and configuration fixes would benefit from three rounds to catch all related dependencies.

• 3

  Real-World Testing Methodology: We tested this approach using 705 bugs (1,178 code changes) from Google's internal codebases. The agent started at the exact pre-fix state of the codebase, explored for up to three rounds, then generated insights judged against the 'ground truth' targets (the actual engineering goals behind the bugs). This method avoids theoretical benchmarks by using real engineering data, ensuring evaluations reflect actual developer workflows. The practical takeaway is that teams can replicate this by analyzing their own bug-fixing patterns to identify underlying goals—like grouping recent issues around specific symptoms to uncover shared objectives without manual analysis.

What changed