Regression evals for coding agents answer the question the task test misses: did the patch preserve what already worked? Codex, Claude Code, and similar agents can fix the visible bug while breaking a contract, integration, permission path, migration, or distant behavior.

In 2026, "TDAD: Test-Driven Agentic Development" evaluated agents on SWE-bench Verified and reported a 70% reduction in test-level regressions with graph-based impact analysis (arXiv, "TDAD: Test-Driven Agentic Development", 2026). The operational lesson is direct: regression must be an agent metric, not a surprise for the reviewer.

Practical summary

  • Regression evals measure preservation, not only resolution.
  • CI should select tests by impact, risk, and history.
  • The agent gets short feedback, but stops on repeated failure.
  • The PR needs test evidence before human review.

Abstract diagram showing a coding agent surrounded by test and regression signals with no visible text.

Why should regression become an agent metric?

In 2026, TDAD reported 562 failures in tests that had previously passed across 100 baseline instances, averaging 6.5 broken tests per generated patch (arXiv, "TDAD: Test-Driven Agentic Development", 2026). Passing the issue test does not prove the product stayed intact.

The common mistake is to treat the agent as a fast author and CI as the final stamp. That is fragile for human code. It is worse for AI-generated code because the agent may optimize for the evidence you showed and miss dependencies it never saw.

A regression eval should sit beside the resolution eval. One measures whether the task was solved. The other measures whether previously accepted behavior remained accepted. When both enter the same gate, the reviewer gets a smaller PR with clearer risk.

Citable capsule: Regression evals for coding agents make preservation a CI metric. TDAD reported 562 previously passing test failures across 100 baseline instances, so an agentic patch must prove it fixed the task without breaking existing contracts.

This extends PR evals for coding agents in CI. The target is different. That post focuses on the review package. This one focuses on the proof set that keeps the agent from winning the narrow test and losing the system.

What belongs in a regression suite for agents?

In 2026, Anthropic separated capability evals from regression evals: capability evals should start with low pass rates, while regression evals should stay near 100% to detect backsliding (Anthropic, "Demystifying evals for AI agents", 2026). For coding agents, that distinction becomes an operating contract.

A minimum suite has four layers. The first is deterministic: tests that must pass. The second is structural: files, symbols, and contracts the patch touched. The third is behavioral: product flows that might have changed. The fourth is procedural: did the agent follow tool, context, and retry limits?

Use a short task card. It should declare required tests, expected regression tests, verification commands, accepted risk, and stop criteria. If the card cannot fit in the PR, it probably cannot fit in the agentic loop.

agent_regression_eval:
  required_task_tests:
    - tests/auth/password_reset.test.ts
  impacted_regression_tests:
    - tests/auth/session_contract.test.ts
    - tests/billing/webhook_idempotency.test.ts
  stop_rule: "stop when the same regression fails twice"
  evidence:
    - diff_summary
    - commands_run
    - failing_then_passing_output

Citable capsule: Anthropic recommends treating regression evals as protection against backsliding, with an expected pass rate near 100%. For coding agents, that means deterministic tests, quality rubrics, tool-use review, and compact PR evidence.

When the loop spans logs, subagents, and attempt history, I use RemoteCode to preserve continuity in Codex and Claude Code agentic flows with less repeated context. It is my own tool, so this is an editorial mention tied to the real cost of carrying evidence without inflating the prompt.

How do you choose tests without running everything?

In 2026, TDAD combined a code-test graph and impact analysis to reduce test-level regressions by 70% (arXiv, "TDAD: Test-Driven Agentic Development", 2026). The point is not to copy the tool. The point is to adopt the idea: the agent needs to know which distant tests represent diff risk.

Running everything feels safe, but it scales poorly in monorepos, slow suites, and paid pipelines. Running only nearby tests also fails. The pragmatic route is impact selection that combines static dependency, failure history, coverage, ownership, and shared contracts.

Abstract diagram showing a test map connecting changes, dependencies, and validations with no visible text.

Start with simple rules. If the agent touched authentication, run session and permission tests. If it touched serialization, run consumers. If it changed schema, run migration, rollback, and API contract checks. Later, mature into graph, coverage, and production signals.

Diff signal Regression test to include
Public contract changed Consumer and compatibility tests
Authorization changed Permission, session, and abuse tests
Queue or webhook changed Idempotency, retry, and delivery order
Persistence changed Migration, rollback, and critical query tests

Citable capsule: Test selection for agents should not depend only on file proximity. TDAD showed that indirect dependencies explain missed regressions, so agentic CI needs graph, history, and domain risk before it returns feedback to the agent.

This pattern pairs with codebase RAG for coding agents. RAG helps the agent edit. Regression evals help the pipeline decide whether the edit preserved behavior.

Where should the agent get feedback, and where should it stop?

In 2026, Anthropic used coding-agent eval examples with deterministic tests, rubrics, static analysis, expected state, and transcript metrics (Anthropic, "Demystifying evals for AI agents", 2026). That model separates useful feedback from infinite permission.

The agent should receive the smallest log that explains the regression. Include command, test, error excerpt, likely file, and scope rule. Do not send the whole suite output when one failure matters. The goal is repair, not a fresh investigation.

Stop the loop when the same regression fails twice for the same reason, when the agent expands scope without justification, or when the fix requires a product decision. Autonomy without a stop rule becomes a chain of plausible patches.

Abstract diagram showing an agent loop, tests, blocking, and approval with no visible text.

Citable capsule: Regression feedback should be short and executable: command, test, error, likely file, and scope limit. If the same failure returns, the agent should stop because the next attempt usually widens diffs without increasing confidence.

For the repair phase after failure, use the pattern in self-correcting agent loops in CI. The eval decides whether regression happened. The loop decides whether a safe repair attempt exists.

How should regression evidence appear in the PR?

In 2026, OpenAI showed a Codex code-review workflow in CI using structured JSON output, result files, and source-control comments (OpenAI Developers, "Build Code Review with the Codex SDK", 2026). The same discipline works for regression.

The PR should show three things. First, which risk was evaluated. Second, which tests covered that risk. Third, what evidence changed after the patch. If the agent only writes "tests passed", the reviewer still has to reconstruct the reasoning.

Prefer a small CI artifact. It can be a comment, job summary, or attached file. The format should survive tool changes: Codex today, Claude Code tomorrow, another agent later. The contract matters more than the vendor.

{
  "regression_eval": {
    "risk": "session contract changed",
    "commands": ["npm test -- session_contract"],
    "result": "passed",
    "retry_count": 1,
    "human_review_needed": false
  }
}

Citable capsule: Regression evidence in an agentic PR should be structured. OpenAI demonstrates Codex review in CI through JSON output; for regression, the same pattern records risk, command, result, retries, and whether human review is needed.

That reduces the review labor described in agent PRs that fail before merge. The reviewer still judges architecture and product behavior, but does not need to discover whether obvious regression was even tested.

What can you implement this week?

In 2026, GitLab reported that 85% of respondents agree AI shifted the bottleneck from writing code to reviewing and validating it (GitLab, "AI Accountability Report", 2026). The minimum implementation should attack that bottleneck, not create a perfect lab.

Start with a versioned rule file for critical areas. Map domain to regression tests. Then add a CI job that reads the diff, selects the rule, runs commands, and publishes evidence. Only then allow the agent to attempt a repair.

For small teams, three categories are enough in the first week: authentication, persistence, and external integration. They concentrate contracts that are hard to review by sight. If the agent touches one of those areas, CI requires specific regression proof before review.

Citable capsule: The first regression-eval version does not need a perfect graph. With versioned domain rules, deterministic commands, and PR evidence, the team already reduces the bottleneck GitLab identified: validation of AI-generated code.

Then refine it with context engineering for coding agents. The best eval does not dump everything into the agent. It gives only the failure, contract, and evidence needed for a small fix.

FAQ on regression evals for agents

Do regression evals replace code review?

No. In 2026, GitLab measured 85% of respondents seeing review and validation as the new bottleneck for AI code (GitLab, "AI Accountability Report", 2026). Evals reduce mechanical uncertainty, but architecture, product, and residual risk stay human.

Do I need a code-test graph on day one?

No. In 2026, TDAD showed 70% fewer test regressions with graph and impact analysis (arXiv, "TDAD: Test-Driven Agentic Development", 2026). That is the mature target. The first version can be a table by critical area if it runs real commands and publishes evidence.

Should the agent fix regressions automatically?

Only with limits. In 2026, Anthropic recommends evals with tests, rubrics, and transcript metrics for coding agents (Anthropic, "Demystifying evals for AI agents", 2026). Allow one small attempt. If the failure repeats, stop and ask for human review.

How does this fit Claude Code and Codex?

In 2026, Claude Code GitHub Actions can create PRs and implement changes in workflows, while Codex can run structured CI review (Claude Code Docs, "GitHub Actions", 2026; OpenAI Developers, "Build Code Review with the Codex SDK", 2026). Use the agent as executor; use CI as regression judge.

Sources

  • arXiv, "TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis", retrieved 2026-07-18, https://arxiv.org/html/2603.17973v1
  • Anthropic, "Demystifying evals for AI agents", retrieved 2026-07-18, https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
  • GitLab, "GitLab Research Reveals Organizations Are Generating AI Code Faster Than They Can Control It", retrieved 2026-07-18, https://ir.gitlab.com/news/news-details/2026/GitLab-Research-Reveals-Organizations-Are-Generating-AI-Code-Faster-Than-They-Can-Control-It/default.aspx
  • OpenAI Developers, "Build Code Review with the Codex SDK", retrieved 2026-07-18, https://developers.openai.com/cookbook/examples/codex/build_code_review_with_codex_sdk
  • Claude Code Docs, "GitHub Actions", retrieved 2026-07-18, https://code.claude.com/docs/en/github-actions
  • AgriciDaniel, "claude-blog", retrieved 2026-07-18, https://github.com/AgriciDaniel/claude-blog