Coding agent observability is the practice of recording what Codex, Claude Code, and similar agents did, with which tools, permissions, tokens, errors, and evidence. Without it, the PR may look normal while the team cannot explain cost, risk, retries, or failure cause.
In 2026, GitHub Blog in "Agent pull requests are everywhere" reported more than 60 million Copilot code reviews, 10x growth in under a year, and more than one in five reviews involving an agent (GitHub Blog, "Agent pull requests are everywhere", 2026, retrieved 2026-07-20). That volume needs an operational trail, not trust in the agent summary.
Practical summary
- The minimum trail records tool, permission, cost, and result.
- Codex JSONL becomes a CI artifact, not an open log.
- Claude Code OpenTelemetry should start redacted by default.
- The PR needs short evidence humans can actually review.
Why did observability become part of the harness?
In 2026, arXiv in "AI Agent Pull Requests on GitHub" analyzed 33,596 agent PRs across 2,807 repositories and found co-active pairs in 40.2% of repositories under exact temporal overlap (arXiv, "AI Agent Pull Requests on GitHub", 2026, retrieved 2026-07-20). Observability became part of the harness because parallel authorship without a trail becomes invisible cost.
The harness is not only prompt, sandbox, and test. It also needs to answer who started the task, which agent executed it, which tool requested permission, which command failed, which subagent participated, and which evidence reached the PR.
The common mistake is keeping only the final diff. The diff shows the result, not the route. In an agentic workflow, the route matters because a permission failure, broad MCP access, or repeated test can explain why a small patch consumed hours.
Citation capsule: Coding agent observability turns agentic execution into an auditable trail. arXiv analyzed 33,596 PRs and found coactivity in 40.2% of repositories; when agents work in parallel, the PR needs tool, permission, retry, and proof.
Which events should the agent record?
In 2026, the same arXiv study reported that co-active pairs accounted for 79.4% of all agent-generated PRs under exact overlap (arXiv, "AI Agent Pull Requests on GitHub", 2026, retrieved 2026-07-20). The right event explains coordination, cost, or risk without copying the whole conversation.
Record task start, relevant tool calls, permission decisions, failures, retries, file changes, MCP calls, subagent creation, test result, and turn closure. Do not record secrets, full prompts, or raw tool output unless there is a clear access control reason.
Practical experience: when I put an agent in a backend pipeline, I do not start with the dashboard. I start with one stable event line: task, tool, permission, proof command, and stop reason. Then the panel grows from that contract.
| Event | Question it answers |
|---|---|
| Session start | Who requested the task and on which commit? |
| Tool use | Which tool tried to act and why? |
| Permission | Was the action accepted, blocked, or rejected? |
| Test | What proof ran before the summary? |
| Closure | Did the agent finish, fail, or request a human? |
Citation capsule: The minimum event in an agentic loop is not a full transcript. It is an auditable line with session, agent, tool, decision, result, and artifact. Since 79.4% of agent PRs were in co-active pairs, coordination has to become data.
For long tasks, context cost belongs in the trail too. I use RemoteCode to keep continuity in agentic Codex and Claude Code workflows without repeating unnecessary context when work crosses sessions, logs, and evidence; it is my own tool, so this is an editorial mention tied to observability and token cost.
How do you use Codex JSONL without leaking context?
In 2026, OpenAI Developers in "Non-interactive mode" says codex exec --json emits JSON Lines events such as thread start, turn start, completion, failure, items, and error (OpenAI Developers, "Non-interactive mode", 2026, retrieved 2026-07-20). Use that stream as a machine contract, not a public dump.
In CI, save JSONL as a restricted artifact and generate a smaller PR summary. The summary should list commands, changed files, MCP use, test result, and stop reason. The complete JSONL remains available for authorized debugging.
{
"agent_run": {
"tool": "codex",
"mode": "ci",
"trace": "restricted-artifact",
"summary": "pr-comment",
"human_review": "required"
}
}Citation capsule: Codex JSONL should feed observability, not replace review. OpenAI documents turn, item, failure, and error events; in CI, those events should live in a restricted artifact with a safe PR summary.
How do you connect Claude Code to OpenTelemetry safely?
In 2026, Claude Code Docs in "Monitoring" sets default intervals of 60 seconds for metrics and 5 seconds for logs, and exports metrics, events, and traces through OpenTelemetry (Claude Code Docs, "Monitoring", 2026, retrieved 2026-07-20). A safe setup starts redacted by default.
The key is separating agent telemetry from the telemetry of the code under test. The docs state that Claude Code OTEL_* variables are not automatically passed to subprocesses such as Bash, hooks, MCP servers, and language servers.
Citation capsule: OpenTelemetry for Claude Code should begin with redacted metrics and events. The docs state default intervals of 60 seconds for metrics and 5 seconds for logs, while separating agent variables from subprocesses.
If you already use hooks, treat them as event sources. They record decisions, blocks, and exceptions. The dashboard should show useful blocking, not just successful runs.
Where do cost, tokens, and context fit in the dashboard?
In 2026, Claude Code Docs in "Monitoring" lists input_tokens, output_tokens, cache_read_tokens, and cache_creation_tokens on API request events (Claude Code Docs, "Monitoring", 2026, retrieved 2026-07-20). The dashboard needs consumption by task, not only monthly total.
Tokens alone explain little. Compare cost with task type, code area, retry count, tool count, and final result. An expensive run that produced proof may be acceptable. A cheap run that opened a PR without tests is still risk.
Citation capsule: An agent dashboard should combine tokens, tool use, retry count, and result. Claude Code exposes token and cache fields on API events; the metric becomes useful only when tied to PR, test, block, and operational risk.
What minimum CI contract fits this week?
In 2026, arXiv in "AI Agent Pull Requests on GitHub" measured textual conflict at 41.7% for cross-agent pairs versus 19.8% for intra-agent pairs when replaying real merges (arXiv, "AI Agent Pull Requests on GitHub", 2026, retrieved 2026-07-20). The minimum contract should reveal concurrency before it becomes expensive conflict.
Start with one agent-run.json artifact for each agentic workflow. It should record tool, base commit, final commit, permission mode, enabled MCP, verification commands, result, retries, and whether human review happened.
| Field | Practical rule |
|---|---|
| Identity | Declare tool, session, and accountable human author. |
| Permission | Record sandbox, network, MCP, and exceptions. |
| Proof | List command run and verifiable result. |
| Cost | Keep tokens, retries, and duration in a restricted artifact. |
| Stop | Explain whether the agent finished, failed, or requested review. |
Citation capsule: The minimum CI observability contract is one artifact per run and one summary per PR. Since cross-agent conflict reached 41.7% in the arXiv study, every task should declare identity, permission, proof, cost, and stop rule.
FAQ about coding agent observability
Does observability replace human review?
No. In 2026, arXiv in "These Aren't the Reviews You're Looking For" concluded that AI-generated PRs receive more automation-mediated interaction than independent human evaluation (arXiv, "These Aren't the Reviews You're Looking For", 2026, retrieved 2026-07-20). Observability improves review, but it does not decide architecture.
Can I store the full agent transcript?
Only with clear access control. In 2026, Claude Code Docs in "Data usage" says local transcripts are stored in plaintext under ~/.claude/projects/ for 30 days by default (Claude Code Docs, "Data usage", 2026, retrieved 2026-07-20). For CI, prefer a restricted artifact and redacted summary.
Is Codex JSONL enough for audit?
It helps, but not alone. In 2026, OpenAI documents JSONL with events and item types, including commands, file changes, MCP calls, and web searches (OpenAI Developers, "Non-interactive mode", 2026, retrieved 2026-07-20). You still need to map events to risk, PR, and human decision.
Which metrics should enter the dashboard first?
Start with cost, tokens, duration, retries, tool calls, permission blocks, and CI result. In 2026, Claude Code documents API events with tokens, estimated cost, duration, retries, and request id (Claude Code Docs, "Monitoring", 2026, retrieved 2026-07-20). Then segment by team and task type.
Closing
A coding agent without observability is fast until the first hard question. Who authorized it? Which tool ran? Why did it cost so much? Which test proved it? Which MCP participated? Without answers, the team reviews feeling, not engineering.
Start small: restricted JSONL, redacted OTel, PR summary, and event contract. The goal is not to surveil people. It is to make the agentic loop debuggable, cheaper to review, and safe enough to join the SDLC.
Sources consulted
- GitHub Blog, "Agent pull requests are everywhere. Here's how to review them", retrieved 2026-07-20, https://github.blog/ai-and-ml/generative-ai/agent-pull-requests-are-everywhere-heres-how-to-review-them/
- arXiv, "AI Agent Pull Requests on GitHub: Frequency, Structure, and Merge Conflict Rates", retrieved 2026-07-20, https://arxiv.org/abs/2607.04697
- arXiv, "These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests", retrieved 2026-07-20, https://arxiv.org/abs/2605.02273
- OpenAI Developers, "Non-interactive mode", retrieved 2026-07-20, https://developers.openai.com/codex/non-interactive-mode
- Claude Code Docs, "Monitoring", retrieved 2026-07-20, https://code.claude.com/docs/en/monitoring-usage
- Claude Code Docs, "Hooks reference", retrieved 2026-07-20, https://code.claude.com/docs/en/hooks
- Claude Code Docs, "Data usage", retrieved 2026-07-20, https://code.claude.com/docs/en/data-usage
