[agentrx-optimizer] Daily Workflow Optimization

Executive Summary

AgentRx-style analysis of the last 24-48h of gh-aw agent runs (284 runs total, 29 failures) found that the pre-downloaded, usage-only telemetry bundle over-classifies failures as generic driver_exit (24 of 29) because it lacks per-turn agent transcript data. Re-fetching the agent artifact set for the single worst-offending workflow, Design Decision Gate 🏗️ (4 of 29 failures, the most of any workflow), reclassified all of its failures as agent_logic and exposed a concrete, repeating root cause: the agent gets stuck retrying push_to_pull_request_branch against a branch state the tool can never accept (detached HEAD / unfetched branch / disallowed files), burns its entire 30-invocation budget, and the job dies on a 429 "Maximum LLM invocations exceeded" error. This pattern accounts for 4 of the last 6 failures (67%) in that workflow.

AgentRx Evidence

Critical step: Execute Claude Code CLI (agent job) → repeated push_to_pull_request_branch tool calls, terminal_reason api_error
Failure category: agent_logic (invocation-cap exhaustion via unrecoverable tool retry loop)
Frequency / impact: 4 of 6 Design Decision Gate failures in the last 2 days (67%); each burns ~15–20 min of runner time and up to 30/30 LLM invocations before failing
Representative run IDs: §32787203288, §32771830842, §32776331176, §32687694593

AgentRx Artifacts

IR summary (trajectory_ir.json, stage ir): 8 trajectories normalized (4 invocation-cap failures, 2 secondary "allowed-files" push failures, 2 successful baselines) from .github/workflows/design-decision-gate.lock.yml runs, run-dir=/tmp/gh-aw/agent/agentrx/runs/gh-aw-daily.

Static/dynamic invariant + check stages: failed with RuntimeError: Static invariants JSON parse failed — the Copilot CLI endpoint used by AgentRx for invariant generation has no authentication in this sandbox (Error: No authentication information found). No invariant/checker output was produced; this section is empty by necessity, not by omission.

Judge stage: completed but produced num_judges: 0 for all 8 trajectories (no LLM judge endpoint available) — classification below is derived from direct inspection of the raw 5_agent.txt transcripts, not from AgentRx's LLM judge.

Manual transcript findings (grounding for the recommendation):

violation	evidence	fix_type	rationale
Agent retries `push_to_pull_request_branch` repeatedly against a branch in detached-HEAD / not-yet-fetched state until the 30-invocation cap is hit	Runs §32787203288, §32771830842, §32776331176: tool returns `ERR_CONFIG: ... detached-HEAD state and no GitHub environment variables ... available`, then `ERR_SYSTEM: fatal: Needed a single revision`, then `could not be fetched (no credentials for private repositories)` — 3+ distinct error shapes across attempts, ending in `API Error: Request rejected (429) · Maximum LLM invocations exceeded (30 / 30)`	adding precondition checks before expensive tools	The underlying condition (branch not fetched / detached HEAD) cannot be fixed by retrying the same tool call with different arguments — it requires a workflow-level fix (fetch the PR head branch during checkout). Burning the full invocation budget on a structurally unfixable retry converts a fast, cheap failure into the most expensive possible one.
Bundle push rejected for modifying files outside `allowed-files` config	Run §32766584101: `##[error]Cannot push to pull request branch: bundle modifies files outside the allowed-files list (pkg/cli/add_interactive_git.go)`	adding precondition checks before expensive tools	Same shape as above at a smaller scale (2/6 failures): the agent only discovers the allowed-files boundary after attempting the push, in the `safe_outputs` job, rather than being told the boundary up front.
Prompt references skills not registered for this engine (`Unknown skill: adr-report-templates`, `Unknown skill: reporting`) early in run §32787203288	Two consecutive failed skill invocations before the agent falls back to manual steps	prompt tightening to reduce invalid tool invocations	Minor relative to the invocation-cap issue, but adds avoidable wasted turns at the start of every run of this workflow.
24/29 (83%) of all-repo failures in the base 24h summary are labeled generic `driver_exit` with `turns: 0`, vs. `agent_logic` with real turn counts once `agent` artifacts are fetched	Compare `run-32787203288` in `/tmp/gh-aw/aw-mcp/logs/summary.json` (`failure_kind: driver_exit`, `turns` absent) vs. the same run via `logs` MCP tool with `artifacts:["agent"]` (`failure_kind: agent_logic`, `turns: 32`)	adding missing telemetry attributes for better triage	The default observability bundle for this repo's runs doesn't include agent transcripts, so most failures are triaged as an undifferentiated "driver exited" bucket even when the real cause (a normal agent turn loop that ran out of budget) is fully recoverable from data the CI job already produced.

Known limitations: static/dynamic/check/judge AgentRx stages require an authenticated LLM endpoint (Copilot CLI) not available in this sandbox; the table above substitutes direct transcript evidence gathered via the logs MCP tool (artifacts:["agent"], 2 calls, workflow-scoped) per the fallback guidance.

Recommended Optimization

Add a precondition/fail-fast guard around push_to_pull_request_branch in the Design Decision Gate workflow (and any other safe-output workflow that calls it from a PR-check context) so a structurally-unfixable branch state fails in 1 attempt instead of consuming the full 30-invocation budget.

Concretely:

In .github/workflows/design-decision-gate.md (source for design-decision-gate.lock.yml), ensure the PR head branch is fetched during checkout (checkout.fetch including the head ref) so push_to_pull_request_branch never hits the "not present in checkout and could not be fetched" branch of the error.
Have the safe-outputs MCP server (or the workflow prompt) surface the allowed-files list to the agent before it starts editing, so files like pkg/cli/add_interactive_git.go are never staged for a bundle that will be rejected.
Add a turn-budget guard in the prompt: after 1–2 failed push_to_pull_request_branch attempts with the same ERR_CONFIG/ERR_SYSTEM class of error, instruct the agent to stop retrying and report the blocker rather than trying variant commands (as seen in §32776331176's repeated --help / cherry-pick attempts).

This is the highest-impact fix because it is the single most frequent failure mode observed (67% of this workflow's recent failures), it's the most expensive per occurrence (full budget burn + ~15–20 min runner time each time), and the fix is entirely in workflow configuration/prompt — no AgentRx-stage dependency required to validate it.

Secondary, lower-priority item: switch the daily log-collection default to include the agent artifact set (not just usage) so failure_kind classification stops defaulting to driver_exit for lack of turn data — this would make future daily optimizer runs more accurate without needing a manual re-fetch per workflow.

Where to implement: .github/workflows/design-decision-gate.md (checkout/fetch config + prompt), and the safe-outputs MCP server's push_to_pull_request_branch handler (surface allowed-files proactively).

Validation Plan

After the fix, watch the next 5–10 Design Decision Gate runs via status/logs (workflow-scoped, artifacts:["agent"]) for push_to_pull_request_branch failures.
Success metric: 0 occurrences of terminal_reason: api_error / "Maximum LLM invocations exceeded" for this workflow over the next 2 days, and any push_to_pull_request_branch failure resolves within 1–2 attempts (turns roughly at the ~15-turn baseline of successful runs, e.g. §32804607534 at 28 turns, rather than hitting the 30-turn cap).
Expected duration improvement: failing runs currently average ~14 min (10–20 min range); a fast-fail should bring failed runs down to the ~7 min baseline of successful runs that don't touch ADR files.

References

Generated by ⚡ Daily AgentRx Trace Optimizer · claude · agent · 206.9 AIC · ⌖ 12.1 AIC · ⊞ 8.8K · ◷

expires on Aug 31, 2026, 10:00 PM UTC-08:00