[ab-advisor] Experiment campaign for ci-coach: A/B test prompt_style

🧪 Experiment Campaign: ci-coach

Workflow file: .github/workflows/ci-coach.md
Selected dimension: prompt_style
Triggered by: ab-testing-advisor on 2026-05-15

Background

The ci-coach workflow is a daily CI optimization coach that analyzes workflow runs and proposes efficiency improvements via pull request. Its current prompt is highly prescriptive — it defines six numbered phases with detailed time budgets, extensive safety checklists, and repeated prohibitions. This verbosity may be consuming more tokens than necessary without improving output quality, making prompt_style the highest-impact dimension to experiment on.

Hypothesis

H0 (null): Switching from the current detailed, phase-structured prompt to a concise goal-oriented prompt does not change the quality of CI optimization proposals (measured by PR merge rate and output completeness score).

H1 (alternative): A concise prompt reduces token consumption by ≥25% while maintaining or improving proposal quality, because the agent can apply its own reasoning structure rather than spending tokens parsing a prescribed multi-phase framework.

Experiment Configuration

Add the following experiments: block to the workflow frontmatter:

experiments:
  prompt_style:
    variants: [detailed, concise]
    description: "Tests whether a condensed goal-oriented prompt produces equivalent or better CI optimization proposals compared to the current verbose phase-structured prompt"
    hypothesis: "H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality"
    metric: token_count_per_run
    secondary_metrics: [pr_created_rate, run_duration_ms, output_word_count]
    guardrail_metrics:
      - name: run_success_rate
        direction: min
        threshold: 0.85
      - name: empty_output_rate
        direction: max
        threshold: 0.05
    min_samples: 20
    weight: [50, 50]
    start_date: "2026-05-15"
    analysis_type: mann_whitney
    issue: 0

Variant descriptions:

• detailed: Baseline — current verbose prompt with 6 numbered phases, time budgets, repeated safety checklists, and prescriptive step-by-step instructions (~1,800 words).
• concise: Condensed prompt (~350 words) that states the goal, lists available data sources, defines output format, and delegates phase structure to the agent's judgment.

Workflow Changes Required

Wrap the current verbose phase framework in a detailed conditional and add a short concise variant block.

Before (current prompt body — excerpt):

## Analysis Framework

Follow the optimization strategies defined in the `ci-optimization-strategies` shared module:

### Phase 1: Study CI Configuration (5 minutes)
...
### Phase 2: Analyze Test Coverage (10 minutes)
...
### Early Exit Gate (mandatory after Phase 2)
...
### Phase 3: Identify Optimization Opportunities (10 minutes)
...
### Phase 4: Cost-Benefit Analysis (3 minutes)
...
### Phase 5: Implement and Validate Changes (8 minutes)
...
### Phase 6: No Changes Path
...

After (with experiment conditionals):

{{#if experiments.prompt_style == "concise" }}
## Task

Analyze CI workflows (`.github/workflows/ci.yml`, `cgo.yml`, `cjs.yml`) using pre-downloaded data in `/tmp`. Identify the top 3 highest-impact optimizations for cost and speed. If you find actionable improvements, make focused changes, validate with `make lint && make build && make test-unit && make recompile`, and create a PR. If CI is healthy, call `noop`. Never modify test code to hide failures.

**Data**: `/tmp/ci-summary.json` (start here), `/tmp/ci-runs.json`, `/tmp/ci-artifacts/`.
**PR format**: Use `###` headers; keep under 600 words; use `<details>` for long diffs.
{{else}}
## Analysis Framework

... (existing verbose phase content unchanged) ...
{{/if}}

Success Metrics

Statistical Design

• Variants: detailed (baseline), concise (treatment)
• Assignment: Round-robin via gh-aw experiments runtime (cache-based)
• Minimum runs per variant: 20 (estimated from ~5 runs/week on weekdays = ~4 weeks per variant)
• Expected experiment duration: ~8 weeks total (4 weeks × 2 variants interleaved)
• Analysis approach: Mann-Whitney U test on token counts (non-parametric; suitable for skewed cost distributions)

Implementation Steps

• Add experiments: section to frontmatter (update issue: field after this issue is created)
• Add conditional blocks to workflow prompt body using {{#if experiments.prompt_style == "concise" }}
• Run gh aw compile ci-coach to regenerate lock file
• Monitor experiment artifact uploaded per run to /tmp/gh-aw/experiments/state.json
• After ≥20 runs per variant, analyze variant distribution via workflow run artifacts
• Document findings and promote winning variant

References

• A/B Testing in gh-aw
• Workflow file: .github/workflows/ci-coach.md

Infrastructure note: analysis_type, tags, and notify fields are fully implemented in both pkg/workflow/compiler_experiments.go and actions/setup/js/pick_experiment.cjs — no infrastructure improvements are needed at this time.

Generated by 🧪 Daily A/B Testing Advisor · ● 3.6M · ◷

• expires on May 29, 2026, 11:14 AM UTC