[ab-advisor] Experiment campaign for agent-performance-analyzer: A/B test caveman

🧪 Experiment Campaign: agent-performance-analyzer

Workflow file: .github/workflows/agent-performance-analyzer.md
Selected dimension: caveman_mode
Triggered by: ab-testing-advisor on 2026-05-19

Background

The Agent Performance Analyzer is a meta-orchestrator that analyzes AI agent performance across the repository. It currently uses a comprehensive 648-line prompt with detailed instructions across 5 phases. This experiment tests whether extreme prompt compression (the "caveman" principle: "why use many token when few do trick") preserves output quality, allowing us to identify prompt verbosity waste and reduce token consumption without sacrificing effectiveness.

Hypothesis

H0 (Null): Prompt compression via caveman mode does not significantly change the effective token count or output quality.

H1 (Alternative): Caveman mode reduces effective token consumption by ≥20% while maintaining output quality (measured by discussion engagement, issue creation rate, and agent assessment completeness) within 10% of the baseline.

Experiment Configuration

Add the following experiments: block to the workflow frontmatter (use the rich object form so all metadata is self-documenting):

experiments:
  prompt_compression:
    variants: [verbose, caveman]
    description: "Test whether extreme prompt compression preserves output quality for meta-orchestrator workflows"
    hypothesis: "H0: no change in effective_tokens. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%"
    metric: effective_tokens
    secondary_metrics: [run_duration_seconds, issues_created, discussion_engagement_score, assessment_completeness_score]
    guardrail_metrics:
      - name: run_success_rate
        direction: min
        threshold: ">=0.90"
      - name: output_quality_score
        direction: min
        threshold: ">=0.70"
    min_samples: 14
    weight: [50, 50]
    start_date: "2026-05-20"
    analysis_type: mann_whitney
    tags: [cost_optimization, prompt_engineering, meta_orchestrator]
    notify:
      issue: __ISSUE_NUMBER__
    issue: __ISSUE_NUMBER__

Variant descriptions:

verbose: Current comprehensive prompt with detailed instructions, examples, and phase breakdowns (baseline)
caveman: Heavily compressed prompt using minimal tokens - removes redundancy, uses terse phrasing, eliminates examples where unnecessary

Workflow Changes Required

The correct handlebars syntax for comparing against a specific variant value is:

{{#if (eq experiments.prompt_compression "caveman")}}
  ...caveman content...
{{else}}
  ...verbose content...
{{/if}}

Wrap the main instruction body (lines 42-610 approximately) in a conditional block. The caveman variant should:

Remove redundant headers and explanations - combine related sections
Use terse phrasing - "Analyze agent outputs: clarity, accuracy, completeness, actionability" vs. current verbose descriptions
Remove examples - the agent knows how to format issues and discussions
Compress phase instructions - "Phase 1 (10m): Load metrics from /tmp/gh-aw/repo-memory/default/metrics/, query recent agent outputs, build agent profiles" vs. current step-by-step breakdown
Keep critical structure - maintain phase numbers, key responsibilities, shared memory paths

Example transformation (lines 176-250):

Before (verbose):

## Workflow Execution

Execute these phases each run:

## Shared Memory Integration

**Access shared repo memory at `/tmp/gh-aw/repo-memory/default/`**

This workflow shares memory with other meta-orchestrators (Campaign Manager and Workflow Health Manager) to coordinate insights and avoid duplicate work.

**Shared Metrics Infrastructure:**

The Metrics Collector workflow runs daily and stores performance metrics in a structured JSON format:

1. **Latest Metrics**: `/tmp/gh-aw/repo-memory/default/metrics/latest.json`
   - Most recent daily metrics snapshot
   - Quick access without date calculations
   - Contains all workflow metrics, engagement data, and quality indicators

2. **Historical Metrics**: `/tmp/gh-aw/repo-memory/default/metrics/daily/YYYY-MM-DD.json`
   - Daily metrics for the last 30 days
   - Enables trend analysis and historical comparisons
   - Calculate week-over-week and month-over-month changes

After (caveman):

## Execution

**Shared memory:** `/tmp/gh-aw/repo-memory/default/`  
**Metrics:** `metrics/latest.json` (current), `metrics/daily/YYYY-MM-DD.json` (30d history)  
**Coordinate with:** Campaign Manager, Workflow Health Manager via `shared-alerts.md`

Success Metrics

Metric	Type	Target
effective_tokens	Primary	Reduce by ≥20%
run_duration_seconds	Secondary	Maintain or improve
issues_created	Secondary	Within 10% of baseline
discussion_engagement_score	Secondary	Within 10% of baseline
assessment_completeness_score	Secondary	Within 10% of baseline
run_success_rate	Guardrail	Must not drop below 90%
output_quality_score	Guardrail	Must not drop below 70%

Statistical Design

Variants: verbose (baseline), caveman (compressed)
Assignment: Round-robin via gh-aw experiments runtime (cache-based)
Minimum runs per variant: 14 (based on daily trigger = 2 weeks of data)
Expected experiment duration: 28 days (14 runs per variant)
Analysis approach: Mann-Whitney U test (non-parametric, robust to token count distribution skew)

Implementation Steps

Add experiments: section to frontmatter
Create caveman variant of main prompt body using {{#if (eq experiments.prompt_compression "caveman")}}
Run gh aw compile agent-performance-analyzer to regenerate lock file
Monitor experiment artifact uploaded per run to /tmp/gh-aw/experiments/state.json
After 28 days (14 runs/variant), analyze variant distribution via workflow run artifacts
Document findings and promote winning variant

References

A/B Testing in gh-aw
Workflow file: .github/workflows/agent-performance-analyzer.md

Note: The experiment infrastructure is complete - all fields (analysis_type, tags, notify) are fully implemented in both the Go compiler and JavaScript picker.

Generated by 🧪 Daily A/B Testing Advisor · ● 6.1M · ◷

expires on Jun 2, 2026, 11:37 AM UTC