[token-consumption] Daily AIC Consumption Report - 2026-08-25

Executive Summary

Status: Report Generation Failed - Backend Connectivity Unavailable

The daily AIC consumption report for 2026-08-25 cannot be generated because neither Sentry nor Grafana MCP servers are configured in the workflow execution environment. The workflow has no access to the telemetry backends required to query OpenTelemetry spans containing AIC data.

Impact:

Zero AIC consumption visibility for the last 24 hours
No workflow-level cost attribution available
Cannot identify high-consumption workflows or anomalies
Daily cost monitoring and optimization workflows blocked

Root Cause Analysis

Missing MCP Server Configuration

The workflow's .goose/mcp.json contains only two servers:

github (for GitHub API read operations)
safeoutputs (for safe GitHub write operations)

Missing critical servers:

sentry MCP server (required for Sentry trace/span queries)
grafana MCP server (required for Grafana/Tempo trace queries)

Evidence

$ cat .goose/mcp.json | jq -r '.mcpServers | keys[]'
github
safeoutputs

No Sentry or Grafana CLI commands available:

$ sentry-mcp find_organizations
bash: sentry-mcp: command not found

$ grafana-mcp --help
bash: grafana-mcp: command not found

Expected Configuration

According to the OpenTelemetry guide, the workflow needs:

Sentry MCP Server

mcp-servers:
  sentry:
    command: "npx"
    args: ["@sentry/mcp-server@0.33.0"]
    allowed:
      - whoami
      - find_organizations
      - find_projects
      - get_trace_details
      - search_events
      - list_events
    env:
      SENTRY_ACCESS_TOKEN: ${{ secrets.SENTRY_ACCESS_TOKEN }}
      SENTRY_HOST: ${{ env.SENTRY_HOST || 'sentry.io' }}

Grafana MCP Server (for Tempo)

mcp-servers:
  grafana:
    command: "npx"
    args: ["@grafana/mcp-server@latest"]
    allowed:
      - list_datasources
      - tempo_get-attribute-names
      - tempo_get-attribute-values
      - tempo_traceql-search
      - tempo_get-trace
    env:
      GRAFANA_URL: ${{ secrets.GRAFANA_URL }}
      GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}

Missing Authentication Secrets

The workflow requires these secrets to authenticate with telemetry backends:

Sentry

SENTRY_ACCESS_TOKEN - Sentry API token with read access to spans/events
SENTRY_HOST - Sentry instance URL (default: sentry.io)

Grafana/Tempo

GRAFANA_URL - Grafana instance URL
GRAFANA_TOKEN - Grafana API token with datasource read access

Status: These secrets are not configured in the workflow frontmatter or visible in the execution environment.

Observability Architecture Context

How AIC Data is Emitted

From actions/setup/js/send_otlp_span.cjs:

// For jobs that own token usage (agent, detection, engine), gh-aw.aic is ALWAYS
// emitted as a numeric attribute — defaulting to 0 when no data is available.
// This guarantees Sentry EAP infers the field as numeric (not string) so that
// sum()/avg()/percentile() aggregations work without manual schema configuration,
// and Tempo indexes it so { span."gh-aw.aic" > 0 } is queryable immediately.

Key AIC fields emitted on spans:

gh-aw.aic (primary, always numeric via doubleValue)
gh_aw.aic (compatibility alias)
agent_usage.aic (legacy field)
aic (generic fallback)

Workflow attribution fields:

gh-aw.workflow.name
github.workflow
github.run_id
gh-aw.run.id

Expected Query Pattern

The workflow should:

Sentry: Query dataset: spans with span.op:ai* or span.op:gen_ai*, extracting gh-aw.aic and workflow metadata
Grafana/Tempo: Use TraceQL {.service.name =~ "gh-aw.*" && span."gh-aw.aic" > 0} to find AIC-bearing spans
Aggregate: Group by workflow name, sum AIC, calculate per-workflow and per-run metrics
Report: Top 10 consumers, P95 AIC/event, anomaly detection

Current status: Step 1 (connectivity) is blocked, so no queries can execute.

Recommendations

Immediate Action (Unblock Reporting)

1. Add Sentry MCP Server to Workflow

Edit .github/workflows/daily-aic-consumption-report.md frontmatter:

mcp-servers:
  sentry:
    command: "npx"
    args: ["@sentry/mcp-server@0.33.0"]
    allowed:
      - whoami
      - find_organizations
      - find_projects
      - list_events
      - search_events
      - get_trace_details
    env:
      SENTRY_ACCESS_TOKEN: ${{ secrets.SENTRY_ACCESS_TOKEN }}
      SENTRY_HOST: "sentry.io"

2. Add Grafana MCP Server to Workflow

  grafana:
    command: "npx"
    args: ["@grafana/mcp-server@latest"]
    allowed:
      - list_datasources
      - tempo_get-attribute-names
      - tempo_get-attribute-values
      - tempo_traceql-search
      - tempo_get-trace
    env:
      GRAFANA_URL: ${{ secrets.GRAFANA_URL }}
      GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}

3. Configure Required Secrets

Add these repository secrets via Settings → Secrets and variables → Actions:

SENTRY_ACCESS_TOKEN
GRAFANA_URL
GRAFANA_TOKEN

4. Whitelist Network Destinations

Add to workflow frontmatter network.allowed:

network:
  allowed:
    - "*.sentry.io"
    - "*.grafana.com"  # or your Grafana instance domain

Longer-Term Improvements

5. Create a Shared Telemetry Import

Extract common MCP server configuration into .github/workflows/shared/telemetry-backends.md so all observability workflows inherit the same connectivity without duplication.

6. Add Workflow Self-Check

Add a connectivity validation step that calls sentry whoami and grafana list_datasources at workflow start, failing fast with a clear error if backends are unreachable.

7. Fallback to Local Artifacts

When backends are unavailable, fall back to querying local OTLP JSONL artifacts (otel.jsonl) uploaded in the agent artifact of recent workflow runs. This provides degraded but still useful AIC visibility.

Workflow Context

Repository: github/gh-aw
Run ID: 32845054130
Actor: pelikhan
Workflow: daily-aic-consumption-report.yml
Time Window: Last 24 hours (2026-08-24 12:00 UTC to 2026-08-25 12:00 UTC)
Report Date: 2026-08-25

Labels: token-consumption, observability, workflow-config, blocked

Generated by 📊 Daily AIC Consumption Report (Sentry + Grafana OTel) · goose · sonnet45 · 65 AIC · ⊞ 6.7K · ◷

expires on Aug 26, 2026, 4:04 AM UTC-08:00

[token-consumption] Daily AIC Consumption Report - 2026-08-25 - Backend Connectivity Failure