[token-consumption] Daily AIC Consumption Report - 2026-08-24

Executive Summary

Unable to generate AIC consumption report due to critical telemetry infrastructure gaps. Both Sentry and Grafana MCP servers are unavailable, and no local telemetry artifacts exist as fallback. This report documents the configuration issues and provides actionable recommendations.

Infrastructure Status

Critical Missing Components

What Was Expected

The workflow declares these imports in .github/workflows/daily-token-consumption-report.md:

imports:
  - shared/mcp/sentry.md
  - shared/mcp/grafana.md

These imports define MCP servers requiring:

• Sentry MCP: SENTRY_ACCESS_TOKEN secret
• Grafana MCP: GRAFANA_URL and GRAFANA_SERVICE_ACCOUNT_TOKEN secrets

What Was Found

1. Lock file MCP servers (.github/workflows/daily-token-consumption-report.lock.yml):

   • Only github and safeoutputs MCP servers compiled
   • No sentry or grafana MCP servers present

2. Runtime environment:

   • SENTRY_ACCESS_TOKEN: not set
   • GRAFANA_URL: not set
   • GRAFANA_SERVICE_ACCOUNT_TOKEN: not set
   • OTLP environment variables are configured but point to endpoints requiring authentication

3. OTLP endpoints configured (for span export, not query):

   • Sentry: https://o205451.ingest.us.sentry.io/api/4511347087179777/integration/otlp
   • Grafana: endpoint configured in GH_AW_OTLP_ENDPOINTS but query credentials missing

Root Cause Analysis

MCP Server Configuration Gap

The workflow markdown imports shared/mcp/sentry.md and shared/mcp/grafana.md, which declare MCP server configurations. However:

1. Secrets not configured: The required GitHub secrets are not available in the repository or workflow environment.
2. Compilation failure: Without the secrets, the compiler cannot validate or include these MCP servers in the lock file.
3. No fallback mechanism: The workflow has no fallback to query telemetry through alternative means (HTTP API, local artifacts, etc.).

OTLP vs. Query Separation

The workflow correctly exports telemetry spans to Sentry and Grafana via OTLP (OpenTelemetry Protocol), but exporting spans and querying historical spans require different authentication:

• Export (working): Uses x-sentry-auth header configured in OTEL_EXPORTER_OTLP_HEADERS
• Query (missing): Requires SENTRY_ACCESS_TOKEN for Sentry MCP server and GRAFANA_SERVICE_ACCOUNT_TOKEN for Grafana MCP server

Impact

• No AIC visibility: Cannot track AI Credits consumption across workflows
• No cost optimization: Cannot identify high-cost workflows or anomalies
• No trend analysis: Cannot compare daily/weekly/monthly usage patterns
• Blocked reporting: Daily automation workflow fails without telemetry access

Recommendations

1. Configure Required Secrets (Priority: Critical)

Add these GitHub repository secrets:

# Sentry MCP Server
SENTRY_ACCESS_TOKEN: <user-auth-token-with-scopes>
  # Required scopes: org:read, project:read, event:read, team:read

# Grafana MCP Server  
GRAFANA_URL: <grafana-instance-url>
  # Example: https://your-org.grafana.net

GRAFANA_SERVICE_ACCOUNT_TOKEN: <service-account-token>
  # Required permissions: read access to Tempo datasource

2. Verify Secret Availability (Priority: High)

After adding secrets, verify they're accessible to the workflow:

# In workflow: test secret availability
- name: Verify secrets
  run: |
    echo "SENTRY_ACCESS_TOKEN: ${SENTRY_ACCESS_TOKEN:+set}"
    echo "GRAFANA_URL: ${GRAFANA_URL:+set}"
    echo "GRAFANA_SERVICE_ACCOUNT_TOKEN: ${GRAFANA_SERVICE_ACCOUNT_TOKEN:+set}"

3. Recompile Workflow (Priority: High)

After configuring secrets, recompile the workflow to include MCP servers in the lock file:

make recompile
# or
gh aw compile .github/workflows/daily-token-consumption-report.md

Verify the lock file includes both MCP servers:

grep -A 5 "mcp_servers:" .github/workflows/daily-token-consumption-report.lock.yml

Expected output should include sentry and grafana entries.

4. Implement Fallback Query Mechanism (Priority: Medium)

Add a fallback mechanism when MCP servers are unavailable:

• Option A: Query Sentry HTTP API directly using OTLP credentials (if they include read permissions)
• Option B: Export local telemetry artifacts as workflow artifacts for offline analysis
• Option C: Use GitHub API to estimate token usage from workflow run logs (approximate, not authoritative)

5. Add MCP Server Health Check (Priority: Low)

Add a workflow step to validate MCP server connectivity before running the analysis:

- name: Health check MCP servers
  run: |
    # Test Sentry connectivity
    sentry-mcp whoami || echo "Sentry MCP unavailable"
    
    # Test Grafana connectivity  
    grafana-mcp list_datasources || echo "Grafana MCP unavailable"

References

• Workflow source: .github/workflows/daily-token-consumption-report.md
• Sentry MCP config: .github/workflows/shared/mcp/sentry.md
• Grafana MCP config: .github/workflows/shared/mcp/grafana.md
• Current run: github/gh-aw run ID 32724530162

Next Actions

1. Repository admin: Configure the three required secrets listed in Recommendation #1
2. Workflow maintainer: Recompile the workflow after secrets are configured
3. This workflow: Re-run this workflow after infrastructure is fixed
4. Engineering team: Consider implementing fallback mechanisms for resilience

This report was generated by the Daily AIC Consumption Report workflow. The workflow is designed to aggregate and analyze AI Credits consumption across all agentic workflows, but cannot function without proper telemetry infrastructure.

Generated by 📊 Daily AIC Consumption Report (Sentry + Grafana OTel) · goose · sonnet45 · 82.2 AIC · ⊞ 6.7K · ◷

• expires on Aug 25, 2026, 4:05 AM UTC-08:00