Skip to content

secret-digger-claude: switch to Haiku, lower max-turns to 4#1956

Merged
lpcox merged 3 commits intomainfrom
copilot/optimize-claude-token-usage-again
Apr 13, 2026
Merged

secret-digger-claude: switch to Haiku, lower max-turns to 4#1956
lpcox merged 3 commits intomainfrom
copilot/optimize-claude-token-usage-again

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 12, 2026

Secret Digger (Claude) accounts for 63% of total Claude spend ($10.71/21 runs) due to Sonnet 4.6 cache write charges (~$0.43/run) that provide no cross-run benefit — Anthropic's 5-minute cache TTL vs. hourly schedule means every run pays full cold-cache cost.

Changes

  • secret-digger-claude.md

    • max-turns: 84 (actual usage is always 3; this is a runaway guard, not a functional limit)
    • Added GH_AW_MODEL_AGENT_CLAUDE: "claude-haiku-4-5-20251001" in engine.env — workflow-scoped override, does not affect smoke-claude or security-guard
  • secret-digger-claude.lock.yml — recompiled + post-processed; reflects --max-turns 4 and GH_AW_MODEL_AGENT_CLAUDE: claude-haiku-4-5-20251001 in the agent job env

engine:
  id: claude
  max-turns: 4
  env:
    BASH_DEFAULT_TIMEOUT_MS: "1800000"
    BASH_MAX_TIMEOUT_MS: "1800000"
    GH_AW_MODEL_AGENT_CLAUDE: "claude-haiku-4-5-20251001"

Expected impact

Metric Before After
Cost/run $0.51 ~$0.018
21-run period $10.71 ~$0.38
Cache write cost/run ~$0.43 ~$0.012

Haiku is well-suited to this task (bash-heavy read-only shell forensics). The task structure and cache-memory persistence are unchanged.

…s to 4

- Add GH_AW_MODEL_AGENT_CLAUDE env override to use claude-haiku-4-5-20251001
  (workflow-scoped, does not affect other Claude workflows)
- Reduce max-turns from 8 to 4 (actual usage is always 3 turns)
- Recompile lock file to reflect these changes

Expected impact: -96% cost/run ($0.51 → ~$0.018)

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b2159c34-4a33-4d0d-a898-469353139ffd

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot AI changed the title [WIP] Optimize Claude token usage for cost efficiency secret-digger-claude: switch to Haiku, lower max-turns to 4 Apr 12, 2026
Copilot AI requested a review from lpcox April 12, 2026 23:50
@lpcox
Copy link
Copy Markdown
Collaborator

lpcox commented Apr 13, 2026

@copilot resolve the merge conflicts in this pull request

…de-token-usage-again

# Conflicts:
#	.github/workflows/secret-digger-claude.lock.yml

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 13, 2026

@copilot resolve the merge conflicts in this pull request

Resolved in bbd10ac. The conflict was in the lock file — main added --session-state-dir to the awf command while our branch changed --max-turns 8 to 4. The merge keeps both: --session-state-dir /tmp/gh-aw/sandbox/agent/session-state from main and --max-turns 4 + GH_AW_MODEL_AGENT_CLAUDE: claude-haiku-4-5-20251001 from our branch.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 85.24% 85.33% 📈 +0.09%
Statements 85.11% 85.20% 📈 +0.09%
Functions 87.57% 87.57% ➡️ +0.00%
Branches 77.58% 77.63% 📈 +0.05%
📁 Per-file Coverage Changes (1 files)
File Lines (Before → After) Statements (Before → After)
src/docker-manager.ts 86.3% → 86.6% (+0.31%) 85.8% → 86.2% (+0.31%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test Results

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

@github-actions
Copy link
Copy Markdown
Contributor

🔥 Smoke Test Results

Test Result
GitHub MCP (list merged PRs) ✅ PR #1952 fetched
GitHub.com connectivity ✅ HTTP 200
File write/read ⚠️ Template vars not injected — skipped

Overall: PASS

@lpcox, @Copilot — all critical checks passed.

📰 BREAKING: Report filed by Smoke Copilot

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test: GitHub Actions Services Connectivity ✅

All checks passed:

Service Check Result
Redis host.docker.internal:6379 PING PONG
PostgreSQL host.docker.internal:5432 pg_isready ✅ accepting connections
PostgreSQL smoketest db SELECT 1 ✅ returned 1

Note: redis-cli was unavailable (apt/sudo not accessible), so Redis was tested via Python socket — response was +PONG.

🔌 Service connectivity validated by Smoke Services

@github-actions

This comment has been minimized.

@github-actions
Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note: Java required a custom local Maven repository path (-Dmaven.repo.local) since ~/.m2 was root-owned in this runner environment.

Generated by Build Test Suite for issue #1956 · ● 849.6K ·

@github-actions

This comment has been minimized.

@github-actions
Copy link
Copy Markdown
Contributor

🔮 The ancient spirits stir; this smoke-test agent has passed through the firewall paths and marked this run in the ledger of signals.

🔮 The oracle has spoken through Smoke Codex

@lpcox lpcox marked this pull request as ready for review April 13, 2026 14:35
@lpcox lpcox requested a review from Mossaka as a code owner April 13, 2026 14:35
Copilot AI review requested due to automatic review settings April 13, 2026 14:35
@lpcox lpcox merged commit e543784 into main Apr 13, 2026
50 of 52 checks passed
@lpcox lpcox deleted the copilot/optimize-claude-token-usage-again branch April 13, 2026 14:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces operational cost for the Secret Digger (Claude) workflow by switching its Claude model to Haiku and lowering the configured max-turns limit to better match observed usage.

Changes:

  • Lower engine.max-turns from 8 → 4 for the Secret Digger (Claude) workflow.
  • Add a workflow-scoped GH_AW_MODEL_AGENT_CLAUDE override to pin the agent model to claude-haiku-4-5-20251001.
  • Recompile the corresponding *.lock.yml to reflect the new max-turns and model override in generated job env/commands.
Show a summary per file
File Description
.github/workflows/secret-digger-claude.md Lowers max-turns and pins the workflow’s Claude agent model to Haiku via engine.env.
.github/workflows/secret-digger-claude.lock.yml Regenerated lock file reflecting --max-turns 4 and the pinned GH_AW_MODEL_AGENT_CLAUDE in the workflow jobs.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 1

DISABLE_BUG_COMMAND: 1
DISABLE_ERROR_REPORTING: 1
DISABLE_TELEMETRY: 1
GH_AW_MODEL_AGENT_CLAUDE: claude-haiku-4-5-20251001
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the detection job’s Execute Claude Code CLI step, GH_AW_MODEL_AGENT_CLAUDE is set in env, but the command only references GH_AW_MODEL_DETECTION_CLAUDE for --model. Consider removing GH_AW_MODEL_AGENT_CLAUDE here (or, if the intent is to pin the detection model too, set GH_AW_MODEL_DETECTION_CLAUDE instead) to avoid confusion about which model the detection run uses.

Suggested change
GH_AW_MODEL_AGENT_CLAUDE: claude-haiku-4-5-20251001

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

⚡ Claude Token Optimization2026-04-12 — Secret Digger (Claude)

3 participants