chore(agent): add shell efficiency section to appended instructions#3207
Merged
Conversation
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
React Doctor found no issues in the changed files. 🎉 Reviewed by React Doctor for commit |
9efcb25 to
34b8e29
Compare
Nudges the model to batch sequential shell commands with && and emit independent tool calls in one response, cutting full-context round trips. Generated-By: PostHog Code Task-Id: 2e63ca05-ef07-4251-9348-687a07ecfa93
34b8e29 to
6394e8e
Compare
Contributor
|
Reviews (1): Last reviewed commit: "feat(agent): add shell efficiency sectio..." | Re-trigger Greptile |
2 tasks
charlesvien
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bash is the single biggest LLM cost line in PostHog Code sessions (in a sampled 30-day window: 16k Bash generations at ~111k avg input tokens each, ~half of all spend). Each tool call re-sends the entire conversation, so sequential one-command-per-call shell usage multiplies full-context round trips.
Changes
Adds a
SHELL_EFFICIENCYsection toAPPENDED_INSTRUCTIONStelling the model to batch related commands with&&, emit independent tool calls in one response, read multiple files at once, and not rerun commands for output it already has.Follow-up commits add the same section to the workspace-server local session prompt and the cloud agent-server prompt, so Codex sessions (which don't receive
APPENDED_INSTRUCTIONS) and cloud runs carry it too.How did you test this?
pnpm --filter @posthog/agent typecheckpasses.Benchmarks: -19% Bash calls, -12% turns, -15% input tokens, no quality loss
Each run scaffolds a fresh git fixture repo in a temp dir and executes a fixed task through the Claude Agent SDK (
claude-sonnet-4-6,settingSources: []so local config can't contaminate results). Control = no Shell Efficiency section, treatment = this PR's wording. Metrics captured per run: Bash calls, turns, input tokens, cost, task success (with post-hoc verification where applicable).Sweep 1: wording comparison (5 synthetic tasks, 4 variants, n=6 each, 120 runs)
&&example (shipped)The concrete example outperformed a more forceful directive; shipped wording is the example variant.
Sweep 2: real-workload tasks (n=6 each, 36 runs)
Tasks modeled on the top Bash patterns mined from 5k real PostHog Code session commands (grep/search ~17%, git operations ~12%, per-package script runs ~4%).
-19% Bash calls, -12% turns, -15% input tokens, 36/36 success in both variants.
Example collapse (git-workflow, typical run):
Since each tool call re-sends the full conversation (~111k avg input tokens in production sessions), fewer round trips compounds: the turns reduction is the number that translates to spend.
Known limit: models still run check/build/test scripts individually to attribute failures per step (unchanged across all wordings, including "ALWAYS chain"). Arguably correct behavior; not worth forcing via prompt.
Automatic notifications
Created with PostHog Code