Persist builder cost/context telemetry into metrics.json + persisted-first read path (#83, #135)

[richard-devbot]

Summary

Two halves of one data pipe, shipped together: builder-contract cost/context telemetry → persisted cumulative metrics.

Write path (BLE-6.1, #135) — src/core/harness/telemetry.js is the shared, pure extraction point for the builder contract's structured cost / context / execution fields (legacy bare-number cost tolerated; non-numeric values ignored by extraction — the contract gate's builder_v2_cost_values_are_numeric check is what fails validation on them). sdlc_validate now records telemetry on every validation (retries cost money too): pinned cost_recorded / context_recorded events plus an incremental updateRunMetrics write. This also fixes a latent bug where a v2 object builder.cost was written directly into the cost_recorded event's usd field.

Persistence (#83) — updateRunMetrics gains an atomic in-lock increment path: cumulative_cost_usd, cumulative_tokens {input, output, total}, cumulative_tool_calls add instead of overwrite, and per-stage stage_cost_usd / stage_tokens maps accumulate per key (cost/tokens split evenly across a task's canonical stages, mirroring the stage-elapsed normalization so multi-stage tasks are never double-counted). Existing overwrite/merge semantics for the pre-#83 fields are unchanged.

Read path (#83) — resolveRunTotals prefers persisted cumulative metrics and falls back to event recompute for legacy runs, so dashboard polls stop paying O(events) for totals that are already on disk, and the numbers survive event rotation/archival. Wired through the dashboard state builder, the rollup index (which previously collapsed object-shaped cumulative_tokens to 0), index-served lite runs, the Run Analytics client payload (stageCost / stageTokens — data flows now, UI rendering is later), the run reporter's cost summary (source: metrics|events), the pipeline-state rollup (cost_context.cumulative_tokens + per-stage cost_usd/tokens), and pipeline status text/JSON.

Schema-version decision

Additive-tolerant, no version bump. metrics.json has never carried a schema_version; every reader defaults missing fields, and the new fields are only materialized when data exists. The cumulative_tokens object doubles as the "persisted totals are authoritative" marker — unrelated metrics updates (goal-loop resets, harvest) never stamp it onto legacy files, so legacy runs keep rendering from event recompute (covered by tests). pipeline-state.json stays at schema_version: 1: the new fields are additive with null/zero defaults, breaking no v1 reader. Nothing registers in migrations.js.

Acceptance criteria

• ✅ cost.estimated_usd / cost.actual_usd / execution.tools_used / context profile+workflow+source counts (+ optional token & context-pressure gauge fields) extracted at validate time via the shared helper
• ✅ cost_recorded events update metrics.json token totals incrementally (in-lock, concurrent writers both land)
• ✅ Per-stage stage_cost_usd / stage_tokens maps accumulate alongside the existing fields
• ✅ cost_recorded / context_recorded pinned event contracts (documented next to the schema in HARNESS.md)
• ✅ derive.js prefers persisted cumulative metrics; legacy runs without token fields still render
• ✅ Totals in the pipeline-state rollup; pipeline status --json exposes them
• ✅ Per-stage cost/tokens in the Run Analytics state payload (full-parse and index-served runs)
• ✅ HARNESS.md documents the full metrics.json schema (new section only — surgical diff)
• ✅ Tests: accumulation, incremental update + concurrency, legacy fallback, malformed-input tolerance, extraction unit tests, end-to-end sdlc_validate integration, pipeline-state totals

Notes for #136 (not implemented here, not painted into a corner)

context.tokens_used / tokens_available gauges persist as context_tokens_used / context_tokens_available (the fields pipeline-state already exposed), and context_recorded carries them per task — the threshold classifier for context-pressure warnings can consume both without another schema change.

Verification

npm test (505 pass) · npm run lint · npm run validate (196 agents) · node scripts/security-audit.mjs · git diff --check — all green.

Closes #83
Closes #135
Part of #134

🤖 Generated with Claude Code

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[coderabbitai]

Warning

Review limit reached

@richard-devbot, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 24a4ec24-70b0-4b12-b228-392417f5a071

📥 Commits

Reviewing files that changed from the base of the PR and between a4c3dd1 and 9b0905e.

📒 Files selected for processing (20)

• docs/HARNESS.md
• src/commands/pipeline.js
• src/core/harness/goal-loop.js
• src/core/harness/pipeline-state.js
• src/core/harness/run-state.js
• src/core/harness/telemetry.js
• src/index.js
• src/integrations/pi/rstack-sdlc.ts
• src/observability/alerts/engine.js
• src/observability/collectors/reporter.js
• src/observability/dashboard/state/client-state.js
• src/observability/dashboard/state/index.js
• src/observability/dashboard/state/rollup-index.js
• src/observability/dashboard/state/runs.js
• src/observability/metrics/derive.js
• tests/extension-cost-context.test.js
• tests/harness-telemetry.test.js
• tests/observability-cost-metrics.test.js
• tests/pipeline-loop.test.js
• tests/pipeline-state.test.js

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cost-context-telemetry-83-135

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[richard-devbot]

Adversarial review (FIX-THEN-MERGE) findings applied + branch brought up to date with main:

• F1 (BLOCKING) double-count — increments now key on the builder contract's canonical content hash: a replayed/re-validated identical contract is a no-op, a genuine retry (new content) counts again. goal-loop resets clear the prior attempt's builder.json so a stale contract can't be replayed and re-costed. The loop budget cap now enforces on real spend, not phantom spend.
• F2 silent drift — a failed persisted increment (after its cost_recorded event landed) emits metrics_write_failed; resolveRunTotals detects it and falls back to event recompute rather than reporting a stale total.
• F3 mid-run upgrade — the first increment materializing cumulative_tokens on a run with prior history seeds totals from an event recompute; no dropped history.
• F4 tool calls — cumulative_tool_calls now fed by execution.tool_calls (invocations), distinct from tools_used_count (distinct names); docs match code.

Gates: 530 tests, lint, validate, security-audit, diff-check all green (includes the merged #132/#133 tests).