Skip to content

fix: don't prefix rlm harness metrics with rlm_#1909

Open
mikasenghaas wants to merge 1 commit into
mainfrom
fix/rlm-metrics-no-prefix
Open

fix: don't prefix rlm harness metrics with rlm_#1909
mikasenghaas wants to merge 1 commit into
mainfrom
fix/rlm-metrics-no-prefix

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

  • The rlm harness's rlm @metric scraped rlm's session meta.json and recorded each numeric metric under an rlm_ prefix (rlm_turns, rlm_prompt_tokens, rlm_completion_tokens, rlm_compactions_count, ...).
  • Drop the prefix so the metrics are recorded under their own names (turns, prompt_tokens, completion_tokens, compactions_count, ...). Non-numeric fields (e.g. stop_reason) are still skipped.

Breaking

  • Metric keys emitted by the v1 RLMHarness lose their rlm_ prefix. Anything consuming these (dashboards, filters, downstream analysis) must migrate rlm_<name><name> (e.g. rlm_turnsturns).

Note

Medium Risk
Breaking change to metric key names for v1 RLM harness rollouts; consumers must update from rlm_<name> to <name>.

Overview
Breaking: The v1 RLMHarness rlm @metric no longer prefixes numeric fields from session meta.json with rlm_. Keys such as turns, prompt_tokens, and compactions_count are recorded under their original names instead of rlm_turns, rlm_prompt_tokens, etc.

Non-numeric metrics (e.g. stop_reason) are still omitted. Downstream dashboards, filters, or analysis that keyed on rlm_* names need to migrate to the unprefixed keys.

Reviewed by Cursor Bugbot for commit b7a7435. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Remove rlm_ prefix from RLM harness metric keys

The RLMHarness.rlm method in harness.py was prefixing each metric key with rlm_ before returning. It now returns the original key names from meta['metrics'] unchanged.

📊 Macroscope summarized b7a7435. 1 file reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

The rlm harness `rlm` metric surfaced rlm's session metrics under an
`rlm_` prefix (`rlm_turns`, `rlm_prompt_tokens`, ...). Drop the prefix so
the metrics are recorded under their own names (`turns`, `prompt_tokens`,
...).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@macroscopeapp

macroscopeapp Bot commented Jul 1, 2026

Copy link
Copy Markdown

Approvability

Verdict: Needs human review

This change modifies the naming convention for RLM harness metrics, removing the 'rlm_' prefix. While small, this is a runtime behavior change that could affect downstream metric consumers (dashboards, monitoring, alerts) expecting the prefixed names.

You can customize Macroscope's approvability policy. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant