Option A: tag replay resume points via trace.info

[mikasenghaas]

Option A — replay buffer via trace.info tags (no core schema change)

Design comparison PR (draft, not for merge). Pairs with Option B (#1896), which is identical except it stores the same metadata as typed MessageNode fields. The two branches differ in only three files — replay_common/selector.py (two accessors), the one tag-write line in the compact harness, and (B-only) verifiers/v1/graph.py — so the diff between #1895 and #1896 is the A/B decision.

What this is

A runnable skeleton of the replay-buffer slice: tag compaction resume points during generation, then turn old rollouts into new training tasks. Producer = the compact example env; consumer = a new replay distribution.

Four separate tasksets (pick one per env via --taskset.id)

• replay_recheck ("try again") — seed the full rollout + an appended user turn (Check your work, fix if wrong) and re-roll.
• replay_compaction_after — resume from a compaction message; the model continues solving.
• replay_compaction_before — resume before a compaction; the model writes the compaction itself, then continues.
• replay_judge — present the rollout's transcript and ask "was this correct? yes/no", graded against the original rollout's actual reward (original_reward > judge_threshold) — a self-supervised correctness label.

Each env selects exactly one task type — no mode-mixing.

Scoring

• recheck / compaction_* reuse the original env's verifier (config.inner): score runs its rewards/metrics over the replay trace with the original task swapped in (e.g. a math verifier checks the replayed continuation's final answer against the original ground truth).
• judge compares the model's yes/no verdict to the original reward (no inner, no sandbox).

Provenance (original task + reward) rides on the task (offline) or trace.info["replay"] (online). Compaction tags are authored from graph structure (no program sensor); recheck/judge use the structural final-answer leaf (tag-free, work on any rollout). Failed-tool-call tagging stays deprioritized (program.py reverted).

Structure — one package, five modules

environments/replay/ ships a shared replay_common library + the four selectable taskset modules. replay_common.base holds everything shared (BaseReplayTaskset/BaseReplayHarness parameterized by KIND, buffer sourcing, seeding, snapshot restore, scoring); each taskset module is ~3 lines fixing its KIND and bundling a harness (auto-selected via default_harness_id).

• Offline (mode=offline): load_tasks materializes one task per KIND resume point from a buffer glob (.../rollouts/step_*/train_rollouts.jsonl), seeding task.prompt with the root→node prefix.
• Online (mode=online): load_tasks returns pool_size virtual slots; the bundled harness samples that KIND from the live buffer per rollout (and serves offline too, seeding the same program from the task's materialized prompt). Sampling is keyed by task index, so a GRPO group replays the same source+point. Empty buffer → stop("replay_buffer_empty") (warmup).

Sandbox / exec replay (snapshot_ref)

• Runtime.snapshot()/restore() stubs (verifiers/v1/runtimes/base.py), durable-ref contract documented. Restore is wired (offline: taskset setup; online: harness). Capture (per-turn) needs a framework hook — deliberate follow-up; refs stay None until the sandbox-snapshot feature lands.

The Option-A choice

Tags + snapshot refs live in trace.info (node_tags/snapshots maps keyed by node_id). No change to MessageNode/Trace. Trade-off: untyped dict joined by node_id (safe across the dump→reload we control; fragile if the graph is re-derived). Reader isolated in replay_common/selector.py, so A→B is a two-function swap.

Files (+515)

• verifiers/v1/runtimes/base.py — snapshot/restore stubs
• environments/compact/compact/{annotate,harness}.py — find + tag compaction before/after into trace.info
• environments/replay/replay_common/{base,selector}.py — shared base + the A reader
• environments/replay/replay_{recheck,judge,compaction_after,compaction_before}/ — the four tasksets + pyproject.toml

🤖 Generated with Claude Code

[macroscopeapp]

🟡 Medium compact/annotate.py:32

compaction_after_nodes() collects every node that is a second-or-later child of a multi-child parent (kids[1:]), assuming all such children are post-compaction branch starts. When the graph deduplicates identical user messages by (parent, message_hash), a later turn that produces the same rewritten user prompt reuses the existing user node and attaches its assistant response as an additional child. This function mis-tags that later assistant node as a compaction_after resume point instead of the reused user node, so replay resumes from the wrong place and skips the compaction prompt.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/compact/compact/annotate.py around line 32:

`compaction_after_nodes()` collects every node that is a second-or-later child of a multi-child parent (`kids[1:]`), assuming all such children are post-compaction branch starts. When the graph deduplicates identical user messages by `(parent, message_hash)`, a later turn that produces the same rewritten user prompt reuses the existing user node and attaches its assistant response as an additional child. This function mis-tags that later assistant node as a `compaction_after` resume point instead of the reused user node, so replay resumes from the wrong place and skips the compaction prompt.

[macroscopeapp]

🟡 Medium compact/annotate.py:42

compaction_before_nodes() unconditionally drops the highest-numbered leaf (leaves[:-1]), assuming it is the final-answer branch. On failed or interrupted rollouts there is no final-answer leaf, so the function silently omits the last real pre-compaction leaf. Replay selectors built from the returned tags will miss a valid resume point on crashed/timed-out traces. Consider checking the run status before dropping the last leaf, or document the assumption that this function is only called on successful rollouts.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/compact/compact/annotate.py around line 42:

`compaction_before_nodes()` unconditionally drops the highest-numbered leaf (`leaves[:-1]`), assuming it is the final-answer branch. On failed or interrupted rollouts there is no final-answer leaf, so the function silently omits the last real pre-compaction leaf. Replay selectors built from the returned tags will miss a valid resume point on crashed/timed-out traces. Consider checking the run status before dropping the last leaf, or document the assumption that this function is only called on successful rollouts.

[macroscopeapp]

🟡 Medium replay_common/base.py:203

_sample() shuffles the file list and returns the first matching rollout from the first file that has any, so a file with one matching rollout is as likely to be picked as a file with hundreds. This skews online sampling toward sparse files instead of giving each resume point in the buffer equal probability. Consider collecting all matching points across all files (or using a two-stage draw weighted by per-file point counts) before choosing one.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/replay/replay_common/base.py around line 203:

`_sample()` shuffles the file list and returns the first matching rollout from the first file that has any, so a file with one matching rollout is as likely to be picked as a file with hundreds. This skews online sampling toward sparse files instead of giving each resume point in the buffer equal probability. Consider collecting all matching points across all files (or using a two-stage draw weighted by per-file point counts) before choosing one.