feat: list every rollout in the eval dashboard, queued ones as pending

[mikasenghaas]

Summary

• The --rich eval dashboard only rendered rollouts that had already started — _groups filtered on rollout.trace is not None — so with a low --max-concurrent the queued rollouts were invisible until they began. eval <taskset> -n 8 -c 1 showed only the single rollout in flight.
• Added a PENDING phase to Rollout: a rollout now starts PENDING (queued behind the concurrency cap) and advances to SETUP the moment run() begins.
• The dashboard now lists every rollout from the first frame, keyed off the task each Rollout already carries at construction. Queued rollouts render as [pending] (dim) — only the task is known yet — until they leave the queue, then progress through the normal lifecycle.
• _groups groups by rollout.task.idx (always available) and sorts still-pending rollouts last (no start time), so in-flight/finished ones stay on top and pending ones trail in task order.

Verification

Ran eval reverse_text_v1 -n 8 -c 1 end-to-end (real Prime inference model, max_concurrent=1) and snapshotted the live dashboard as it progressed.

Timelines across frames:

• pending: [8, 7, 6, 5, 4, 3, 2, 1, 0, 0]
• done: [0, 0, 1, 2, 3, 4, 5, 6, 7, 8]

First frame (t=0.0s) — all eight register as pending immediately:

0/8 · 0.0s · reward — · err —
────────────────────────────────────────────────────────────────
  [pending ] task idx=0
  [pending ] task idx=1
  [pending ] task idx=2
  [pending ] task idx=3
  [pending ] task idx=4
  [pending ] task idx=5
  [pending ] task idx=6
  [pending ] task idx=7

Mid-run (t=20.6s) — processed one by one under -c 1: three done, one running, four still pending:

3/8 · 21s · reward 0.33 · err 0.00
  [success ] task idx=0 · … · agent_completed  reward=1.00 ·  8s
  [success ] task idx=1 · … · single_turn      reward=0.00 ·  8s
  [success ] task idx=2 · … · single_turn      reward=0.00 ·  5s
  [rollout ] task idx=3 · … · 0 turns · 0 branches               0.1s
  [pending ] task idx=4
  [pending ] task idx=5
  [pending ] task idx=6
  [pending ] task idx=7

Final frame (t=79.1s) — all eight processed: 8/8.

Before this change the first two frames would show only task idx=0; the rest of the batch was invisible until each one started.

---

Note

Low Risk
Display and lifecycle labeling only; no changes to scoring, concurrency, or rollout execution semantics beyond initial phase default and SETUP transition timing.

Overview
The --rich eval dashboard now lists every rollout from the first frame, including ones still waiting on --max-concurrent. Previously _groups only included rollouts with a trace, so queued work was invisible until run() started.

Rollout lifecycle: New Phase.PENDING. Each rollout begins PENDING and switches to SETUP when run() creates the trace.

Dashboard: [pending] (dim) rows show only the task (rollout.task); grouping uses task.idx instead of trace timing. Pending rollouts sort last via _started (+inf when no trace).

^{Reviewed by Cursor Bugbot for commit ed8e0db. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

List queued rollouts as pending in the eval dashboard

• Adds a PENDING phase to Rollout in rollout.py; newly constructed rollouts start in this phase and transition to SETUP when run() begins.
• Updates grouping in eval.py to use rollout.task.idx instead of rollout.trace.task.idx, so rollouts without a trace are included and sorted after in-flight ones.
• Renders pending rollouts as dimmed rows showing the task identity and [pending] marker, with no result or elapsed time.
• Behavioral Change: rollouts now start in PENDING rather than SETUP, which affects any code that inspects the initial phase before run() is called.

^{Macroscope summarized ed8e0db.}

[macroscopeapp]

Approvability

Verdict: Approved

This PR adds a display-only feature showing queued rollouts as 'pending' in the eval dashboard CLI. Changes are self-contained to visualization logic with no impact on core rollout behavior.

^{You can customize Macroscope's approvability policy. Learn more.}