Empirically validate the pipeline on Pi/GLM-5.2 and set compensation depth

[dasirra]

Pending human review. Edit freely before implementing.

Phase 3 · Part of #2

User story

As the forge maintainer, I want to run building and planning once on small, surface-free real inputs on Pi with GLM-5.2 and observe whether the adversarial roles genuinely attack their outputs, so that I invest in open-model compensations based on evidence rather than speculation.

Proposed behavior

Pick one small, self-contained issue whose acceptance is checkable via CLI or files, needing no browser and no native surface, so the run can complete regardless of surface availability. Run the full building command on Pi/GLM-5.2, and also run one planning command on Pi/GLM-5.2 so the mermaid diagrams (a planning output, never exercised by a building-only run) actually get produced and inspected. Record what happens across the known risks, then produce a short findings report recommending, per risk, whether the matching compensation is needed and how deep. If either run cannot complete, the report records where it broke and that becomes the top compensation priority. If everything passes cleanly, the report states which compensations can be deferred or dropped.

Known risks to observe:

• Evaluator rigor (building): does the round-one evaluator produce at least one finding that forced a change to the artifact before pass, or does it rubber-stamp.
• Orchestrator rule retention (building): does the long orchestrator keep its rarely-firing rules (amendment, restart, no unverified grounding).
• Contract integrity (building): does the definition-of-done contract stay intact across rounds.
• Mermaid validity (planning): do the generated diagrams come out syntactically valid.
• Critic rigor (planning): does the round-one critic produce at least one finding that forced a change to the epic before it was filed, or does it rubber-stamp.

Acceptance criteria

• The validation input chosen needs no browser and no native surface, so the run completes regardless of surface availability.
• A findings report covers each known risk with an observed result: evaluator rigor, orchestrator rule retention, and contract integrity from a building run; mermaid validity and critic rigor from a planning run.
• Evaluator and critic rigor are recorded via a checkable signal, not a subjective judgment: whether the round-one pass produced at least one finding that forced a change to the artifact/epic before it was accepted.
• For each risk, the report gives a go/no-go and a depth recommendation for its compensation.
• The report references the concrete building and planning runs it is based on (the inputs used and the outcomes).

Out of scope

• Building the compensations themselves (that is Open-model hardening (gated tracking issue) #7).
• Validating interview beyond what these runs exercise.

Grounding

• The building and planning commands being exercised: commands/building.md, commands/planning.md. EXISTS.
• The evaluator's binary pass/fail discipline and forbidden verdicts: commands/building.md lines 264-293. EXISTS.
• The rarely-firing rules under test: amendment rule commands/building.md lines 199-203; restart rule lines 302-305; "never relay unverified grounding" lines 342-347. EXISTS.
• The contract structure whose integrity is checked: commands/building.md lines 33-47. EXISTS.
• The mermaid requirement in planning: commands/planning.md lines 100-104. EXISTS.
• The Pi binding (Add a Pi/GLM-5.2 binding so the same commands run on Pi #5) and the provisioned harness (Provision a runnable Pi, GLM-5.2 and Playwright harness #4) this run depends on. NEW until those land.
• The findings report. NEW.

Dependencies

Blocked by #5. Blocks #7 (hard gate).

---

Granular contract to be negotiated by generator/evaluator at /forge:building time and posted back to this issue as a comment.