[BLE-4.2] Agent-11 goal contract: structured goal_evaluation drives the goal loop (#128)

[richard-devbot]

Summary

Completes the BLE-4 goal loop: agent 11 (feedback-loop) now emits a structured goal_evaluation in feedback.json, and the harness goal evaluator consumes it as an evidence-gated writer of the existing verdict protocol — no prose parsing, no second verdict path, and the harness still never calls a model.

Harness (src/core/harness/goal-check.js)

• normalizeGoalEvaluation / goalVerdictsFromFeedback: per-criterion results (met / not_met / unknown) become judge verdicts through the SAME protocol as goal-verdict.json — same iteration-freshness rules (older or missing stamp = stale inside a loop iteration), judge: "agent.11-feedback-loop".
• Evidence-or-nothing: a claim is consumed only when every listed evidence path exists on disk (run-dir first, then project root). unknown results and unevidenced claims are rejected with recorded reasons and fall through to the existing ASK_USER path — the harness never trusts unevidenced claims.
• An explicit goal-verdict.json entry outranks the agent-11 evaluation for the same criterion, so a human/host verdict always wins.
• validateGoalEvaluation: contract check for the section in the same {ok, checks, issues} style as the builder/validator checks (feat(harness): builder contract completeness as a shared harness capability #154 pattern); all new APIs exported from src/index.js.
• Evaluations now surface agent_goal_evaluation: { present, consumed, rejected, issues } for observability; ASK_USER reasons explain why an agent claim was not consumed.

Agent contract (agents/sdlc/11-feedback-loop.md)

• New Task 8: evaluate the run against the active goal.json and emit goal_evaluation with all the [BLE-4.2] Update Agent 11 goal contract #128 fields (status, consistency_score, critical_count, failing_stages[], recommended_rerun_stages[], requires_human_decision, reason) plus per-criterion results with evidence paths, iteration stamp (read from loop_iteration_started events), maintenance-taxonomy-routed rerun stages, and retry/block recommendation. Boundary stated in the prompt: the agent RECOMMENDS with evidence; goal-check.js decides.

Docs

• docs/HARNESS.md: verdict protocol documents the agent-11 writer path.
• docs/loop-recipes.md: architecture-satisfaction recipe now uses stage 11 as the packaged reviewer and reruns 11-feedback-loop each iteration so the verdict stays fresh; direct goal-verdict.json remains the human override.

Invariants held (PR #193 review)

• Harness never calls a model; only structured JSON is read.
• No new state writes — the evaluator remains persist-nothing; loop bounds and reset discipline untouched.
• Iteration-stamped freshness identical to the file-verdict path (missing stamp = stale in iteration context; one-shot evaluations accept unstamped).

Tests

• Agent-11 met with existing evidence satisfies a judge criterion (PASS).
• Unevidenced claim and nonexistent evidence paths rejected → ASK_USER with the rejection reason surfaced.
• unknown result never consumed → ASK_USER.
• Stale (older/missing iteration stamp) evaluation ignored inside a loop; valid one-shot.
• Taxonomy-tagged not_met remediation maps to stage resets (RETRY with recommended stages); recommendation: "block" → BLOCK.
• Explicit goal-verdict.json outranks agent-11 for the same criterion.
• validateGoalEvaluation / normalizeGoalEvaluation schema + junk-shape coverage.
• Loop integration: fresh agent-11 not_met resets its stages on iteration 1, goes stale on iteration 2 → ask_user.

Verification

npm test (479 pass) · npm run lint · npm run validate (196 agents) · node scripts/security-audit.mjs · git diff --check — all green.

Closes #128
Closes #126

🤖 Generated with Claude Code

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[coderabbitai]

Warning

Review limit reached

@richard-devbot, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 12 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c8dfe0a1-9e97-4d0c-ba32-9a87863fc8ee

📥 Commits

Reviewing files that changed from the base of the PR and between ed6b66c and 2a3dbb5.

📒 Files selected for processing (7)

• agents/sdlc/11-feedback-loop.md
• docs/HARNESS.md
• docs/loop-recipes.md
• src/core/harness/goal-check.js
• src/index.js
• tests/harness-goal-check.test.js
• tests/pipeline-loop.test.js

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/goal-contract-128

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[richard-devbot]

Review response — all five BLOCK findings fixed, one commit + test each:

1. HIGH, precedence inversion via id-less shorthand (d0eb67f): verdict pools are now separate; the explicit goal-verdict.json pool is consumed FULLY (id match and the id-less single-judge shorthand) before any agent-11 verdict is considered. Test covers shorthand human FAIL beating agent met and shorthand human PASS beating agent not_met.
2. MEDIUM, over-stamping (044a981): for the agent path only, an iteration stamp GREATER than the current iteration is rejected as malformed (recorded in rejected); goal-verdict.json writers remain trusted. Test: iteration: 99 at iteration 2 → not consumed; exact current stamp still works.
3. MEDIUM, self-refresh wiring dropped (78f5dad): agent-11 verdicts now UNION criterion.rerun_stages into their recommended resets (agent can add stages, never drop the recipe's wiring); explicit verdicts keep documented replace semantics — decision documented in HARNESS.md. Task 8 step 4 now requires self-inclusion of 11-feedback-loop on not_met. The loop test now asserts the self-sustaining path (reset 06 AND 11 → fresh iteration-2 evaluation → complete) instead of the stall.
4. MEDIUM/LOW, evidence containment + honesty (810f303): evidence must resolve (path.resolve + relative-prefix check) to a regular FILE inside the run dir or project root — .. traversal, ., /, directories, and outside absolute paths are rejected. HARNESS.md and loop-recipes.md now state the gate checks evidence existence, not relevance. Tests: outside absolute path, ../ traversal, ., and a bare directory all rejected; contained absolute path still accepted.
5. LOW, duplicate criterion_id (2a3dbb5): conflicting duplicates (differing result/recommendation) are consumed by NEITHER side — dropped with a recorded rejection (→ ASK_USER) and a dedicated goal_evaluation_criteria_no_conflicting_duplicates FAIL check in validateGoalEvaluation. Identical duplicates dedupe to the first.

Gates re-run from the worktree: npm test 483 pass (was 479; +4 net new tests, one rewritten), npm run lint, npm run validate (196 agents), node scripts/security-audit.mjs, git diff --check — all green.

Not addressed by design (per review scope): wiring validateGoalEvaluation into the stage-11 validator profile — being filed as its own issue.

🤖 Generated with Claude Code