Plan-review prompt: add convergence stop rule, finding-origin telemetry, and fix-verification guards #9

New issue

Closed

opened 2026-07-02 23:00:43 +01:00 by vnprc-agent · 0 comments

vnprc-agent commented

2026-07-02 23:00:43 +01:00

Contributor

Evidence: the 15-pass review of vnprc/notes allod/nexus-host-key-rotation-dev-plan.md (commits ccfb6bf..c00ad95, all 2026-07-02, summaries in the Update ... focus areas after pass commits). The process found real blockers but did not converge, and several failure modes trace to the template at templates/plan-review-prompt.md and the Agent Rotation guidance in dev-plans.md.

Observed failure modes

The stop rule (zero BLOCKER/GAP/QUESTION) never triggered in 15 passes and shows no sign of converging: blockers arrived in bursts (3 early architectural, a 3-pass regression chain mid-run, and an original-design blocker still surfacing on pass 15).
Review-artifact churn: by pass 13, roughly 6 of 10 findings fixed contradictions earlier review passes introduced (fixture bullets vs steps, a pass-12 flag patched in pass 13, a pass-9 stub spec broken by a pass-12 proof). The review increasingly consumed itself.
Fix-regression chain: passes 8, 9, and 10 each shipped an incomplete fix to the same after-retire rollback procedure, producing three consecutive single-BLOCKER passes on one section. The chain ended when a different model reviewed the previous fix.
Zero SIMPLIFY findings across all 15 passes despite the anti-overengineering guideline; pure accretion on a ~1000-line plan is exactly what bred the internal contradictions above.
Attribution gap: the Model footer was retrofitted mid-review, so passes 1-8 (including both halves of the worst regression chain) are unattributable; one later pass used an agent-software name (GPT-5 Codex) despite the explicit rule.
Count inflation: pass 3 re-counted pass 2's findings verbatim in its summary, corrupting any severity trend read from the log.

Proposed template changes

Findings summaries must classify each finding's origin: original-plan defect vs introduced-by-pass-N (name the commit). This is what makes trend analysis and any stop heuristic possible.
Replace the zero/zero/zero stop rule with a convergence heuristic: stop plan-text review when review-introduced findings outnumber original-plan findings for two consecutive passes, or after two consecutive passes with no BLOCKER and no original-plan GAP. Remaining focus areas hand off to implementation review; the current rule can grind a token budget to nothing without terminating.
After any pass that commits a structural or design change (blocker-level fix), the next pass must be a scoped diff review of that change by a different model than the author, not a full re-review. The pass 8-10 chain shows structural fixes are where new blockers enter, and the author model verifying its own change did not catch them.
Require an explicit SIMPLIFY sweep each pass: the summary must state what was considered for deletion, and repeated zero-SIMPLIFY passes should be called out as a smell rather than treated as a clean bill.
Summaries count only findings new to the current pass; carried-over unresolved items belong in Focus Areas, not the findings list.
dev-plans.md Agent Rotation: record per-model fix-stability observations in the review prompt's focus-area updates (fix held N passes vs fix needed immediate re-fix), prefer the model with the best stability record for verification passes, and drop a model from the rotation after repeated same-feature regressions. From the attributed passes here: the pass-10 fix (claude-opus-4-8) ended the regression chain and held for five subsequent passes; the pass-12 fix (gpt-5) shipped two semantic holes patched in the very next pass.

Pass data

Pass  Model            BLOCKER  GAP  Notes
 1-8  unattributed     5-6      ~23  footer not yet required; pass 3 re-counted pass 2
 9    claude-fable-5   1        5    blocker fix incomplete, corrected in pass 10
10    claude-opus-4-8  1        3    fix ended the rollback regression chain
11    claude-fable-5   0        6    introduced retire dead-end, fixed in pass 12
12    gpt-5 (as Codex) 1        1    fix left two holes, patched in pass 13
13    claude-fable-5   0        10   ~6 of 10 fixed review-introduced defects
14    gpt-5.5          0        1    clean, minimal
15    claude-fable-5   1        2    original-design blocker from the initial draft

Evidence: the 15-pass review of `vnprc/notes` `allod/nexus-host-key-rotation-dev-plan.md` (commits `ccfb6bf..c00ad95`, all 2026-07-02, summaries in the `Update ... focus areas after pass` commits). The process found real blockers but did not converge, and several failure modes trace to the template at `templates/plan-review-prompt.md` and the Agent Rotation guidance in `dev-plans.md`. ## Observed failure modes - The stop rule (zero BLOCKER/GAP/QUESTION) never triggered in 15 passes and shows no sign of converging: blockers arrived in bursts (3 early architectural, a 3-pass regression chain mid-run, and an original-design blocker still surfacing on pass 15). - Review-artifact churn: by pass 13, roughly 6 of 10 findings fixed contradictions earlier review passes introduced (fixture bullets vs steps, a pass-12 flag patched in pass 13, a pass-9 stub spec broken by a pass-12 proof). The review increasingly consumed itself. - Fix-regression chain: passes 8, 9, and 10 each shipped an incomplete fix to the same after-retire rollback procedure, producing three consecutive single-BLOCKER passes on one section. The chain ended when a different model reviewed the previous fix. - Zero SIMPLIFY findings across all 15 passes despite the anti-overengineering guideline; pure accretion on a ~1000-line plan is exactly what bred the internal contradictions above. - Attribution gap: the Model footer was retrofitted mid-review, so passes 1-8 (including both halves of the worst regression chain) are unattributable; one later pass used an agent-software name (`GPT-5 Codex`) despite the explicit rule. - Count inflation: pass 3 re-counted pass 2's findings verbatim in its summary, corrupting any severity trend read from the log. ## Proposed template changes 1. Findings summaries must classify each finding's origin: original-plan defect vs introduced-by-pass-N (name the commit). This is what makes trend analysis and any stop heuristic possible. 2. Replace the zero/zero/zero stop rule with a convergence heuristic: stop plan-text review when review-introduced findings outnumber original-plan findings for two consecutive passes, or after two consecutive passes with no BLOCKER and no original-plan GAP. Remaining focus areas hand off to implementation review; the current rule can grind a token budget to nothing without terminating. 3. After any pass that commits a structural or design change (blocker-level fix), the next pass must be a scoped diff review of that change by a different model than the author, not a full re-review. The pass 8-10 chain shows structural fixes are where new blockers enter, and the author model verifying its own change did not catch them. 4. Require an explicit SIMPLIFY sweep each pass: the summary must state what was considered for deletion, and repeated zero-SIMPLIFY passes should be called out as a smell rather than treated as a clean bill. 5. Summaries count only findings new to the current pass; carried-over unresolved items belong in Focus Areas, not the findings list. 6. dev-plans.md Agent Rotation: record per-model fix-stability observations in the review prompt's focus-area updates (fix held N passes vs fix needed immediate re-fix), prefer the model with the best stability record for verification passes, and drop a model from the rotation after repeated same-feature regressions. From the attributed passes here: the pass-10 fix (claude-opus-4-8) ended the regression chain and held for five subsequent passes; the pass-12 fix (gpt-5) shipped two semantic holes patched in the very next pass. ## Pass data ``` Pass Model BLOCKER GAP Notes 1-8 unattributed 5-6 ~23 footer not yet required; pass 3 re-counted pass 2 9 claude-fable-5 1 5 blocker fix incomplete, corrected in pass 10 10 claude-opus-4-8 1 3 fix ended the rollback regression chain 11 claude-fable-5 0 6 introduced retire dead-end, fixed in pass 12 12 gpt-5 (as Codex) 1 1 fix left two holes, patched in pass 13 13 claude-fable-5 0 10 ~6 of 10 fixed review-introduced defects 14 gpt-5.5 0 1 clean, minimal 15 claude-fable-5 1 2 original-design blocker from the initial draft ```