pr-explain: require one rule per quiz item and an unambiguous stem #153

Merged

vnprc merged 1 commit from agent/quiz-item-rules into master

2026-08-22 02:46:46 +01:00

allod-agent commented

2026-08-22 02:42:30 +01:00

Member

The quiz-authoring prompt for allod pr explain gains three rules, so generated quiz items stop failing the way one item in the allod/archetypes#39 report did.

Nothing about the tool's behavior, output shape, or validator changes. The prompt is read from disk at run time, so this takes effect on the next report generated and alters no existing one. All three pr-explain suites pass unchanged. If it reads wrong, revert the commit: there is no state, no regenerated report, and nothing that branches on the prompt's text.

What prompted it

An item in the report for allod/archetypes#39 asked why the fleet gate rejects a given assignment. Its scenario read "Both exported hosts contain guest-a, and one host also contains legacy", and its key was "rejects guest-a for cardinality two and rejects legacy because inventory did not select it for microVM". Three separate failures, each now addressed:

A compound key. The item tested two independent rules — cardinality on one side of the fleet comparison, membership on the other — so a reader who understood one half was marked wrong with no signal about which half, and the feedback had to compress two explanations into one sentence.
An ambiguous verb carrying the scenario. "This host contains legacy" reads, to anyone who knows a hypervisor runs guests of both runtimes, as an ordinary and correct arrangement. The item needed it to mean "legacy is in this host's microvm.vms attrset", which it never said.
A distractor that wins under the plain reading. The rejected option said libvirt names are ignored, which is true if "contains" means what it normally means: the gate reads only config.microvm.vms and cannot see libvirt domains at all. The item punished a reasonable parse rather than a misconception.

What changed

pr-explain/quiz-prompt.md, three additions and a renumbering:

Two rules in the per-item list. Test one rule per item, splitting any key that needs an and to join two independent findings. Name the construct the scenario turns on rather than a verb that could carry more than one meaning, because the reader cannot recover an internal meaning the stem did not state.
One sentence in the pre-submission audit: re-read every stem with each word's ordinary meaning and confirm the key still wins. When a distractor becomes defensible under that reading, the stem is what to fix, never the distractor.
One line in "Finish mechanically", so both are verified at the end rather than only stated in the middle.

Risk

Low. The prompt is instruction text consumed by an authoring pass; no code path branches on it, and pr_explain_emit_asset reads it from disk, so there is no embedded copy to fall out of sync. The added length is roughly 150 words against a 6.7 KB prompt, immaterial next to the whole quoted report that pass already receives.

The rules are stated, not enforced. The validator cannot detect a compound key or an ambiguous stem, so the new checklist line is the only gate and it is a self-check by the authoring model — the same enforcement every other authoring rule in this file has. If that proves insufficient, the next step is a validator rule, which this change does not attempt.

Validation

bash tests/pr-explain/components.sh — 86 passed.
bash tests/pr-explain/command.sh — 706 passed.
bash tests/pr-explain/validation.sh — 104 passed.

No report was regenerated. Doing so costs a full authoring run against a live pull request, and no test asserts on this file's prose. Whether the next generated quiz is actually better is unmeasured here, and the first regenerated report is where that gets checked.

Refs allod/archetypes#39.

The quiz-authoring prompt for `allod pr explain` gains three rules, so generated quiz items stop failing the way one item in the allod/archetypes#39 report did. Nothing about the tool's behavior, output shape, or validator changes. The prompt is read from disk at run time, so this takes effect on the next report generated and alters no existing one. All three `pr-explain` suites pass unchanged. If it reads wrong, revert the commit: there is no state, no regenerated report, and nothing that branches on the prompt's text. ## What prompted it An item in the report for allod/archetypes#39 asked why the fleet gate rejects a given assignment. Its scenario read "Both exported hosts contain `guest-a`, and one host also contains `legacy`", and its key was "rejects `guest-a` for cardinality two and rejects `legacy` because inventory did not select it for microVM". Three separate failures, each now addressed: 1. **A compound key.** The item tested two independent rules — cardinality on one side of the fleet comparison, membership on the other — so a reader who understood one half was marked wrong with no signal about which half, and the feedback had to compress two explanations into one sentence. 2. **An ambiguous verb carrying the scenario.** "This host contains `legacy`" reads, to anyone who knows a hypervisor runs guests of both runtimes, as an ordinary and correct arrangement. The item needed it to mean "`legacy` is in this host's `microvm.vms` attrset", which it never said. 3. **A distractor that wins under the plain reading.** The rejected option said libvirt names are ignored, which is true if "contains" means what it normally means: the gate reads only `config.microvm.vms` and cannot see libvirt domains at all. The item punished a reasonable parse rather than a misconception. ## What changed `pr-explain/quiz-prompt.md`, three additions and a renumbering: - Two rules in the per-item list. Test one rule per item, splitting any key that needs an `and` to join two independent findings. Name the construct the scenario turns on rather than a verb that could carry more than one meaning, because the reader cannot recover an internal meaning the stem did not state. - One sentence in the pre-submission audit: re-read every stem with each word's ordinary meaning and confirm the key still wins. When a distractor becomes defensible under that reading, the stem is what to fix, never the distractor. - One line in "Finish mechanically", so both are verified at the end rather than only stated in the middle. ## Risk Low. The prompt is instruction text consumed by an authoring pass; no code path branches on it, and `pr_explain_emit_asset` reads it from disk, so there is no embedded copy to fall out of sync. The added length is roughly 150 words against a 6.7 KB prompt, immaterial next to the whole quoted report that pass already receives. The rules are stated, not enforced. The validator cannot detect a compound key or an ambiguous stem, so the new checklist line is the only gate and it is a self-check by the authoring model — the same enforcement every other authoring rule in this file has. If that proves insufficient, the next step is a validator rule, which this change does not attempt. ## Validation - `bash tests/pr-explain/components.sh` — 86 passed. - `bash tests/pr-explain/command.sh` — 706 passed. - `bash tests/pr-explain/validation.sh` — 104 passed. No report was regenerated. Doing so costs a full authoring run against a live pull request, and no test asserts on this file's prose. Whether the next generated quiz is actually better is unmeasured here, and the first regenerated report is where that gets checked. Refs allod/archetypes#39.

allod-agent added 1 commit

2026-08-22 02:42:30 +01:00