allod/tools

Watch

Fork 0

memory backpass: run the skill as a program, not a prompt #277

Merged

vnprc merged 11 commits from agent/memory-backpass-command into master

2026-10-19 05:45:58 +01:00

allod-agent commented

2026-10-19 04:41:10 +01:00

Member

allod memory backpass runs the memory backward pass as a program: it reads the session traces not yet analyzed, fills the skill's two prompts, calls a bulk-analysis model once per trace and a synthesis model once, quote-checks every finding, writes the ledger lines and prunes, and opens one pull request on the private memory repository. Nothing changes on a running machine: the command exists but nothing runs it until the archetypes timer (the remaining slice of allod/strategy#55) calls it. What stays identical: every other namespace, the skill text in allod/memory, which stays the spec and supplies the prompts at run time. What proves it: the Go tests drive the command end to end against fixture traces, fixture memory repos with their own origins, a fake model seat and a stub forge, covering the funnel, the quote check, the one-line-per-rule-and-session rule, the hash state file, retry and fallback, the timeout, report-only, write mode and the budget-hook drop; the sandboxed go-checks flake check passes; one cold cross-family review (two reads of the diff, implementation and tests) returned 24 findings, 17 of them verified real and fixed in one round, two declined as fixture-only, the rest comment items handled by the trim. If it is wrong: revert; nothing depends on the namespace yet, and the command never pushes a default branch, so a bad run is one PR to close.

Refs allod/strategy#55
Closes allod/tools#276

What it does

allod memory backpass --traces <dir>... [--public <dir>] [--private <dir>] [--bulk-seat <cmd>] [--fallback-seat <cmd>] [--synthesis-seat <cmd>] [--timeout <s>] [--jobs <n>] [--report-only], documented in docs/allod-memory.md.
A seat is an opaque command the deployment supplies (a wrapper per model, chosen by its agent-roster skill); it receives the prompt file path as its last argument and answers on stdout. Exit status is never the gate, parsed content is; every call runs under a hard timeout in its own process group and is killed by group id, never by pattern.
Prompts are read from the public checkout's skills/memory-backpass/ at run time and filled by a single-pass literal substitution that never rescans substituted text, so a trace containing & or {{TRACE}} reaches the model byte for byte. Both must open with the self marker.
The state file is <traces>/.memory-backpass/analyzed.txt in sha256sum format; a trace that grew is pending again; a failed trace is never marked.
Exit 0 on a complete run, 1 for a refusal or a failed record/submit, 3 when at least one trace failed analysis after the rest of the run completed.

Decisions worth a look

Coverage pruning (skill step 6) applies to gap lines only: an ignored or harm line names an instruction that is in the index by construction, so a literal coverage check would prune every such sighting on sight. Age pruning applies to all kinds.
The synthesis gates (at most five edits, two sessions, verbatim evidence) are enforced by the prompt and reported per edit in the PR body for the human; the code does not revert a model's hunk.
No model judgment is used for prune coverage: the issue allows two seats and nothing else.
The implementation is about 2500 lines of Go (1000 of tests, 28 test functions), above the brief's target; the ten steps, two worktrees, forge parsing and the ledger rules each carry their share.

Risk

R3. The command opens a PR on a memory repository through allod change and commits a state file to the traces checkouts; it never pushes a default branch and never merges. The privacy boundary is the traces repository's own; the command sends trace text only to the seats the deployment names.

Validation

gofmt -l cmd/allod empty; go vet ./cmd/allod/... clean
go test ./cmd/allod ok; go test -tags site,secret ./cmd/allod ok, run by the integrator after the fix round and again after the comment trim
nix build .#checks.x86_64-linux.go-checks passes on the final tree, which is the witness that the fixture budget hook really executes in the sandbox: the first version passed that build only because the hook could not exec there and the test read the failure as a budget failure; the fix round made the assertion load-bearing (the branch must carry the kept edit)
git diff --check clean
Not run: a real model seat or a real forge; the tests use a fake seat script and a stub forge. A hand run against real traces is the next step and belongs to the deployment that holds them.

`allod memory backpass` runs the memory backward pass as a program: it reads the session traces not yet analyzed, fills the skill's two prompts, calls a bulk-analysis model once per trace and a synthesis model once, quote-checks every finding, writes the ledger lines and prunes, and opens one pull request on the private memory repository. Nothing changes on a running machine: the command exists but nothing runs it until the archetypes timer (the remaining slice of allod/strategy#55) calls it. What stays identical: every other namespace, the skill text in `allod/memory`, which stays the spec and supplies the prompts at run time. What proves it: the Go tests drive the command end to end against fixture traces, fixture memory repos with their own origins, a fake model seat and a stub forge, covering the funnel, the quote check, the one-line-per-rule-and-session rule, the hash state file, retry and fallback, the timeout, report-only, write mode and the budget-hook drop; the sandboxed `go-checks` flake check passes; one cold cross-family review (two reads of the diff, implementation and tests) returned 24 findings, 17 of them verified real and fixed in one round, two declined as fixture-only, the rest comment items handled by the trim. If it is wrong: revert; nothing depends on the namespace yet, and the command never pushes a default branch, so a bad run is one PR to close. Refs allod/strategy#55 Closes allod/tools#276 ## What it does - `allod memory backpass --traces <dir>... [--public <dir>] [--private <dir>] [--bulk-seat <cmd>] [--fallback-seat <cmd>] [--synthesis-seat <cmd>] [--timeout <s>] [--jobs <n>] [--report-only]`, documented in `docs/allod-memory.md`. - A seat is an opaque command the deployment supplies (a wrapper per model, chosen by its `agent-roster` skill); it receives the prompt file path as its last argument and answers on stdout. Exit status is never the gate, parsed content is; every call runs under a hard timeout in its own process group and is killed by group id, never by pattern. - Prompts are read from the public checkout's `skills/memory-backpass/` at run time and filled by a single-pass literal substitution that never rescans substituted text, so a trace containing `&` or `{{TRACE}}` reaches the model byte for byte. Both must open with the self marker. - The state file is `<traces>/.memory-backpass/analyzed.txt` in `sha256sum` format; a trace that grew is pending again; a failed trace is never marked. - Exit 0 on a complete run, 1 for a refusal or a failed record/submit, 3 when at least one trace failed analysis after the rest of the run completed. ## Decisions worth a look - Coverage pruning (skill step 6) applies to `gap` lines only: an `ignored` or `harm` line names an instruction that is in the index by construction, so a literal coverage check would prune every such sighting on sight. Age pruning applies to all kinds. - The synthesis gates (at most five edits, two sessions, verbatim evidence) are enforced by the prompt and reported per edit in the PR body for the human; the code does not revert a model's hunk. - No model judgment is used for prune coverage: the issue allows two seats and nothing else. - The implementation is about 2500 lines of Go (1000 of tests, 28 test functions), above the brief's target; the ten steps, two worktrees, forge parsing and the ledger rules each carry their share. ## Risk R3. The command opens a PR on a memory repository through `allod change` and commits a state file to the traces checkouts; it never pushes a default branch and never merges. The privacy boundary is the traces repository's own; the command sends trace text only to the seats the deployment names. ## Validation - `gofmt -l cmd/allod` empty; `go vet ./cmd/allod/...` clean - `go test ./cmd/allod` ok; `go test -tags site,secret ./cmd/allod` ok, run by the integrator after the fix round and again after the comment trim - `nix build .#checks.x86_64-linux.go-checks` passes on the final tree, which is the witness that the fixture budget hook really executes in the sandbox: the first version passed that build only because the hook could not exec there and the test read the failure as a budget failure; the fix round made the assertion load-bearing (the branch must carry the kept edit) - `git diff --check` clean - Not run: a real model seat or a real forge; the tests use a fake seat script and a stub forge. A hand run against real traces is the next step and belongs to the deployment that holds them.

allod-agent added 6 commits

2026-10-19 04:41:10 +01:00

memory: add 'allod memory backpass' (allod/tools#276 ) e124a9fb5a

Drives skills/memory-backpass/SKILL.md as code: a deterministic pass over
pending traces (indexing, prompt filling, the quote check, ledger lines and
prunes) spending a model only on the bulk-analysis and synthesis seats the
skill names. Idempotent via a sha256sum-format state file, report-only and
write modes, and a run report built around the funnel the skill's drop
rate is the alarm for.

memory: test allod memory backpass against fixture traces and memory trees a46f71aa1e

Unit tests pin the template filler, the quote fold, item gating, ledger
dedupe/corroboration, and the two prune rules directly; integration tests
run the real command against git fixtures with a fake seat script and a
stub forge, covering the funnel, retry/fallback, timeout, the hash-marker
state file, write mode's worktrees and PR, and the budget-failure drop.

docs: add allod memory backpass page, link it from README b5d759eb56

In docs/allod-trace.md's shape: what the command is for, every option, the
seat contract, the step map citing the skill's step numbers, the state
file, what each mode writes, exit codes, the report, and how a deployment
points it at its models.

memory: fix round on allod memory backpass (verified cold-review findings) 46388ccf3e

- pruneByCoverage no longer reads ledger.md/README.md/retired.md: a pre-run
  gap line was "covered" by its own text and pruned on every run.
- the memory-budget hook now runs as 'bash <main checkout>/.hooks/memory-
  budget --worktree' with cwd set to the worktree, never the worktree's own
  (seat-writable) copy via its #!/usr/bin/env bash shebang, which has no
  env in the nix sandbox.
- step 7 commits the ledger lines as their own commit before the synthesis
  call; a second budget failure now reverts with 'git checkout -- .' and
  'git clean -fd' instead of selectively checking out claimed paths, which
  left stray untracked files behind.
- an edit's paths are marked claimed only once the whole edit is kept, so a
  partly-absent edit no longer hides its real path from "edited,
  unreported".
- the synthesis gates (session/quote counts, the five-edit limit) are
  enforced as a reviewer-facing report, not by reverting a hunk: a kept
  edit's evidence is checked against this run's actual fold quotes.
- a failed 'change record'/'change submit' no longer commits analyzed.txt
  anywhere and exits 1, naming the failing step; the traces stay pending
  for a worktree that will carry the same ledger lines next run.
- an unreadable trace is a recorded failure (and a 3 exit), not a silent
  skip; a symlinked trace is skipped with one stderr line; two --traces
  checkouts sharing a basename no longer collide under --jobs.
- report-only now prints the fold, matching skill step 10.
- docs and usage text corrected: a public ledger line is only ever listed
  under To relay, never edited.

memory: strengthen backpass tests, fix a silently-broken forge stub, docs 754a616099

- the stub forge's 'pr view' used printf, which interprets \n and put real
  newlines inside a JSON string: every PR view silently failed to parse
  and no test actually exercised the relay check. Switched to a quoted
  heredoc, like 'pr list' already used, and added assertions on it
  (#101 unrelayed, #102 excluded; the rejected title reaching the
  synthesis prompt via a new FAKE_PROMPT_COPY hook in the fake seat).
- new unit tests pin the ledger.md self-coverage regression directly
  (an uncovered pre-run gap survives, a genuinely covered one is pruned),
  and the age-prune test now actually exercises its "not old enough"
  branch instead of retesting "session not found" by accident.
- write-mode and budget-failure tests now find the run's branch with
  'git for-each-ref' instead of recomputing today's date (flaky across
  midnight), assert the kept edit's content is really on the branch (the
  one check that would have caught the budget hook not running at all),
  assert a dropped edit leaves the worktree clean, and assert origin's
  default branch sha never moves.
- new partial-failure test: one trace garbled on every attempt while two
  others analyze normally, exit 3, and analyzed.txt names exactly the two
  that succeeded.
- docs/allod-memory.md: corrected the step-6 prune row and the report
  section for the fold now printed in --report-only.

memory: cut the backpass comments to contracts, traps and pointers 9bce7c9e7f

allod-agent commented

2026-10-19 04:58:41 +01:00

Author

Member

Owner request, recorded before the change: when no seat is given, the command should use whatever model pi itself is configured to use, since that is always the local inference stack. The change: with --bulk-seat or --synthesis-seat absent (option and environment both), the seat is a built-in pi seat that runs script -qec "pi -p '@<prompt file>'" /dev/null under the same timeout and process-group kill, and the report names it as pi with the provider and model read from pi's own settings file. A seat value of exactly pi selects the same built-in explicitly. The fallback seat keeps no default. Documented in the usage text and docs/allod-memory.md; covered by a test with a stub pi and script on PATH.

Owner request, recorded before the change: when no seat is given, the command should use whatever model pi itself is configured to use, since that is always the local inference stack. The change: with `--bulk-seat` or `--synthesis-seat` absent (option and environment both), the seat is a built-in `pi` seat that runs `script -qec "pi -p '@<prompt file>'" /dev/null` under the same timeout and process-group kill, and the report names it as `pi` with the provider and model read from pi's own settings file. A seat value of exactly `pi` selects the same built-in explicitly. The fallback seat keeps no default. Documented in the usage text and `docs/allod-memory.md`; covered by a test with a stub `pi` and `script` on PATH.

allod-agent commented

2026-10-19 04:59:51 +01:00

Author

Member

Owner request, recorded before the change: a bare allod memory backpass should do the right thing on each machine, so the long option line is never typed. The change: every input resolves in a fixed order and the report prints what was chosen and why. Traces: --traces, then ALLOD_BACKPASS_TRACES (path list, colon-separated), then every registry entry marked "traces": true whose checkout is on disk, then ~/work/traces-<hostname>, then ~/work/traces. Private memory: --private, then ALLOD_BACKPASS_PRIVATE, then the one memory-marked registry entry other than allod/memory, then ~/work/agent-memory. Public memory: --public, then ALLOD_BACKPASS_PUBLIC, then the registry as before. A refusal names every place that was looked at. The registry marker is documented for the inventory to set; a deployment whose checkouts follow the registry convention needs nothing.

Owner request, recorded before the change: a bare `allod memory backpass` should do the right thing on each machine, so the long option line is never typed. The change: every input resolves in a fixed order and the report prints what was chosen and why. Traces: `--traces`, then `ALLOD_BACKPASS_TRACES` (path list, colon-separated), then every registry entry marked `"traces": true` whose checkout is on disk, then `~/work/traces-<hostname>`, then `~/work/traces`. Private memory: `--private`, then `ALLOD_BACKPASS_PRIVATE`, then the one memory-marked registry entry other than `allod/memory`, then `~/work/agent-memory`. Public memory: `--public`, then `ALLOD_BACKPASS_PUBLIC`, then the registry as before. A refusal names every place that was looked at. The registry marker is documented for the inventory to set; a deployment whose checkouts follow the registry convention needs nothing.

allod-agent referenced this pull request from a commit

2026-10-19 05:17:38 +01:00

memory: default the bulk and synthesis seats to a built-in 'pi' seat

allod-agent referenced this pull request from a commit

2026-10-19 05:17:38 +01:00

memory: resolve traces/public/private by option, env, registry, convention

allod-agent added 5 commits

2026-10-19 05:17:38 +01:00

memory: default the bulk and synthesis seats to a built-in 'pi' seat 95d2360448

allod/tools#277. When --bulk-seat/--synthesis-seat and their env vars are
all absent, run pi itself (its own configured provider and model) through
script -qec for a terminal, reusing runSeat's process-group timeout and
kill machinery rather than a second copy of it. A seat value of exactly
'pi' selects the same built-in explicitly for any seat, including
fallback, which otherwise keeps no default. Refuses up front when the
built-in is selected and pi or script is missing from PATH, and refuses a
prompt path containing a single quote (it sits inside the inner shell's
quotes). The report names the built-in from $HOME/.pi/agent/settings.json's
defaultProvider/defaultModel, falling back to "pi (configured default)".

memory: test and document the built-in pi seat bacf96ecd0

Stub 'script' (parses -qec <cmd> <file>, execs sh -c "$cmd") and stub 'pi'
(strips the leading @ and answers from the fixture responses dir) prove
the default path runs and names itself pi (fixture/model-x) from a fixture
settings.json. A second test pins the up-front refusal when pi and script
are both missing from PATH. docs/allod-memory.md gets a "The built-in pi
seat" section and the option/deployment-pointer text updated to match.

memory: resolve traces/public/private by option, env, registry, convention 976c62f68c

allod/tools#277 (owner follow-up). A bare 'allod memory backpass' now
works on any machine whose checkouts follow the registry convention: one
resolver (resolveFirst, memory_backpass_resolve.go) tries, in order, the
option, an ALLOD_BACKPASS_* path (ALLOD_BACKPASS_TRACES is colon-
separated), a registry marker (a new "traces": true field alongside
"memory", registry.go) or the existing single-private-entry rule, then a
fixed-name convention path (traces-<hostname>, traces, agent-memory).
Finding nothing is a refusal listing every source tried; an ambiguous
registry match is still an immediate refusal, unchanged. The run report
gets a "## Inputs" section, first, naming each resolved path and which
step found it.

memory: test input resolution by convention, registry, env, and the refusal f254a7c94d

Replaces the old unisolated "requires --traces" test (it depended on the
real machine's ~/work layout) with four isolated ones: resolution by
convention (WORK_DIR holding traces-<hostname> and agent-memory), by a
"traces": true registry marker, by the ALLOD_BACKPASS_* variables, and the
refusal when nothing matches, asserting every source it names. The
existing explicit-flag tests are untouched. docs/allod-memory.md describes
the resolution order for all three inputs and the new "## Inputs" report
section.

memory: trim the seat and resolver comments 1f843e580e

allod-agent commented

2026-10-19 05:17:38 +01:00

Author

Member

Both owner requests above are on the branch. A bare allod memory backpass now works on a machine whose checkouts follow the registry convention: traces resolve by option, ALLOD_BACKPASS_TRACES, a "traces": true registry entry, then ~/work/traces-<hostname> and ~/work/traces; private memory by option, ALLOD_BACKPASS_PRIVATE, the registry, then ~/work/agent-memory; the report opens with an ## Inputs section naming each path and its source. With no seat given, the bulk and synthesis seats are a built-in pi seat that runs script -qec "pi -p '@<prompt>'" /dev/null under the same timeout and process-group kill, named in the report as pi (<provider>/<model>) from pi's own settings file. The fallback seat has no default. Covered by tests with a stub pi and script on PATH and fixture registries; gofmt, vet, both tagged test runs and the sandboxed go-checks flake check pass on the final tree.

Both owner requests above are on the branch. A bare `allod memory backpass` now works on a machine whose checkouts follow the registry convention: traces resolve by option, `ALLOD_BACKPASS_TRACES`, a `"traces": true` registry entry, then `~/work/traces-<hostname>` and `~/work/traces`; private memory by option, `ALLOD_BACKPASS_PRIVATE`, the registry, then `~/work/agent-memory`; the report opens with an `## Inputs` section naming each path and its source. With no seat given, the bulk and synthesis seats are a built-in `pi` seat that runs `script -qec "pi -p '@<prompt>'" /dev/null` under the same timeout and process-group kill, named in the report as `pi (<provider>/<model>)` from pi's own settings file. The fallback seat has no default. Covered by tests with a stub `pi` and `script` on PATH and fixture registries; gofmt, vet, both tagged test runs and the sandboxed `go-checks` flake check pass on the final tree.