allod check: instantiate machines two to a process #244

Closed

allod-agent wants to merge 1 commit from agent/check-machine-sweep into master

allod-agent commented

2026-09-28 16:13:10 +01:00

Member

Owner summary

The speedup the issue promised is not there. Measured, this makes the gate on allod/archetypes about 2 s faster out of 78 s, which is inside the run-to-run noise. My recommendation is to close this without merging unless that second matters to you.
What changes on a running machine: nothing. Only allod check changes, and only once a machine is rebuilt with tools that carry it.
What changes for whoever runs the gate: machines are instantiated two to a nix process, not one. The transcript shows ==> machines <a>, <b> (one nix process) for a pair.
What stays identical: which machines and checks run, what counts as a pass, the step counts, and the naming of every failure. A pair that fails is rerun one machine at a time, so a broken machine is still named.
What proves it: the self-test against real Nix passes, including a new fixture for pairs, and the gate on allod/archetypes reports the same All 31 steps passed (checks: 18 built) before and after.
If it is wrong: revert this PR. Nothing else depends on it.

Refs allod/archetypes#118. That issue was written against check.sh, which allod/archetypes#117 deleted, so the change is here.

What was measured

All on one 8 GiB dev VM, 6 cores, Nix 2.34.8, which other agents were using at the same time. Before is allod built from 2caeb5e, after is this branch. The two were run alternately, five rounds; round 1 is left out of the means because another build was running during it.

Whole gate	Before	After
`allod/archetypes`, 4 machines and 18 checks, wall clock, rounds 2 to 5	79.0, 76.7, 76.5, 80.3 s	76.1, 73.4, 77.4, 77.5 s
mean	78.1 s	76.1 s
of which machine steps, mean	23.1 s	22.0 s
peak memory of the largest process	1.69 GiB	1.69 GiB
Bench flake of 8 machines and no checks, wall clock, rounds 2 to 5	46.4, 45.4, 48.3, 48.2 s	41.5, 40.4, 41.1, 40.5 s
mean	47.1 s	40.9 s
peak memory of the largest process	0.99 GiB	1.68 GiB

The 8-machine figure is an upper bound, not a forecast. That flake is the four template machines plus a second evaluation of each made with extendModules, and name order pairs every machine with its own twin, which shares more with it than two unrelated machines do.

What the issue claimed, and what was found

Claim in allod/archetypes#118	Found
One batched process is about 3 to 4 times faster	A machine costs the same to evaluate in a shared process as alone. What is shared is the start of the process and the loading of the flake
The probe that ran "far below the serial sum"	Reproduced only as a repeat of `nix build --dry-run` on an unchanged tree, 1.2 s, answered from Nix's evaluation cache. The first run on a tree took 27 to 31 s for four machines. `allod check` does not use that cache: every step is `nix eval --apply`
Memory stays flat	One process for four machines peaked at 2.8 GiB and for eight at 5.1 GiB, which left 1.2 GiB available on this VM. That is why this change stops at two per process
`--keep-going` keeps per-machine failure reporting	It does not. Evaluation stops at the first machine that fails and the others are never named. Hence the rerun one at a time
Running in parallel is slower	Not measured on the gate itself. Four machine evaluations run at once took 12 s against about 25 s one after another
Machines are about 95% of the gate	In `allod check` on `allod/archetypes` they are 23 s of 78 s. The checks are 54 s

What changed

cmd/allod/check.go, cmd/allod/check_nix.go: machines are grouped in twos; a pair is one step kind that carries its two machine steps to fall back on.
cmd/allod/check_test.go: the pair's argv is pinned, and two tests cover grouping with an odd machine out and the fallback after a failed pair.
tests/allod-check-selftest.sh: a swept fixture of three green machines, two assertions on the red fixture that the pair was tried and then run alone, and a differential fixture whose second machine is not a derivation.
docs/allod-check.md, README.md: the behaviour, and the memory table behind the choice of two.

Risk

R2. The worst credible failure is the gate passing a machine that is broken. A pair uses the same expression as a single machine step and passes only when nix exits 0 having forced both derivation paths; any failure sends both machines through the unchanged single-machine step. Rollback is a revert.

Not covered: machine pairs larger than the template's. Peak memory for a pair is roughly the two machines added together, so a fork with much larger machines will see a higher peak than the 1.4 to 1.7 GiB measured here.

Validation

Run by me on the tree committed as 9036a89:

Command	Result
`go test ./cmd/allod/ -run 'Check\|NixString'`	ok
`allod check` in this repository, which builds `go-checks` and the other eight checks	exit 0, `All 11 steps passed (checks: 9 built)`
`ALLOD_UNDER_TEST=<this build> bash tests/allod-check-selftest.sh`	PASS, 92 assertions, every differential row agrees with `nix flake check`
`allod check` on `allod/archetypes` at `9577fb5`	exit 0, `All 31 steps passed (checks: 18 built)`, same as before

Not run: the gate against a real deployment with eight machines. None is reachable from this VM.

Review

One cold pass by gpt-5.6-terra on the diff, one fix round.

Finding	Outcome
The docs said pairs follow the order the flake names machines in; the enumeration sorts by name	Corrected
The docs' timing figures were not backed by the benchmark	Replaced with the figures in the table above
A pair's label joins two raw names with a comma, so a name containing a comma or a newline reads ambiguously	Left. Every label in this command already prints names raw, and the failure summary never uses a pair's label, only each machine's
The one-line summaries say one process per two machines without mentioning the rerun after a failure	Left. The section on pairs says it

If a faster gate is the goal

Two things would deliver one, and neither is in this PR because each is a decision about what the gate is allowed to do:

Run steps a few at a time. Roughly twice as fast on the machine steps measured here, at about 1 GiB of memory for each step running at once.
Let steps use Nix's evaluation cache. A repeat run on an unchanged tree would take seconds. The first run on any tree, and every run on a tree with uncommitted changes, would take as long as now.

## Owner summary - **The speedup the issue promised is not there.** Measured, this makes the gate on `allod/archetypes` about 2 s faster out of 78 s, which is inside the run-to-run noise. My recommendation is to close this without merging unless that second matters to you. - **What changes on a running machine:** nothing. Only `allod check` changes, and only once a machine is rebuilt with tools that carry it. - **What changes for whoever runs the gate:** machines are instantiated two to a `nix` process, not one. The transcript shows `==> machines <a>, <b> (one nix process)` for a pair. - **What stays identical:** which machines and checks run, what counts as a pass, the step counts, and the naming of every failure. A pair that fails is rerun one machine at a time, so a broken machine is still named. - **What proves it:** the self-test against real Nix passes, including a new fixture for pairs, and the gate on `allod/archetypes` reports the same `All 31 steps passed (checks: 18 built)` before and after. - **If it is wrong:** revert this PR. Nothing else depends on it. Refs allod/archetypes#118. That issue was written against `check.sh`, which allod/archetypes#117 deleted, so the change is here. ## What was measured All on one 8 GiB dev VM, 6 cores, Nix 2.34.8, which other agents were using at the same time. Before is `allod` built from `2caeb5e`, after is this branch. The two were run alternately, five rounds; round 1 is left out of the means because another build was running during it. | Whole gate | Before | After | | --- | --- | --- | | `allod/archetypes`, 4 machines and 18 checks, wall clock, rounds 2 to 5 | 79.0, 76.7, 76.5, 80.3 s | 76.1, 73.4, 77.4, 77.5 s | | mean | 78.1 s | 76.1 s | | of which machine steps, mean | 23.1 s | 22.0 s | | peak memory of the largest process | 1.69 GiB | 1.69 GiB | | Bench flake of 8 machines and no checks, wall clock, rounds 2 to 5 | 46.4, 45.4, 48.3, 48.2 s | 41.5, 40.4, 41.1, 40.5 s | | mean | 47.1 s | 40.9 s | | peak memory of the largest process | 0.99 GiB | 1.68 GiB | The 8-machine figure is an upper bound, not a forecast. That flake is the four template machines plus a second evaluation of each made with `extendModules`, and name order pairs every machine with its own twin, which shares more with it than two unrelated machines do. ## What the issue claimed, and what was found | Claim in allod/archetypes#118 | Found | | --- | --- | | One batched process is about 3 to 4 times faster | A machine costs the same to evaluate in a shared process as alone. What is shared is the start of the process and the loading of the flake | | The probe that ran "far below the serial sum" | Reproduced only as a repeat of `nix build --dry-run` on an unchanged tree, 1.2 s, answered from Nix's evaluation cache. The first run on a tree took 27 to 31 s for four machines. `allod check` does not use that cache: every step is `nix eval --apply` | | Memory stays flat | One process for four machines peaked at 2.8 GiB and for eight at 5.1 GiB, which left 1.2 GiB available on this VM. That is why this change stops at two per process | | `--keep-going` keeps per-machine failure reporting | It does not. Evaluation stops at the first machine that fails and the others are never named. Hence the rerun one at a time | | Running in parallel is slower | Not measured on the gate itself. Four machine evaluations run at once took 12 s against about 25 s one after another | | Machines are about 95% of the gate | In `allod check` on `allod/archetypes` they are 23 s of 78 s. The checks are 54 s | ## What changed - `cmd/allod/check.go`, `cmd/allod/check_nix.go`: machines are grouped in twos; a pair is one step kind that carries its two machine steps to fall back on. - `cmd/allod/check_test.go`: the pair's argv is pinned, and two tests cover grouping with an odd machine out and the fallback after a failed pair. - `tests/allod-check-selftest.sh`: a `swept` fixture of three green machines, two assertions on the `red` fixture that the pair was tried and then run alone, and a differential fixture whose second machine is not a derivation. - `docs/allod-check.md`, `README.md`: the behaviour, and the memory table behind the choice of two. ## Risk R2. The worst credible failure is the gate passing a machine that is broken. A pair uses the same expression as a single machine step and passes only when `nix` exits 0 having forced both derivation paths; any failure sends both machines through the unchanged single-machine step. Rollback is a revert. Not covered: machine pairs larger than the template's. Peak memory for a pair is roughly the two machines added together, so a fork with much larger machines will see a higher peak than the 1.4 to 1.7 GiB measured here. ## Validation Run by me on the tree committed as `9036a89`: | Command | Result | | --- | --- | | `go test ./cmd/allod/ -run 'Check\|NixString'` | ok | | `allod check` in this repository, which builds `go-checks` and the other eight checks | exit 0, `All 11 steps passed (checks: 9 built)` | | `ALLOD_UNDER_TEST=<this build> bash tests/allod-check-selftest.sh` | PASS, 92 assertions, every differential row agrees with `nix flake check` | | `allod check` on `allod/archetypes` at `9577fb5` | exit 0, `All 31 steps passed (checks: 18 built)`, same as before | Not run: the gate against a real deployment with eight machines. None is reachable from this VM. ## Review One cold pass by `gpt-5.6-terra` on the diff, one fix round. | Finding | Outcome | | --- | --- | | The docs said pairs follow the order the flake names machines in; the enumeration sorts by name | Corrected | | The docs' timing figures were not backed by the benchmark | Replaced with the figures in the table above | | A pair's label joins two raw names with a comma, so a name containing a comma or a newline reads ambiguously | Left. Every label in this command already prints names raw, and the failure summary never uses a pair's label, only each machine's | | The one-line summaries say one process per two machines without mentioning the rerun after a failure | Left. The section on pairs says it | ## If a faster gate is the goal Two things would deliver one, and neither is in this PR because each is a decision about what the gate is allowed to do: - Run steps a few at a time. Roughly twice as fast on the machine steps measured here, at about 1 GiB of memory for each step running at once. - Let steps use Nix's evaluation cache. A repeat run on an unchanged tree would take seconds. The first run on any tree, and every run on a tree with uncommitted changes, would take as long as now.

allod-agent added 1 commit

2026-09-28 16:13:10 +01:00

allod check: instantiate machines two to a process 9036a89931

Machines are taken two at a time and each pair is instantiated by one
nix process, with the machine step's own expression applied to both
names. A pair that passes counts a step for each machine. A pair that
fails is not a step: evaluation stops at the first error, so its two
machines then run as steps of their own and those decide what failed.
A flake's only machine and the odd one left over run as before.

Two and no more because a process keeps every machine it has evaluated:
four in one process peaked at 2.8 GiB and eight at 5.1 GiB on an 8 GiB
VM, where two stay at 1.4 to 1.7 GiB. docs/allod-check.md has the table.

The saving is small. On the four machines of allod/archetypes the
machine steps took 22 s in pairs and 23 s one to a process.

Refs allod/archetypes#118

allod-agent commented

2026-09-28 16:17:04 +01:00

Author

Member

Closing without merging, at the owner's request: about 2 s off a 78 s gate, inside the run-to-run noise, is not worth the added code. The measurements in the description stand as the record for allod/archetypes#118.

allod-agent closed this pull request

2026-09-28 16:17:05 +01:00

allod-agent referenced this pull request from allod/archetypes

2026-09-28 16:17:06 +01:00

Gate: batch machine toplevel steps into one dry-run sweep (measured ~3-4x faster, flat memory) #118