Split the flake check gate into per-machine and per-check runs #34

allod-agent · 2026-08-01T04:25:00+01:00

allod-agent commented

2026-08-01 04:25:00 +01:00

This replaces the routine nix flake check gate with ./check.sh, a split gate that runs the same current nixosConfigurations and checks coverage one nix process at a time. The reason is memory pressure: the native whole-flake command evaluates all machines and checks in one process, and runtime-module-selection already dominates the peak. This PR is a pressure valve for allod/archetypes#31, not the destination.

The repair pass keeps Bash as a scheduler and keeps Nix responsible for flake semantics. Machine and check names are enumerated from the flake as JSON records, not delimiter text, and Nix constructs quoted installable suffixes for discovered names. Ordinary names stay on normal flake installables; names containing control characters that the installable parser cannot safely represent fall back to a Nix expression that dereferences the JSON record structurally. The gate also refuses any top-level output outside this repo's explicit allow-list: handled outputs checks and nixosConfigurations, plus passive repo surfaces lib, vmFacts, and profilesSource. Those passive surfaces are forced as their own steps so throwing top-level outputs fail here the same way they fail under native nix flake check.

Coverage

./check.sh forces each allowed passive top-level output to WHNF. It forces each nixosConfigurations.<name>.config.system.build.toplevel, verifies that value is a derivation, and then forces drvPath. It builds each check for the current system with nix build --no-link. Foreign-system checks are evaluated to drvPath and reported as evaluated but not built, which is more than native nix flake check does without --all-systems. The gate keeps going after a failed step and exits non-zero with a failure summary.

The script refuses to report success if it found no machine and no check to run. Passive top-level output forcing does not make a repo gate meaningful by itself. The script also refuses invalid checks.<system> names that do not contain -, matching Nix's checkSystemName shape rule.

Risk

The residual risk is that this repo now carries a shell wrapper for behavior normally owned by Nix. The wrapper is deliberately narrow: Bash stores JSON records, schedules isolated Nix invocations, and aggregates status; Nix still owns flake evaluation, derivation checks, passive-output forcing, and escaped attr-path construction for ordinary discovered names. The long-term fix remains allod/archetypes#31: reduce or move the expensive guest-behavior assertions until native nix flake check fits the dev VM again, then remove check.sh.

The guest-behavior migration is tracked separately in allod/vm#6, as part of allod/strategy#20. Move those assertions into minimal allod/vm fixtures in coherent groups, prove allod/vm reports matching sabotage before deleting the heavier archetypes reporter, and keep archetypes focused on runtime selection and wiring.

Validation

./check-selftest.sh passes. It now runs ten zero-input fixture flakes covering successful runs, checks-only flakes, dotted machine/check names, tab-containing names that look like injected records, throwing passive outputs, failed machine/check steps, no-op runs, invalid checks system names, unhandled validated outputs such as packages, and unknown typo-like outputs such as chekcs.

./check.sh passes on the real flake: all 17 steps passed, including the three passive top-level outputs plus the existing 14 machine/check steps.

nix shell nixpkgs#shellcheck -c shellcheck --severity=style check.sh check-selftest.sh is clean.

git diff --check is clean.

Refs allod/strategy#20
Refs allod/archetypes#31
Refs allod/vm#6

This replaces the routine `nix flake check` gate with `./check.sh`, a split gate that runs the same current `nixosConfigurations` and `checks` coverage one `nix` process at a time. The reason is memory pressure: the native whole-flake command evaluates all machines and checks in one process, and `runtime-module-selection` already dominates the peak. This PR is a pressure valve for `allod/archetypes#31`, not the destination. The repair pass keeps Bash as a scheduler and keeps Nix responsible for flake semantics. Machine and check names are enumerated from the flake as JSON records, not delimiter text, and Nix constructs quoted installable suffixes for discovered names. Ordinary names stay on normal flake installables; names containing control characters that the installable parser cannot safely represent fall back to a Nix expression that dereferences the JSON record structurally. The gate also refuses any top-level output outside this repo's explicit allow-list: handled outputs `checks` and `nixosConfigurations`, plus passive repo surfaces `lib`, `vmFacts`, and `profilesSource`. Those passive surfaces are forced as their own steps so throwing top-level outputs fail here the same way they fail under native `nix flake check`. ## Coverage `./check.sh` forces each allowed passive top-level output to WHNF. It forces each `nixosConfigurations.<name>.config.system.build.toplevel`, verifies that value is a derivation, and then forces `drvPath`. It builds each check for the current system with `nix build --no-link`. Foreign-system checks are evaluated to `drvPath` and reported as evaluated but not built, which is more than native `nix flake check` does without `--all-systems`. The gate keeps going after a failed step and exits non-zero with a failure summary. The script refuses to report success if it found no machine and no check to run. Passive top-level output forcing does not make a repo gate meaningful by itself. The script also refuses invalid `checks.<system>` names that do not contain `-`, matching Nix's `checkSystemName` shape rule. ## Risk The residual risk is that this repo now carries a shell wrapper for behavior normally owned by Nix. The wrapper is deliberately narrow: Bash stores JSON records, schedules isolated Nix invocations, and aggregates status; Nix still owns flake evaluation, derivation checks, passive-output forcing, and escaped attr-path construction for ordinary discovered names. The long-term fix remains `allod/archetypes#31`: reduce or move the expensive guest-behavior assertions until native `nix flake check` fits the dev VM again, then remove `check.sh`. The guest-behavior migration is tracked separately in `allod/vm#6`, as part of `allod/strategy#20`. Move those assertions into minimal `allod/vm` fixtures in coherent groups, prove `allod/vm` reports matching sabotage before deleting the heavier archetypes reporter, and keep `archetypes` focused on runtime selection and wiring. ## Validation `./check-selftest.sh` passes. It now runs ten zero-input fixture flakes covering successful runs, checks-only flakes, dotted machine/check names, tab-containing names that look like injected records, throwing passive outputs, failed machine/check steps, no-op runs, invalid `checks` system names, unhandled validated outputs such as `packages`, and unknown typo-like outputs such as `chekcs`. `./check.sh` passes on the real flake: all 17 steps passed, including the three passive top-level outputs plus the existing 14 machine/check steps. `nix shell nixpkgs#shellcheck -c shellcheck --severity=style check.sh check-selftest.sh` is clean. `git diff --check` is clean. Refs allod/strategy#20 Refs allod/archetypes#31 Refs allod/vm#6

allod-agent added 4 commits

2026-08-01 04:25:00 +01:00

Split the flake check gate into per-machine and per-check runs 207cc49d88

Refuse to run when the flake grows an output the gate does not cover ca627b8996

Trim the gate script and its self-test to what they must prove d5d9386aa9

Prove the gate red for every way it can fail, and pass a flake with no machines 7042f0a054

allod-agent referenced this pull request

2026-08-01 04:31:14 +01:00

Bring the flake check's memory peak back under the dev VM's headroom #31

vnprc force-pushed agent/check-gate-split from 7042f0a054 to c25f165bd3

2026-08-02 16:52:53 +01:00

Compare

vnprc requested changes

2026-08-02 17:02:10 +01:00

Dismissed

README.md Outdated

					
				@ -55,3 +55,4 @@

				nix build .#nixosConfigurations.nexus.config.system.build.toplevel --no-link

				```

				`./check.sh` is what to run instead of `nix flake check`. The whole-flake

most of the paragraph is unnecessary. just say the memory requirements are too high so we're breaking up the flake check. exact measurements get stale fast and also nobody cares. they ain't gonna read all that mess

allod-agent commented

2026-08-02 17:10:20 +01:00

Review concerns:

check.sh loses nix flake check's unknown-output warning path. The unhandled-output guard only considers names in the hardcoded validated set (check.sh:87-89), so a top-level typo like chekcs is silently ignored by ./check.sh while nix flake check --no-build emits warning: unknown flake output 'chekcs'. Since this script becomes the human gate, I would either fail on every top-level output outside an explicit allow-list (checks, nixosConfigurations, lib, vmFacts, profilesSource) or at least preserve the warning. Failing loud is cleaner here.
The discovered machine/check names are later spliced into CLI attr paths (check.sh:144, check.sh:153, check.sh:159). That is not equivalent to Nix's structural traversal for all valid attr names: checks.x86_64-linux."foo.bar" passes native nix flake check --no-build, but ./check.sh tries to build checks.x86_64-linux.foo.bar and fails. This repo's current names are plain enough, so it is not breaking today's gate, but it is a fidelity gap in the "read names out of the flake" design. A more robust shape is to keep dereferencing inside Nix with builtins.getAttr using env/json inputs, and let Bash only sequence independent Nix invocations and aggregate statuses.

Design-wise, I do not object to a Bash scheduler as a pragmatic memory workaround, because the load-bearing evaluation/build work still stays in Nix. But I would keep the Bash surface deliberately thin and self-tested as a compatibility shim, not let it become a second implementation of flake-check semantics. Long-term, the cleaner fix is still to lower/split runtime-module-selection enough that native nix flake check fits again.

Review concerns: 1. `check.sh` loses `nix flake check`'s unknown-output warning path. The unhandled-output guard only considers names in the hardcoded `validated` set (`check.sh:87-89`), so a top-level typo like `chekcs` is silently ignored by `./check.sh` while `nix flake check --no-build` emits `warning: unknown flake output 'chekcs'`. Since this script becomes the human gate, I would either fail on every top-level output outside an explicit allow-list (`checks`, `nixosConfigurations`, `lib`, `vmFacts`, `profilesSource`) or at least preserve the warning. Failing loud is cleaner here. 2. The discovered machine/check names are later spliced into CLI attr paths (`check.sh:144`, `check.sh:153`, `check.sh:159`). That is not equivalent to Nix's structural traversal for all valid attr names: `checks.x86_64-linux."foo.bar"` passes native `nix flake check --no-build`, but `./check.sh` tries to build `checks.x86_64-linux.foo.bar` and fails. This repo's current names are plain enough, so it is not breaking today's gate, but it is a fidelity gap in the "read names out of the flake" design. A more robust shape is to keep dereferencing inside Nix with `builtins.getAttr` using env/json inputs, and let Bash only sequence independent Nix invocations and aggregate statuses. Design-wise, I do not object to a Bash scheduler as a pragmatic memory workaround, because the load-bearing evaluation/build work still stays in Nix. But I would keep the Bash surface deliberately thin and self-tested as a compatibility shim, not let it become a second implementation of flake-check semantics. Long-term, the cleaner fix is still to lower/split `runtime-module-selection` enough that native `nix flake check` fits again.

allod-agent commented

2026-08-02 17:22:00 +01:00

Owner direction for the repair pass: keep this PR open, but tighten the split gate so Bash is only the scheduler and Nix owns semantic lookup/validation. Repair the two review findings by failing loudly on unknown top-level outputs outside this repo's explicit allow-list and by avoiding CLI attr-path interpolation for discovered machine/check names. Add self-test coverage for both cases. Also trim the README explanation per the inline comment.

Longer-term strategy stays separate: move guest-behavior assertions into minimal allod/vm fixtures in coherent groups, and remove check.sh once native nix flake check fits within the dev VM headroom again.

Owner direction for the repair pass: keep this PR open, but tighten the split gate so Bash is only the scheduler and Nix owns semantic lookup/validation. Repair the two review findings by failing loudly on unknown top-level outputs outside this repo's explicit allow-list and by avoiding CLI attr-path interpolation for discovered machine/check names. Add self-test coverage for both cases. Also trim the README explanation per the inline comment. Longer-term strategy stays separate: move guest-behavior assertions into minimal `allod/vm` fixtures in coherent groups, and remove `check.sh` once native `nix flake check` fits within the dev VM headroom again.

allod-agent added 1 commit

2026-08-02 17:27:07 +01:00

Repair split gate fidelity 2389d780c0

allod-agent added 1 commit

2026-08-02 17:31:38 +01:00

Preserve flake installables in the split gate 07e4693f63

allod-agent commented

2026-08-02 18:14:24 +01:00

Second read-only review pass found two blockers against the final repaired diff:

check.sh can misparse valid attribute names containing tabs. Enumeration emits raw tab-delimited records (check.sh:100, check.sh:109), and Bash parses them with IFS=$'\t' read -r kind first second third (check.sh:145-150). A valid machine name like bad<TAB>nixosConfigurations."good".config.system.build.toplevel makes machine_attrs receive the injected nixosConfigurations."good"... field while the real quoted suffix lands in the discarded fourth field. Verified with a throwaway flake: ./check.sh ran good twice and passed while native nix flake check --no-build failed on the tabbed machine's throwing toplevel. Use a structured/escaped record format rather than raw TSV.
check.sh does not force allowed passive outputs that native nix flake check still forces. The gate allow-lists lib, vmFacts, and profilesSource as passive outputs (check.sh:83-86) and then only enumerates machines/checks (check.sh:102-110). Verified with profilesSource = throw "profilesSource throws on purpose": ./check.sh passed after building the check, while nix flake check --no-build failed while checking profilesSource. Add explicit WHNF forcing for allowed passive outputs, with a self-test fixture that proves a throwing passive output turns the gate red.

The ordinary dotted-name repair and the final return to normal flake installables look good. Failure aggregation, invalid checks.<system> handling, and the no-steps guard also looked sound in this pass.

Second read-only review pass found two blockers against the final repaired diff: 1. `check.sh` can misparse valid attribute names containing tabs. Enumeration emits raw tab-delimited records (`check.sh:100`, `check.sh:109`), and Bash parses them with `IFS=$'\t' read -r kind first second third` (`check.sh:145-150`). A valid machine name like `bad<TAB>nixosConfigurations."good".config.system.build.toplevel` makes `machine_attrs` receive the injected `nixosConfigurations."good"...` field while the real quoted suffix lands in the discarded fourth field. Verified with a throwaway flake: `./check.sh` ran `good` twice and passed while native `nix flake check --no-build` failed on the tabbed machine's throwing toplevel. Use a structured/escaped record format rather than raw TSV. 2. `check.sh` does not force allowed passive outputs that native `nix flake check` still forces. The gate allow-lists `lib`, `vmFacts`, and `profilesSource` as passive outputs (`check.sh:83-86`) and then only enumerates machines/checks (`check.sh:102-110`). Verified with `profilesSource = throw "profilesSource throws on purpose"`: `./check.sh` passed after building the check, while `nix flake check --no-build` failed while checking `profilesSource`. Add explicit WHNF forcing for allowed passive outputs, with a self-test fixture that proves a throwing passive output turns the gate red. The ordinary dotted-name repair and the final return to normal flake installables look good. Failure aggregation, invalid `checks.<system>` handling, and the no-steps guard also looked sound in this pass.

allod-agent commented

2026-08-02 18:40:28 +01:00

Owner direction: fix the two blockers from the second review pass. Replace raw TSV enumeration with a structured or escaped format so valid attribute names containing tabs/newlines cannot inject fields, and force the allowed passive outputs (lib, vmFacts, profilesSource) so this gate preserves native nix flake check's top-level-output forcing. Add self-test coverage for both repairs.

Owner direction: fix the two blockers from the second review pass. Replace raw TSV enumeration with a structured or escaped format so valid attribute names containing tabs/newlines cannot inject fields, and force the allowed passive outputs (`lib`, `vmFacts`, `profilesSource`) so this gate preserves native `nix flake check`'s top-level-output forcing. Add self-test coverage for both repairs.

allod-agent added 1 commit

2026-08-02 18:47:21 +01:00

Close split gate review gaps 788646331e

allod-agent commented

2026-08-02 18:47:52 +01:00

Fixed the two blockers from the second review pass in 7886463.

Replaced raw TSV enumeration with JSON records parsed by jq, so tabs/newlines in attr names cannot inject fields. Ordinary discovered names still run through normal flake installables; control-character names fall back to structural Nix lookup from the JSON record.
Added passive-output steps for lib, vmFacts, and profilesSource, forced to WHNF with builtins.seq, so throwing passive outputs now fail the wrapper like native nix flake check.
Added self-test fixtures for both regressions: tabbed-names and passive-throw.

Validation: ./check-selftest.sh passed; ./check.sh passed all 17 real-flake steps; ShellCheck style run passed; git diff --check passed.

Fixed the two blockers from the second review pass in `7886463`. - Replaced raw TSV enumeration with JSON records parsed by `jq`, so tabs/newlines in attr names cannot inject fields. Ordinary discovered names still run through normal flake installables; control-character names fall back to structural Nix lookup from the JSON record. - Added passive-output steps for `lib`, `vmFacts`, and `profilesSource`, forced to WHNF with `builtins.seq`, so throwing passive outputs now fail the wrapper like native `nix flake check`. - Added self-test fixtures for both regressions: `tabbed-names` and `passive-throw`. Validation: `./check-selftest.sh` passed; `./check.sh` passed all 17 real-flake steps; ShellCheck style run passed; `git diff --check` passed.

vnprc approved these changes

2026-08-02 19:31:15 +01:00

vnprc merged commit 788646331e into master

2026-08-02 19:31:22 +01:00

vnprc deleted branch agent/check-gate-split

2026-08-02 19:31:22 +01:00

Sign in to join this conversation.

No reviewers