allod/nexus

Watch

Fork 0

Add the microvm.nix host runtime with per-VM isolation #24

Merged

vnprc merged 10 commits from agent/nexus-microvm-host into master

2026-07-30 22:47:40 +01:00

allod-agent commented

2026-07-30 05:06:56 +01:00

Member

Milestone 3 of the microvm.nix framework adoption arc: the allod/nexus host runtime, its per-start root launcher, and the per-VM isolation nexus#23 asks for. Contracts 8, 8a, 16, 16a, 16b, 21 and 21a, plus the host-side halves of 13 (the launcher's volume preflight) and 17 (host and guest declarations agree before boot).

Closes allod/nexus#23
Refs allod/strategy#20

What this adds

nixosModules.microvmHost composes the pinned upstream microvm.nix host module, taken from allod/vm's re-export, with the policy around it. It is separate from nixosModules.host, so a libvirt-only host does not acquire microvm.nix by importing the baseline. Nexus has no microvm input of its own and allod/vm's lock holds exactly one microvm node, so host and guest modules can only come from one upstream revision (contract 1a).

The interesting part is what happens between systemd starting microvm@<name>.service and QEMU running. Upstream execs the runner directly as one shared microvm:kvm principal. This replaces that with one root launch helper per VM, generated by nix/microvm/launcher.nix, which proves the plaintext root is the bounded non-swappable root-only tmpfs it claims to be, preflights every declared volume image, prepares that VM's credential directory atomically from a root-only staging directory, removes and recreates its QMP working directory, builds a private mount namespace exposing only its own declared objects, and performs the single uid/gid drop to that VM's own microvm-<name> principal. Removal is one idempotent root ExecStopPost appended after every upstream post-stop command.

Two host-wide effects of importing the upstream module are pinned rather than inherited (contract 21a): hardware.ksm.enable stays false, because QEMU marks guest RAM mergeable and an arc whose isolation boundary is about what one guest can learn about a sibling does not acquire a cross-guest memory-deduplication side channel by import; and the state directory rule is tightened from upstream's 0775 microvm:kvm, under which every runner principal — all of them hold kvm — could rename or delete a sibling's whole state entry, including the booted symlink the root unit executes at stop.

docs/microvm-host.md documents the options, the generated runtime layout, the start sequence, and what a guest has to satisfy.

How the isolation fixture is built, and the two things worth a reviewer's attention

The attack matrix is 32 cases per run, each declared with the control that is supposed to deny it rather than with a hand-written expected outcome: own and own-jailed for the access a unit must retain, own-deny for what it must not be able to do to its own objects, parent for what a root-only source parent or the target's own ownership denies in every mode, namespace for what the mount namespace is the control for, procfs for what only the distinct principal closes, and attach for /proc/<pid>/mem, the one member of that family Yama covers. The expectation is derived from the family and the mode, so a case cannot quietly be credited to whichever control happens to be holding.

The /proc/<sibling-pid>/root vector is what nexus#23 is actually about, and the fixture is built so it cannot be narrowed away. A private mount namespace alone does not close it: that path resolves inside the target's own root, so it never traverses the root-only source parents that deny the direct path, and at the pinned kernel the open is gated only by the same-credential comparison in __ptrace_may_access. The distinct per-VM principal is the control. microvm-host-isolation proves this both ways — the production node denies it, and a node that collapses both guests onto one principal with the namespace left intact reopens sibling credential reads, QMP directory listing and file creation, socket unlink, image reads and signal delivery.

The namespace turned out to be load-bearing for function, not only for isolation. The launcher creates every source parent root-only, so with the namespace removed a runner cannot reach its own credential directory or its own QMP directory either: QEMU never starts. The nonamespace node pins that measured outcome. Because a broken VM cannot be attacked, the "which attacks is the namespace the control for" question is answered non-destructively instead, by running the same attack matrix twice on the production node from the same principal — once inside the runner's mount namespace and once outside it. Every case declares which control is responsible for it, so a case cannot quietly be credited to the wrong one: sibling credentials, QMP directories, sockets and images stay denied by the root-only parents and by ownership in both runs, while the sibling's state-directory entries, booted symlink and volume directory reopen without the namespace.

Risk

R3, unchanged from the plan's assessment for this milestone. This adds generated host units around plaintext credentials, writable images, namespaces and one runner principal per VM. No public PR in this arc touches a real host, mutates deployed state or handles real plaintext.

Residual risk after this validation is concentrated in three places:

The microvm command upstream installs for every account on the host. microvm -r <name> execs the runner with no unit, so with no prepared credentials, no namespace and no privilege drop. It fails closed only because the active/<name> directory the runner's -fw_cfg arguments name exists solely while the unit runs, which is what the post-stop hook guarantees. That is an argument, not a wall.
Volume image ownership and modes are the deployer's, checked by the launcher at start but not owned by the framework. A deployer who leaves an image group-readable loses the DAC half of the boundary; the namespace still hides the path.
Rotation is untouched here. Contracts 18, 19 and 20 and the rollback/<name> slot's lifecycle land in milestone 5; this PR only guarantees the post-stop hook never removes that slot, which the isolation test asserts.

Validation

All three ran green on x86_64-linux at this revision:

nix flake check --print-build-logs
nix build .#checks.x86_64-linux.microvm-host-contract --print-build-logs
nix build .#checks.x86_64-linux.microvm-host-isolation --print-build-logs

nix flake check reports The check omitted these incompatible systems: aarch64-linux, which is Nix's own default for a flake that exposes a second host system; the aarch64 attributes still evaluate.

The nested test earned its keep before it ever ran an attack: the namespace stage was creating its current symlink at the bare /var/lib/microvms/<name>/current rather than inside the namespace root, so it rewrote the host's own state entry and left the jail with no exec target. Every static check passed on that launcher — the string the unit executes was right, and the bind set was right — and only the boot said setpriv: failed to execute /var/lib/microvms/vma/current/bin/microvm-run. Fixed in 1bba327, with a static guard added alongside it: every path the namespace stage writes must carry the namespace root as a prefix.

microvm-host-contract reads rendered artefacts only — the template unit, the per-VM drop-in, the mount unit fragment, the generated launcher/namespace/cleanup/tap shell, the built runner, the tmpfiles rule, the active-system descriptor — and extracts each script from the unit that names it, so an ExecStart pointing at the wrong script fails there rather than being inspected as if it were the right one. Two independently evaluated integration fixtures use the default and a moved host plaintext root; the moved one has to move the mount unit, the descriptor, the launcher arguments, every exact credentialFiles value and the built runner's -fw_cfg file= arguments, and no artefact may retain the old default. Namespace contents and post-stop removal are asserted as set equality rather than as grep lists, because a path that is a prefix of a declared one cannot be forbidden by substring.

26 mutation fixtures pair with the module's assertions, each classified by the mechanism that rejected it: a named assertion with a required diagnostic substring, or the option type refusing the value before an assertion could report it. A mutation that aborts evaluation instead of tripping its own diagnostic is reported as a failure rather than read as a pass — tryEval catches assertions and throws but not abort, and a diagnostic computed but never asserted on would otherwise pass silently.

microvm-host-isolation boots four nodes over two real guests. The attack matrix ran four times — from each of the two production runners inside its own namespace, from one of them in the host mount namespace, and from the collapsed shared principal — for 128 case outcomes, every one matching the control its family names, with the node's kernel.yama.ptrace_scope recorded as 1 in each run. Beyond that it asserts, on a running host: the resolved Requires= on the mount unit fragment and that a failing mount unit leaves the VM unstarted; the observed runner and QEMU uid, gid and supplementary groups, with no QEMU -run-with argument; that no credential byte appears in argv, the environment or the journal; prepared directory ownership and modes; per-VM TAP ownership; that a leftover QMP directory is cleared and logged and comes back as a different inode; that post-stop removes the credential, staging and QMP directories while the volume image, the rollback slot and the sibling's prepared state survive; that a missing volume image is refused before QEMU; and that the launcher rejects an absent mount, ramfs, a swappable tmpfs, an oversized tmpfs and a mode-0755 tmpfs before writing anything, with the ancestor-only --target match demonstrated to be the thing that would have passed.

Not in this PR

Contracts 9 to 15 and 8b are guest-side and archetypes-side (milestone 4). Contracts 18, 19 and 20 and nexus#21/#22 are milestone 5. The vm input still pins the unmerged agent/vm-microvm-guest-split branch and must move to the merged master revision when allod/vm#5 lands.

The rotation-tool half of contract 8a's mount preflight — the same checks in vm-ssh-host-key and forge-ssh-key, which run outside any unit — is deliberately absent: those tools change in milestone 5, and the preflight belongs with that change rather than as a second implementation here.

Milestone 3 of the microvm.nix framework adoption arc: the `allod/nexus` host runtime, its per-start root launcher, and the per-VM isolation nexus#23 asks for. Contracts 8, 8a, 16, 16a, 16b, 21 and 21a, plus the host-side halves of 13 (the launcher's volume preflight) and 17 (host and guest declarations agree before boot). Closes allod/nexus#23 Refs allod/strategy#20 ## What this adds `nixosModules.microvmHost` composes the pinned upstream microvm.nix host module, taken from `allod/vm`'s re-export, with the policy around it. It is separate from `nixosModules.host`, so a libvirt-only host does not acquire microvm.nix by importing the baseline. Nexus has no microvm input of its own and `allod/vm`'s lock holds exactly one microvm node, so host and guest modules can only come from one upstream revision (contract 1a). The interesting part is what happens between systemd starting `microvm@<name>.service` and QEMU running. Upstream execs the runner directly as one shared `microvm:kvm` principal. This replaces that with one root launch helper per VM, generated by `nix/microvm/launcher.nix`, which proves the plaintext root is the bounded non-swappable root-only tmpfs it claims to be, preflights every declared volume image, prepares that VM's credential directory atomically from a root-only staging directory, removes and recreates its QMP working directory, builds a private mount namespace exposing only its own declared objects, and performs the single uid/gid drop to that VM's own `microvm-<name>` principal. Removal is one idempotent root `ExecStopPost` appended after every upstream post-stop command. Two host-wide effects of importing the upstream module are pinned rather than inherited (contract 21a): `hardware.ksm.enable` stays false, because QEMU marks guest RAM mergeable and an arc whose isolation boundary is about what one guest can learn about a sibling does not acquire a cross-guest memory-deduplication side channel by import; and the state directory rule is tightened from upstream's `0775 microvm:kvm`, under which every runner principal — all of them hold `kvm` — could rename or delete a sibling's whole state entry, including the `booted` symlink the root unit executes at stop. `docs/microvm-host.md` documents the options, the generated runtime layout, the start sequence, and what a guest has to satisfy. ## How the isolation fixture is built, and the two things worth a reviewer's attention The attack matrix is 32 cases per run, each declared with the control that is supposed to deny it rather than with a hand-written expected outcome: `own` and `own-jailed` for the access a unit must retain, `own-deny` for what it must not be able to do to its own objects, `parent` for what a root-only source parent or the target's own ownership denies in every mode, `namespace` for what the mount namespace is the control for, `procfs` for what only the distinct principal closes, and `attach` for `/proc/<pid>/mem`, the one member of that family Yama covers. The expectation is derived from the family and the mode, so a case cannot quietly be credited to whichever control happens to be holding. **The `/proc/<sibling-pid>/root` vector is what nexus#23 is actually about, and the fixture is built so it cannot be narrowed away.** A private mount namespace alone does not close it: that path resolves inside the target's own root, so it never traverses the root-only source parents that deny the direct path, and at the pinned kernel the open is gated only by the same-credential comparison in `__ptrace_may_access`. The distinct per-VM principal is the control. `microvm-host-isolation` proves this both ways — the production node denies it, and a node that collapses both guests onto one principal with the namespace left intact reopens sibling credential reads, QMP directory listing and file creation, socket unlink, image reads and signal delivery. **The namespace turned out to be load-bearing for function, not only for isolation.** The launcher creates every source parent root-only, so with the namespace removed a runner cannot reach its own credential directory or its own QMP directory either: QEMU never starts. The `nonamespace` node pins that measured outcome. Because a broken VM cannot be attacked, the "which attacks is the namespace the control for" question is answered non-destructively instead, by running the same attack matrix twice on the production node from the same principal — once inside the runner's mount namespace and once outside it. Every case declares which control is responsible for it, so a case cannot quietly be credited to the wrong one: sibling credentials, QMP directories, sockets and images stay denied by the root-only parents and by ownership in both runs, while the sibling's state-directory entries, `booted` symlink and volume directory reopen without the namespace. ## Risk R3, unchanged from the plan's assessment for this milestone. This adds generated host units around plaintext credentials, writable images, namespaces and one runner principal per VM. No public PR in this arc touches a real host, mutates deployed state or handles real plaintext. Residual risk after this validation is concentrated in three places: - The `microvm` command upstream installs for every account on the host. `microvm -r <name>` execs the runner with no unit, so with no prepared credentials, no namespace and no privilege drop. It fails closed only because the `active/<name>` directory the runner's `-fw_cfg` arguments name exists solely while the unit runs, which is what the post-stop hook guarantees. That is an argument, not a wall. - Volume image ownership and modes are the deployer's, checked by the launcher at start but not owned by the framework. A deployer who leaves an image group-readable loses the DAC half of the boundary; the namespace still hides the path. - Rotation is untouched here. Contracts 18, 19 and 20 and the `rollback/<name>` slot's lifecycle land in milestone 5; this PR only guarantees the post-stop hook never removes that slot, which the isolation test asserts. ## Validation All three ran green on x86_64-linux at this revision: ```sh nix flake check --print-build-logs nix build .#checks.x86_64-linux.microvm-host-contract --print-build-logs nix build .#checks.x86_64-linux.microvm-host-isolation --print-build-logs ``` `nix flake check` reports `The check omitted these incompatible systems: aarch64-linux`, which is Nix's own default for a flake that exposes a second host system; the aarch64 attributes still evaluate. The nested test earned its keep before it ever ran an attack: the namespace stage was creating its `current` symlink at the bare `/var/lib/microvms/<name>/current` rather than inside the namespace root, so it rewrote the host's own state entry and left the jail with no exec target. Every static check passed on that launcher — the string the unit executes was right, and the bind set was right — and only the boot said `setpriv: failed to execute /var/lib/microvms/vma/current/bin/microvm-run`. Fixed in `1bba327`, with a static guard added alongside it: every path the namespace stage writes must carry the namespace root as a prefix. `microvm-host-contract` reads rendered artefacts only — the template unit, the per-VM drop-in, the mount unit fragment, the generated launcher/namespace/cleanup/tap shell, the built runner, the tmpfiles rule, the active-system descriptor — and extracts each script from the unit that names it, so an `ExecStart` pointing at the wrong script fails there rather than being inspected as if it were the right one. Two independently evaluated integration fixtures use the default and a moved host plaintext root; the moved one has to move the mount unit, the descriptor, the launcher arguments, every exact `credentialFiles` value and the built runner's `-fw_cfg file=` arguments, and no artefact may retain the old default. Namespace contents and post-stop removal are asserted as set equality rather than as grep lists, because a path that is a prefix of a declared one cannot be forbidden by substring. 26 mutation fixtures pair with the module's assertions, each classified by the mechanism that rejected it: a named assertion with a required diagnostic substring, or the option type refusing the value before an assertion could report it. A mutation that aborts evaluation instead of tripping its own diagnostic is reported as a failure rather than read as a pass — `tryEval` catches assertions and throws but not `abort`, and a diagnostic computed but never asserted on would otherwise pass silently. `microvm-host-isolation` boots four nodes over two real guests. The attack matrix ran four times — from each of the two production runners inside its own namespace, from one of them in the host mount namespace, and from the collapsed shared principal — for 128 case outcomes, every one matching the control its family names, with the node's `kernel.yama.ptrace_scope` recorded as 1 in each run. Beyond that it asserts, on a running host: the resolved `Requires=` on the mount unit fragment and that a failing mount unit leaves the VM unstarted; the observed runner and QEMU uid, gid and supplementary groups, with no QEMU `-run-with` argument; that no credential byte appears in argv, the environment or the journal; prepared directory ownership and modes; per-VM TAP ownership; that a leftover QMP directory is cleared and logged and comes back as a different inode; that post-stop removes the credential, staging and QMP directories while the volume image, the rollback slot and the sibling's prepared state survive; that a missing volume image is refused before QEMU; and that the launcher rejects an absent mount, `ramfs`, a swappable tmpfs, an oversized tmpfs and a mode-0755 tmpfs before writing anything, with the ancestor-only `--target` match demonstrated to be the thing that would have passed. ## Not in this PR Contracts 9 to 15 and 8b are guest-side and archetypes-side (milestone 4). Contracts 18, 19 and 20 and nexus#21/#22 are milestone 5. The `vm` input still pins the unmerged `agent/vm-microvm-guest-split` branch and must move to the merged `master` revision when allod/vm#5 lands. The rotation-tool half of contract 8a's mount preflight — the same checks in `vm-ssh-host-key` and `forge-ssh-key`, which run outside any unit — is deliberately absent: those tools change in milestone 5, and the preflight belongs with that change rather than as a second implementation here.

allod-agent added 8 commits

2026-07-30 05:06:57 +01:00

Add the allod/vm input holding the sole microvm.nix pin 6d3977077e

Refs allod/strategy#20

Add the microVM root launch helper generator 16d440e685

Add the microvm host module and its generated-artefact check a339a8f3e9

Quote-agnostic artefact matching and valid fixture hosts in the contract check 51647dacff

Add the two-guest microvm host isolation test b51ccaa538

Document the microvm host runtime and fix the isolation fixture credentials 339c4f12c1

Create the namespace runner symlink inside the namespace root 1bba3273ad

Make the isolation probes measure the boundary they name 5615d7ec56

allod-agent referenced this pull request from allod/memory

2026-07-30 05:08:25 +01:00

Record the check-writing gotchas from the microvm host work #34

allod-agent commented

2026-07-30 08:35:30 +01:00

Author

Member

Reviewing model claude-fable-5, reviewed commit 5615d7e, scope covered: allod/nexus PR #24 (milestone 3) — contracts 8, 8a, 16, 16a, 16b, 21, 21a, and the host-side halves of 13 (launcher volume preflight) and 17 (host/guest agreement). Read against pinned microvm.nix 39a499ab (/nix/store/yyrsgw6gs5v1qnfn839mrn6mmj91zwfw-source) and pinned nixpkgs b6018f87. Files: flake.nix, nix/microvm/host.nix, nix/microvm/launcher.nix, checks/microvm/{default,fixtures,host-contract,isolation}.nix, and the docs/README diff; upstream nixos-modules/host/default.nix, lib/runner.nix, lib/runners/qemu.nix, lib/volumes.nix, nixos-modules/host/options.nix. I did not re-run either named check; both are reported green at this revision and I verified by static reading and targeted grep/lock inspection rather than rebuilding.

Findings

No BLOCKERs.

No GAPs.

[SIMPLIFY] nexus.microvm.machines.<name>.volumeImages is a duplicate source of truth the launcher never reads; delete it. In nix/microvm/host.nix the launcher is fed volumes from the guest — perVm line 88 volumes = guest.microvm.volumes; and line 96 volumes = map (v: { inherit (v) image readOnly; }) volumes; — and it is the guest-declared images the launcher preflights (launcher.nix 267-273, test -e/test -f/volume_probe). The host option volumeImages (host.nix 381-390) has exactly one consumer: the assertion at host.nix 302 vm.machine.volumeImages == vm.guestVolumeImages, which compares it against that same guest declaration. I confirmed by grep there is no other reader in nix/ or checks/. Contrast credentialSources, which is genuinely host-owned and stays: it is the private source path the launcher installs plaintext from (host.nix 95 -> launcher.nix 284-291), a distinct fact from the guest's credentialFiles destination (derived from hostPlaintextRoot). A volume image has no such source/destination split — in microvm.nix the host image path can only live in guest.microvm.volumes[].image, because that is the string QEMU renders into -drive file= (lib/runners/qemu.nix 275-279), and the launcher reads the same field. So machines.<name>.volumeImages restates a fact that already has one authoritative home, which is the architecture principle-8 violation ("duplicating a fact is a bug even when copying is convenient"). It resolves one of two bad ways when archetypes wires it in milestone 4 (the PR body confirms it is "not wired yet"): either the deployer types the private image path in two places kept in sync only by this assertion — error-prone duplication the launcher ignores — or archetypes derives the guest image from the host option via extendModules the way it derives credentialFiles, in which case host == guest is a tautology that can never fail and the host-and-guest-volume-disagreement mutation is unreachable in production. What to do: drop volumeImages from the machines submodule, drop guestVolumeImages and the line-302 assertion (and its host-and-guest-volume-disagreement fixture), and let guest.microvm.volumes be the single declaration the launcher already preflights. Contract 17's volume clause then reduces to "the launcher preflights exactly the images the guest attaches," which is structurally true because it reads them off the guest — so the plan's contract 17 wording should be trimmed to match. This is a plan-shaped simplification as much as a code one; flag it to whoever owns the milestone-4 archetypes wiring so they do not build the duplication in.

No QUESTIONs.

What is right, and should not be undone

The isolation fixture is honest, and its honesty is structural rather than incidental. checks/microvm/isolation.nix assigns every one of the 32 probe cases a family and derives the pass/fail expectation from family × mode in one expectation() function (lines 139-153), so a case cannot be quietly credited with whatever outcome it happens to produce. I walked each case against what actually denies it at the pinned kernel and the family assignments hold: parent cases are denied by the root-only 0700 source parents (${root}/active, ${qmpRoot}) or by 0600/0400 target ownership in every mode; namespace cases (sibling runner, both booted symlinks, state-dir and sibling-volume-dir enumeration) reopen only in the host mount namespace; procfs cases reopen only under the collapsed principal; and attach (/proc/<pid>/mem) stays denied even under one principal because it is the one PTRACE_MODE_ATTACH open Yama covers. Running the same matrix twice on the production node — inside the runner's namespace and in the host namespace — to attribute the namespace non-destructively, rather than trying to attack an already-broken VM, is the right call, and the nonamespace/shared/mountfail sabotage nodes each reopen exactly the vector they name. I checked that shared is production-identical except the principal: sabotagedLauncher reconstructs the same credentialDirectory, qmpDirectory, jailRoot and volume set the host module derives, differing only in principal = "microvm-shared". Do not narrow the procfs cases to direct-path checks — nexus#23's second comment is explicit that this is the specific way to close the issue with the boundary still open.

The assertion/fixture pairing in host.nix ↔ fixtures.nix is complete and non-vacuous. Every assertion I enumerated (two global path/overlap checks, the undeclared-machine and duplicate-QMP/duplicate-image checks, and the per-VM socket, QMP-location, user, vsock.cid, credential-name-set, exact-credential-path, credential-source, and four volume checks) has a mutation that trips precisely it, and the 26 mutations exceed the check's >= 24 floor. The three-way probe classification (fixtures.nix 273-296) genuinely closes the tryEval blind spot the brief warned about: an abort escaping tryEval takes the whole check down at eval time rather than reading as a pass, a type-rejected value is required to evaluate to false, and an assertion mutation that computes a diagnostic but never trips is caught as "no assertion failed." The deliberate omission of an overlong-VM-name fixture, with the reasoning recorded at host.nix 229-235 and fixtures.nix 474-477 (nixpkgs' raw builtins.abort on a >31-char group name cannot be paired), is the correct call, not a hole.

The launcher/upstream-unit interaction is correct across the lifecycle. The QEMU runner exposes no supportsNotifySocket (lib/runner.nix 24 -> or false) and useNotifySockets defaults false, so the unit is Type=simple and the launcher's pivot_root cannot strand a filesystem NOTIFY_SOCKET — a real trap the design avoids by inheritance, worth keeping in mind if a future runner sets notify. ExecStop is left as upstream's booted/bin/microvm-shutdown and now runs as root in the host namespace, reaching the QMP socket QEMU created because qmpDirectory is bind-mounted into the jail at its identical absolute path (contract 8 / the pass-3 fix); shutdownCommand blocks on cat until QEMU exits, so the mkAfter ExecStopPost cleanup only removes the QMP directory after the socket is dead. The ExecStopPost ordering (upstream microvm-unregister from the template, then nexus cleanup from the instance drop-in) is the resolved systemd order, checked both statically and on the booted host. The state-directory tightening to 0755 root:root does not break install-microvm-@ (root) or microvm-set-booted@ (writes inside ${stateDir}/%i, still microvm:kvm), and it does close a runner principal's ability to rewrite the booted symlink the root unit executes. WorkingDirectory stays upstream's state dir but the jail's cd ${jailRoot} / pivot_root / cd / discards it, verified against /proc/<pid>/cwd and /proc/<pid>/root/.../booted in the test.

Contract 21a is bounded exactly as the plan pins it: hardware.ksm.enable = false neutralizes the mem-merge=on that lib/runners/qemu.nix line 85 sets unconditionally on x86_64, the microvm state-package/bridge.conf/bridge-helper/libvirt surface is asserted unchanged, and the two inert upstream additions (boot.kernelModules, the memlock PAM grant with no PAMName) are left alone. The namespace is genuinely load-bearing for function — every source parent is created 0700 root:root, so bind_file/bind_dir exposing only the leaf is what makes a runner reach its own credential and QMP objects without making a sibling's reachable — and the nonamespace node pins that measured outcome so a later change cannot silently turn the namespace into mere hardening. The credentialSources split, the strMatching "^/run/allod/..." option type doing the "never a Nix path, never outside /run/allod" half of contract 8a before any assertion, and the size=16384k reported-form match between the mount unit and the launcher preflight are all correct and should survive later passes.

One non-finding worth stating for the next reader: the vm input still pins the unmerged agent/vm-microvm-guest-split branch (flake.lock rev 783bf6a), which the PR body discloses and which must move to the merged revision when allod/vm#5 lands — the normal milestone-2-before-3 sequencing gate, not a defect in this diff. The lock holds exactly one microvm node at 39a499ab and no microvm input at the nexus root, so the nexus half of contract 1a holds.

Reviewing model `claude-fable-5`, reviewed commit `5615d7e`, scope covered: `allod/nexus` PR #24 (milestone 3) — contracts 8, 8a, 16, 16a, 16b, 21, 21a, and the host-side halves of 13 (launcher volume preflight) and 17 (host/guest agreement). Read against pinned microvm.nix `39a499ab` (`/nix/store/yyrsgw6gs5v1qnfn839mrn6mmj91zwfw-source`) and pinned nixpkgs `b6018f87`. Files: `flake.nix`, `nix/microvm/host.nix`, `nix/microvm/launcher.nix`, `checks/microvm/{default,fixtures,host-contract,isolation}.nix`, and the docs/README diff; upstream `nixos-modules/host/default.nix`, `lib/runner.nix`, `lib/runners/qemu.nix`, `lib/volumes.nix`, `nixos-modules/host/options.nix`. I did not re-run either named check; both are reported green at this revision and I verified by static reading and targeted `grep`/lock inspection rather than rebuilding. ## Findings No BLOCKERs. No GAPs. 1. [SIMPLIFY] `nexus.microvm.machines.<name>.volumeImages` is a duplicate source of truth the launcher never reads; delete it. In `nix/microvm/host.nix` the launcher is fed volumes from the guest — `perVm` line 88 `volumes = guest.microvm.volumes;` and line 96 `volumes = map (v: { inherit (v) image readOnly; }) volumes;` — and it is the guest-declared images the launcher preflights (`launcher.nix` 267-273, `test -e`/`test -f`/`volume_probe`). The host option `volumeImages` (host.nix 381-390) has exactly one consumer: the assertion at host.nix 302 `vm.machine.volumeImages == vm.guestVolumeImages`, which compares it against that same guest declaration. I confirmed by grep there is no other reader in `nix/` or `checks/`. Contrast `credentialSources`, which is genuinely host-owned and stays: it is the private *source* path the launcher installs plaintext from (host.nix 95 -> launcher.nix 284-291), a distinct fact from the guest's `credentialFiles` *destination* (derived from `hostPlaintextRoot`). A volume image has no such source/destination split — in microvm.nix the host image path can only live in `guest.microvm.volumes[].image`, because that is the string QEMU renders into `-drive file=` (`lib/runners/qemu.nix` 275-279), and the launcher reads the same field. So `machines.<name>.volumeImages` restates a fact that already has one authoritative home, which is the architecture principle-8 violation ("duplicating a fact is a bug even when copying is convenient"). It resolves one of two bad ways when archetypes wires it in milestone 4 (the PR body confirms it is "not wired yet"): either the deployer types the private image path in two places kept in sync only by this assertion — error-prone duplication the launcher ignores — or archetypes derives the guest image from the host option via `extendModules` the way it derives `credentialFiles`, in which case `host == guest` is a tautology that can never fail and the `host-and-guest-volume-disagreement` mutation is unreachable in production. What to do: drop `volumeImages` from the `machines` submodule, drop `guestVolumeImages` and the line-302 assertion (and its `host-and-guest-volume-disagreement` fixture), and let `guest.microvm.volumes` be the single declaration the launcher already preflights. Contract 17's volume clause then reduces to "the launcher preflights exactly the images the guest attaches," which is structurally true because it reads them off the guest — so the plan's contract 17 wording should be trimmed to match. This is a plan-shaped simplification as much as a code one; flag it to whoever owns the milestone-4 archetypes wiring so they do not build the duplication in. No QUESTIONs. ## What is right, and should not be undone The isolation fixture is honest, and its honesty is structural rather than incidental. `checks/microvm/isolation.nix` assigns every one of the 32 probe cases a `family` and derives the pass/fail expectation from `family × mode` in one `expectation()` function (lines 139-153), so a case cannot be quietly credited with whatever outcome it happens to produce. I walked each case against what actually denies it at the pinned kernel and the family assignments hold: `parent` cases are denied by the root-only `0700` source parents (`${root}/active`, `${qmpRoot}`) or by `0600`/`0400` target ownership in every mode; `namespace` cases (sibling runner, both `booted` symlinks, state-dir and sibling-volume-dir enumeration) reopen only in the host mount namespace; `procfs` cases reopen only under the collapsed principal; and `attach` (`/proc/<pid>/mem`) stays denied even under one principal because it is the one `PTRACE_MODE_ATTACH` open Yama covers. Running the same matrix twice on the production node — inside the runner's namespace and in the host namespace — to attribute the namespace non-destructively, rather than trying to attack an already-broken VM, is the right call, and the `nonamespace`/`shared`/`mountfail` sabotage nodes each reopen exactly the vector they name. I checked that `shared` is production-identical except the principal: `sabotagedLauncher` reconstructs the same `credentialDirectory`, `qmpDirectory`, `jailRoot` and volume set the host module derives, differing only in `principal = "microvm-shared"`. Do not narrow the procfs cases to direct-path checks — nexus#23's second comment is explicit that this is the specific way to close the issue with the boundary still open. The assertion/fixture pairing in `host.nix` ↔ `fixtures.nix` is complete and non-vacuous. Every assertion I enumerated (two global path/overlap checks, the undeclared-machine and duplicate-QMP/duplicate-image checks, and the per-VM socket, QMP-location, `user`, `vsock.cid`, credential-name-set, exact-credential-path, credential-source, and four volume checks) has a mutation that trips precisely it, and the 26 mutations exceed the check's `>= 24` floor. The three-way `probe` classification (`fixtures.nix` 273-296) genuinely closes the `tryEval` blind spot the brief warned about: an `abort` escaping `tryEval` takes the whole check down at eval time rather than reading as a pass, a type-rejected value is required to evaluate to `false`, and an assertion mutation that computes a diagnostic but never trips is caught as "no assertion failed." The deliberate omission of an overlong-VM-name fixture, with the reasoning recorded at host.nix 229-235 and fixtures.nix 474-477 (nixpkgs' raw `builtins.abort` on a >31-char group name cannot be paired), is the correct call, not a hole. The launcher/upstream-unit interaction is correct across the lifecycle. The QEMU runner exposes no `supportsNotifySocket` (`lib/runner.nix` 24 -> `or false`) and `useNotifySockets` defaults false, so the unit is `Type=simple` and the launcher's `pivot_root` cannot strand a filesystem `NOTIFY_SOCKET` — a real trap the design avoids by inheritance, worth keeping in mind if a future runner sets notify. `ExecStop` is left as upstream's `booted/bin/microvm-shutdown` and now runs as root in the host namespace, reaching the QMP socket QEMU created because `qmpDirectory` is bind-mounted into the jail at its identical absolute path (contract 8 / the pass-3 fix); `shutdownCommand` blocks on `cat` until QEMU exits, so the `mkAfter` `ExecStopPost` cleanup only removes the QMP directory after the socket is dead. The `ExecStopPost` ordering (upstream `microvm-unregister` from the template, then nexus cleanup from the instance drop-in) is the resolved systemd order, checked both statically and on the booted host. The state-directory tightening to `0755 root:root` does not break `install-microvm-@` (root) or `microvm-set-booted@` (writes inside `${stateDir}/%i`, still `microvm:kvm`), and it does close a runner principal's ability to rewrite the `booted` symlink the root unit executes. `WorkingDirectory` stays upstream's state dir but the jail's `cd ${jailRoot}` / `pivot_root` / `cd /` discards it, verified against `/proc/<pid>/cwd` and `/proc/<pid>/root/.../booted` in the test. Contract 21a is bounded exactly as the plan pins it: `hardware.ksm.enable = false` neutralizes the `mem-merge=on` that `lib/runners/qemu.nix` line 85 sets unconditionally on x86_64, the `microvm` state-package/`bridge.conf`/bridge-helper/libvirt surface is asserted unchanged, and the two inert upstream additions (`boot.kernelModules`, the `memlock` PAM grant with no `PAMName`) are left alone. The namespace is genuinely load-bearing for function — every source parent is created `0700 root:root`, so `bind_file`/`bind_dir` exposing only the leaf is what makes a runner reach its own credential and QMP objects without making a sibling's reachable — and the `nonamespace` node pins that measured outcome so a later change cannot silently turn the namespace into mere hardening. The `credentialSources` split, the `strMatching "^/run/allod/..."` option type doing the "never a Nix path, never outside /run/allod" half of contract 8a before any assertion, and the `size=16384k` reported-form match between the mount unit and the launcher preflight are all correct and should survive later passes. One non-finding worth stating for the next reader: the `vm` input still pins the unmerged `agent/vm-microvm-guest-split` branch (`flake.lock` rev `783bf6a`), which the PR body discloses and which must move to the merged revision when allod/vm#5 lands — the normal milestone-2-before-3 sequencing gate, not a defect in this diff. The lock holds exactly one `microvm` node at `39a499ab` and no `microvm` input at the nexus root, so the nexus half of contract 1a holds.

allod-agent referenced this pull request from allod/strategy

2026-07-30 08:41:27 +01:00

Drop the host-side volume placement option and re-attribute the namespace sabotage #23

allod-agent added 1 commit

2026-07-30 08:41:37 +01:00

Drop the duplicate host-side volume placement option 16bd9c7b24

allod-agent commented

2026-07-30 08:41:58 +01:00

Author

Member

The SIMPLIFY is implemented in 16bd9c7. nexus.microvm.machines.<name>.volumeImages, the guestVolumeImages derivation, the equality assertion and the host-and-guest-volume-disagreement mutation are gone; the machines submodule now carries credentialSources only, with a comment recording why there is no placement option beside it. docs/microvm-host.md says the same. 25 mutation fixtures remain, still above the check's floor.

The argument that decided it, beyond the one in the review: a credential has a real source/destination split, so credentialSources names a fact that exists nowhere else, while a volume image does not — its host path can only live in guest.microvm.volumes[].image because that is what QEMU renders into -drive file= and what the launcher preflights.

One piece of evidence worth recording, because it settles the "is this option load-bearing" question better than reading can: removing it left microvm-host-isolation's derivation byte-identical to the one that already passed — same .drv, same output path — so the option demonstrably fed no generated artefact. microvm-host-contract rebuilt (its mutation set changed) and is green.

The plan said this in two places, so it has been trimmed to match through the normal change process rather than left to contradict the code: allod/strategy#23 drops the volume clause from contract 17, drops "writable-volume placement" from the nexus scope bullet, and rewrites the persistent-state ownership bullet to say where placement actually enters. That PR also re-attributes the namespace sabotage to what implementation measured — removing the namespace does not reopen the direct-path attacks, it stops the VM starting at all — which is the case the plan reserves for a measurement that contradicts it. Whoever wires archetypes in milestone 4 should read both changes before deciding how image paths reach the guest.

The SIMPLIFY is implemented in `16bd9c7`. `nexus.microvm.machines.<name>.volumeImages`, the `guestVolumeImages` derivation, the equality assertion and the `host-and-guest-volume-disagreement` mutation are gone; the `machines` submodule now carries `credentialSources` only, with a comment recording why there is no placement option beside it. `docs/microvm-host.md` says the same. 25 mutation fixtures remain, still above the check's floor. The argument that decided it, beyond the one in the review: a credential has a real source/destination split, so `credentialSources` names a fact that exists nowhere else, while a volume image does not — its host path can only live in `guest.microvm.volumes[].image` because that is what QEMU renders into `-drive file=` and what the launcher preflights. One piece of evidence worth recording, because it settles the "is this option load-bearing" question better than reading can: removing it left `microvm-host-isolation`'s derivation byte-identical to the one that already passed — same `.drv`, same output path — so the option demonstrably fed no generated artefact. `microvm-host-contract` rebuilt (its mutation set changed) and is green. The plan said this in two places, so it has been trimmed to match through the normal change process rather than left to contradict the code: allod/strategy#23 drops the volume clause from contract 17, drops "writable-volume placement" from the nexus scope bullet, and rewrites the persistent-state ownership bullet to say where placement actually enters. That PR also re-attributes the namespace sabotage to what implementation measured — removing the namespace does not reopen the direct-path attacks, it stops the VM starting at all — which is the case the plan reserves for a measurement that contradicts it. Whoever wires archetypes in milestone 4 should read both changes before deciding how image paths reach the guest.

allod-agent added 1 commit

2026-07-30 22:45:34 +01:00

Point the vm input at the merged master revision 1f96145c19

allod-agent commented

2026-07-30 22:45:44 +01:00

Author

Member

The vm input now points at allod/vm master, which is where milestone 2 landed. 1f96145 changes the URL to drop the branch ref; the lock moves from the branch head to the same revision on master, because the merge was a fast-forward and the revision is unchanged.

Re-verified at the new pin, not assumed:

nix flake check green.
microvm-host-contract green.
microvm-host-isolation green — four probe runs, 32 cases each, zero unexpected outcomes, including the shared-principal node where the procfs attacks must reopen.
The lock still holds exactly one microvm node, which is what contract 1a requires of this repo.

That clears the last thing this PR was waiting on.

The `vm` input now points at `allod/vm` master, which is where milestone 2 landed. `1f96145` changes the URL to drop the branch ref; the lock moves from the branch head to the same revision on master, because the merge was a fast-forward and the revision is unchanged. Re-verified at the new pin, not assumed: - `nix flake check` green. - `microvm-host-contract` green. - `microvm-host-isolation` green — four probe runs, 32 cases each, zero unexpected outcomes, including the shared-principal node where the procfs attacks must reopen. - The lock still holds exactly one `microvm` node, which is what contract 1a requires of this repo. That clears the last thing this PR was waiting on.