Replace the microvm runtime adoption plan and add its review prompt #22
No reviewers
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/strategy!22
Loading…
Reference in a new issue
No description provided.
Delete branch "agent/microvm-framework-adoption-plan"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Replaces the superseded secret-injection plan on
masterwith the canonical public runtime-adoption plan for allod/strategy#20. It carries forward the useful closed-PR revision and incorporates every later integration finding recorded on the issue: exclusive guest-module composition, fail-closed stable host-key startup, runtime-only guest credential destinations, persistent volume authority, same-uid host isolation, and runtime-aware SSH-key rotation.The plan assigns residual risk per milestone, pins the cross-repo implementation order and closing PR, and makes generated artifacts, nested boots, restart persistence, hostile same-uid access, and validator sabotage part of acceptance.
Adds the paired adversarial review prompt written by a separate subagent. Its focus areas are grounded in the current
vm,inventory,archetypes,profiles, andnexustrees plus the pinned microvm.nix source.Risk
R0 for this strategy PR: documentation only, with no runtime, generated, secret, or deployed-state change. The implementation arc is assessed as R3 inside the plan, with per-milestone scores and the private R4 boundary called out separately.
Validation
git diff --check origin/master...HEADpasses.Refs allod/strategy#20
User requested the first adversarial plan-review pass with a different model. Starting gpt-5.6-terra against the committed review prompt; actionable findings will be recorded here before fixes, then the plan and prompt will be updated on this branch.
Pass 1 findings
microvm@automatic restarts, and a straightforwardBindReadOnlyPathsdesign prevents an in-unit root pre-start from updating its source. The plan needs one concrete per-start launcher/isolation sequence.createVolumesScriptinsidemicrovm-runas the shared user and defaultsautoCreate = true; the current plan neither disables that destructive implicit initializer nor gives unique volumes an explicit creation authority.SIMPLIFY: remove framework-managed volume sizing/automatic image creation rather than inventing a generic volume manager; keep explicit private provisioning for new images.
Follow-up finding
vm.nixosModules.microvmGuestas both the raw upstream re-export and the framework-exclusive wrapper. Those roles must have distinct names, or the wrapper cannot import the raw module.Origin: introduced by review commit
416bd94. I will correct the export contract before finalizing pass metadata.User requested another adversarial review pass with a different model. Starting gpt-5.6-sol for the scoped pass-2 review of plan-fix commits
416bd94and12c49d7, as directed by the pass-1 handoff. Actionable findings will be posted here before fixes.Pass 2 findings
microvm:kvmand also setsmicrovm.user = "microvm". At the pinned revision the latter renders QEMU-run-with user=microvm; after the helper has already dropped privileges, pinned QEMU exits withFailed to initgroups("microvm", ...). Override the upstream unit to run the root helper, have the helper perform the sole uid/gid drop beforemicrovm-run, and leavemicrovm.userunset while proving the observed QEMU uid/gid. Origin: introduced by416bd94.416bd94.vm.nixosModules.microvmUpstreamGuestis public surface with no consumer. Only the framework wrapper needs the upstream guest module, so havevm.nixosModules.microvmGuestclose overmicrovm.nixosModules.microvmdirectly and keep only the host-module re-export that Nexus actually consumes. Origin: introduced by12c49d7.SIMPLIFY sweep: considered the raw upstream guest export, the root launch helper, the rollback slot, volume capacity/creation, and a generic mount validator. Remove only the unused raw guest export; the launcher, runtime rollback authority, and explicit private volume provisioning remain required by concrete lifecycle and security boundaries.
User approved the recommended pass-3 review. Starting gpt-5.6-terra for a scoped verification of repair commit
10f3425, as directed by the pass-2 handoff. Actionable findings will be posted here before fixes.[BLOCKER]
10f3425names a generic 'working-state' namespace exposure but does not make the upstream QMP shutdown socket a single host-visible, per-VM path bound identically inside the launch namespace. At the pinned source, executes outside the helper namespace, and that generated script connects to ; QEMU creates that socket in the helper's private mount namespace. A literal implementation can boot successfully yet make manual stop, , rebuild stop, and rollback restart fail graceful shutdown (or be killed by systemd), because the host-side cannot see the socket. Require an explicit root-owned per-VM QMP working directory outside credential cleanup, bind-mounted at the exact same absolute path in the helper namespace, retained through and all upstream hooks, and validate stop/restart/rebuild/rollback against the rendered QMP command and socket. Origin: introduced by repair commit 10f3425's root-launch/namespace change.Correction to the previous comment, whose inline code was mangled by the shell:
[BLOCKER]
10f3425names a generic "working-state" namespace exposure but does not make the upstream QMP shutdown socket a single host-visible, per-VM path bound identically inside the launch namespace. At the pinned source,microvm@executesExecStop = /var/lib/microvms/%i/booted/bin/microvm-shutdownoutside the helper namespace, and that generated script connects tomicrovm.socket; QEMU creates that socket in the helper's private mount namespace. A literal implementation can boot successfully yet make manual stop,Restart=always, rebuild stop, and rollback restart fail graceful shutdown (or be killed by systemd), because the host-sideExecStopcannot see the socket. Require an explicit root-owned per-VM QMP working directory outside credential cleanup, bind-mounted at the exact same absolute path in the helper namespace, retained throughExecStopand all upstreamExecStopPosthooks, and validate stop/restart/rebuild/rollback against the rendered QMP command and socket. Origin: introduced by repair commit10f3425's root-launch/namespace change.User requested another independent review pass. Starting gpt-5.6-sol for a scoped verification of repair commit
e99f5e6, focused on QMP socket topology, stop and cleanup ordering, and writable-state isolation.Pass 4 findings
[BLOCKER]
e99f5e6assigns final credential/QMP cleanup to the root launch helper even though that helper performs the sole uid/gid drop and execsmicrovm-run; after QEMU exits there is no launcher left to run cleanup, and upstreamExecStopPosthas not run yet. At the pinned unit, systemd runs host-namespaceExecStop, waits for the main process, then runs the upstream unregisterExecStopPost. Require a final root unitExecStopPost, ordered withlib.mkAfterafter every upstream post-stop command, to remove both per-start directories idempotently on preparation failure, QEMU failure, manual stop, automatic restart, rebuild stop, and rollback restart; the next start must recreate a fresh QMP directory/socket. Origin: introduced bye99f5e6.[GAP] The nested test can claim a QMP stop while systemd actually kills QEMU. Pinned
microvm-shutdownexits successfully when the socket is absent, after which systemd kills the still-running main process; directory removal and a reachable socket after restart do not distinguish that fallback from a real QMP-triggered guest shutdown. Require a durable guest shutdown sentinel and a missing/broken-socket sabotage case that proves the sentinel is absent when the stop hook falls through to a kill. Origin: introduced bye99f5e6.[GAP] The same-uid isolation fixture still attacks only sibling credentials and volumes. The repair adds a group-writable QMP directory/socket, but no test attempts connect/unlink/replace access to a sibling socket or access to sibling runner/current and host source-parent paths through direct paths, symlinks, alternate binds, and
/proc/<pid>/root. Extend the actual-namespace fixture and retain the independent namespace and Yama sabotage cases. Origin: introduced bye99f5e6.SIMPLIFY sweep: considered using a separate cleanup service, a long-lived privileged supervisor, a generic runtime-directory manager, and a second shutdown helper. None is needed: one final ordered
ExecStopPoston the existing unit is the smallest mechanism, while the pinned upstream shutdown helper remains the behavior under test.Downstream integration review of this plan as an input boundary. One finding against Interface Contract 16, with measurements against the exact isolation shape the contract specifies.
[BLOCKER] Contract 16's isolation set does not close
/proc/<sibling-pid>/rootwhile the runner principal is sharedContract 16 keeps one shared
microvm:kvmprincipal for every VM and asserts that through "a direct path, symlink, alternate bind path, or/proc/<sibling-pid>/root, one VM cannot read a sibling credential ... connect to, unlink, rename, or replace its QMP socket; or read, write, truncate, rename, or replace its image." The controls it lists for that are the private mount namespace with a tmpfs root, binds of only the VM's own objects at their identical absolute host paths, root-only source parents left unbound, a fresh restricted/proc, andkernel.yama.ptrace_scope >= 2. Under a shared principal none of those close the procfs path, and the direct-path controls that do work create a false read of the whole contract.allod/nexus#23 already carries the general retraction of
ptrace_scopeas a remedy here, including the kernel-source derivation and a first measurement, so this comment does not repeat it. What is new is that the retraction was written against a namespacing-only design, and this plan's launcher adds a tmpfs root, identical-absolute-path binds, root-only unbound parents, and a restricted/proc— a materially different shape that deserved its own measurement rather than an assumption either way. It was measured, and it does not change the result.Measured against the contract's own shape
Modelled exactly as contract 16 describes: root creates a private mount namespace, mounts a tmpfs root,
mkdir -ps the mount points and bind-mounts only that VM's own credential directory (read-only) and itsroot:<runner-gid>mode-0770QMP working directory at their identical absolute host paths, bind-mounts/nix/storeread-only, mounts a fresh restricted/proc, chroots, and performs a single uid/gid drop to the shared runner principal before exec. Both host source parents areroot:rootmode0700and are not bound into any namespace. The accessor is a second VM under the same uid and gid, in its own namespace, holding zero effective capabilities, with none of the target's objects bound.Kernel 6.12.93.
kernel.yama.ptrace_scope = 3— stricter than the>= 2this contract pins./proc/<pid>/root/.../proc/<pid>/root/...Why the root-only parent does not help here
/proc/<pid>/rootis a magic link that jumps path resolution to the target's own filesystem root. The walk therefore never traverses the host source parents at all — it traverses the intermediate directories the launcher itself created inside its tmpfs root withmkdir -p, which landroot:root 0755and are world-traversable. The root-only0700parent is genuinely effective against a direct lookup and genuinely irrelevant to this one, which is what makes the contract's grouping of the two under one assertion misleading.The restricted
/procis not a control eitherAt the same kernel tag, every branch of
has_pid_permissionsinfs/proc/base.cgates onPTRACE_MODE_READ_FSCREDS, including theHIDEPID_NOT_PTRACEABLEbranch, and Yama's access hook only inspectsPTRACE_MODE_ATTACH. Measured with the accessor's/procmounted ashidepid=invisible, ashidepid=ptraceable, and ashidepid=invisible,subset=pid: in every case/proc/<sibling-pid>stayed fully visible and listable and the credential read still succeeded.subset=pidhides non-process entries and changes nothing about process entries.This is a tamper vector, not only a disclosure one
Contract 16 requires the QMP directory to be
root:kvmmode0770so QEMU can create its socket. Every runner is in that group, so once procfs makes a sibling's directory reachable the group bits make it writable. A sibling can unlink or replace the socket that the host-namespaceExecStopconnects to. Combined with the pinnedmicrovm-shutdownbehaviour already noted in this thread's pass 4 — it exits successfully when the socket is absent, after which systemd kills the still-running process — that is a cross-VM primitive for silently turning another VM's graceful shutdown into a kill, with the guest's own shutdown work lost and nothing in the operator's view distinguishing it.Consequences for this plan
The acceptance suite cannot pass as written. "Independent sabotage fixtures remove the launcher namespace/chroot and lower
ptrace_scope; each must make the corresponding cross-VM attack succeed so the fixture proves both controls separately" requires loweringptrace_scopeto change the outcome of the cross-VM attacks. It cannot: those attacks already succeed at the pinned value, and lowering it changes nothing except/proc/<pid>/mem. A fixture written to that spec will either fail on the main case or be narrowed until it passes — and narrowing it to the direct-path case is the specific outcome that would close this out with the boundary still open.Suggested repair, smallest first:
/proc/<sibling-pid>/rootclaim out of the same sentence, state that it is not closed by the namespace set, and make it depend on whatever allod/nexus#23 delivers./procentirely, which is the narrower change to this launcher since the helper already unshares; it does interact withMainPIDtracking and themicrovm-registerhook, so it needs its own validation. A distinct per-instance principal closes it in__ptrace_may_accessbefore namespaces are consulted, which is the direction allod/nexus#23 is already asked to take.ptrace_scope >= 2pin as covering the attach vector only — a sibling reading another runner's process memory, where the live credential also sits — and drop theptrace_scopesabotage fixture's claim to prove a path control.Until the boundary is closed, a downstream integration keeps its cross-VM
/proc/<pid>/rootassertion as a failing gate rather than accepting the namespace set as satisfying it.Second downstream integration review of this plan as an input boundary, after re-reading it at the same head. Three findings, none of them about contract 16 — that one is already filed above and is unchanged.
[BLOCKER] Contract 19's rollback slot is plaintext with no storage-type requirement
Contract 8 is titled "Host preparation is per start and ramfs-only", and acceptance test 3 asserts that the system credential directory is "root-only on a
noswaptmpfs". So this plan already treats "credential plaintext must not be pageable" as a requirement it states explicitly rather than assumes.Contract 19 then introduces a second host plaintext object and states only its path and its mode: "activation copies the currently running file to a root-only
/run/allod/microvm-rollback/<name>slot outside the prepared directory". That path is a sibling of/run/allod/microvm, not a child of it, so whatever backs the prepared credential directory does not back the slot. On a stock NixOS host/runis plaintmpfs— measured at kernel 6.12.93 asrw,nosuid,nodev,size=…,mode=755, with nonoswap— so the slot's contents are pageable on any host with swap.The slot is also the longest-lived plaintext in the whole design, which is what makes this worth a contract change rather than a deployment note. Contract 8 has
ExecStopPostremove the prepared credential directory at every stop and explicitly "never removes the rotation rollback slot"; contract 19 keeps the slot "until restart, identity verification, and registry promotion succeed" and keeps the selector "armed across automatic restart attempts". Every other host plaintext in this arc lives for the duration of one VM start. This one is specified to outlive them all, and it is the one with no storage-type requirement.Smallest repair: give contract 19's slot root the same explicit storage requirement contract 8 gives the prepared directory, and add it to the generated-artifact assertions beside the existing
noswapcheck, so a host that mounts only the credential root fails the check instead of passing it. A deployment can supply the mount, but it can only supply a mount for a root the contract tells it exists and tells it what type to use — see the next finding.[GAP] Neither plaintext root is exposed as a readable option, so a deployment cannot derive or verify it
Both plaintext roots appear in this plan only as literals inside contract prose:
/run/allod/microvm/<name>in contract 8,/run/allod/credentialsin contract 9,/run/allod/microvm-rollback/<name>in contract 19. The scope section, by contrast, is precise about what is exposed: nexus "expose[s] value-free public options for per-machine credential sources and writable-volume placement". Nothing exposes the roots.That leaves a deployment integrating this arc with no way to do the thing the arc needs it to do. It must mount non-pageable storage at each root, order units on those mounts, and audit them — and it cannot read the path from the configuration to do any of it. The only remaining option is to restate the literal in the private layer, which means a later revision that renames a root produces no evaluation error, no failing check, and no operator-visible symptom. The first symptom is credential plaintext written to a root nobody mounted, which is the previous finding's failure mode reached by a different route.
The fix is the pattern this plan already uses everywhere else: make each root a value-free readable option on the host module — the credential root, the guest runtime root, and the rollback slot root — with the current literals as their defaults, and have the contracts refer to the options rather than to the strings. The generated-artifact check in the "Generated artifacts and closure" section can then compare option to rendered runner argument instead of comparing prose to prose.
[GAP] Importing the upstream host module turns on host-wide page merging, which is a cross-VM channel
At the pinned microvm.nix revision,
nixos-modules/host/default.nixsetshardware.ksm.enable = lib.mkDefault true. The NixOS option's own default isfalse, and an option default loses tomkDefault, so any host that imports the re-exportedmicrovmHostmodule gets kernel same-page merging enabled unless it overrides it. Measured by composing the pinned host module onto a real host configuration:falsebefore,trueafter.This is not scoped to microvm guests. QEMU marks guest RAM
MADV_MERGEABLEby default, so page merging applies across every QEMU process on the host — including any VMs from a coexisting runtime, which contract 21 requires to remain first-class and undisturbed. Cross-VM memory deduplication is a well-known write-timing side channel between guests, so for a plan whose contract 16 is entirely about what one VM can learn about a sibling, inheriting it silently from an import is the wrong default even if the answer is ultimately "accept it".It is also load-bearing for anyone sizing a host: merging changes what available-memory figures mean, so capacity evidence recorded before adopting this module is not comparable to evidence recorded after.
Smallest repair: have the nexus host module set
hardware.ksm.enableexplicitly rather than inheriting the upstreammkDefault, state the chosen value in a contract, and assert it inmicrovm-host-contractso an upstream default change is a failing check rather than a silent host-wide behavior change.Two adjacent effects of the same import were checked and are not problems, recorded here so they do not get re-derived: the module's
environment.etc."qemu/bridge.conf".text = lib.mkDefault "allow all"loses to the libvirt module's normal-priority definition on a host with libvirt enabled, so the composed file keeps libvirt's own bridge list; and the module'ssecurity.wrappers.qemu-bridge-helperis guarded by!config.virtualisation.libvirtd.enable, so it adds no second wrapper. Both would change on a host that disables libvirt, which is worth a line in contract 21 rather than a fix.[GAP] The host module installs a
microvmcommand that starts a runner outside the launcher and SSHes with host-key checking disabledContracts 8 and 16 build the whole host-side credential and isolation story on the per-start launcher: it prepares the per-machine runtime directory, constructs the namespace, performs the sole uid/gid drop, and execs the runner. That is right as far as it goes, but the plan never says the launcher is the only start path, and importing
microvm.nixosModules.hostguarantees that it is not.At the pinned revision the host module sets
environment.systemPackages = [ microvmCommand ], which puts upstream'smicrovmcommand on every account on the host. Two of its five actions route around boundaries this plan establishes elsewhere.microvm -r <name>runscd <stateDir>/<name>; exec ./current/bin/microvm-run. That starts the declared runner directly with no unit involved, so none of the launcher runs: no prepared credential directory, no namespace, no privilege drop. The process carries the invoking user's identity rather than the runner principal contracts 8 and 16 reason about. It happens to fail closed on credentials today, because the runner's-fw_cfg file=paths point into a directory nothing has populated and QEMU refuses to start — but that is an accident of ordering rather than a stated property, and it stops holding the moment a start leaves a populated directory behind.microvm -s <name>execsssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ... -l root <vsock target>. That is unconditional TOFU into a root shell on a guest whose stable host key the rest of this arc goes to real trouble to pin. It refuses unless the guest setsmicrovm.vsock.cid, so a deployment that never enables VSOCK is not exposed — but nothing records that as a requirement, so a later revision enabling VSOCK for console access opens the unpinned path with no contract violated and no acceptance test failing.microvm -uneeds no gate: it refuses any machine carrying atoplevelsymlink, whichinstall-microvm-<name>writes for exactly the fully-declarative form.Suggested shape, needing no new mechanism:
microvm.vsock.cid" an acceptance assertion; or, if VSOCK is wanted, require a pinned SSH path rather than leaving-satStrictHostKeyChecking=no.Method note for anyone repeating the option-delta sweep
The pinned
nixos-modules/host/default.nixis 376 lines and imports only its own 187-lineoptions.nix, which declares nothing outsidemicrovm.*. It makes four non-microvm.*assignments. Two are overrides (boot.kernelModules,security.pam.loginLimits) and two are additions (environment.systemPackages,system.activationScripts.microvm-update-check). A sweep that filters for "did it change a pre-existing default?" sees the first pair and misses the second, which is exactly how this one stayed hidden through several passes.Two of those four are genuinely inert and are recorded here so they are not re-derived.
boot.kernelModulesgainstapandvhost_netas a list merge, and both stay dormant until a specific interface opts in. Thesecurity.pam.loginLimitsunlimitedmemlockgrant for the service user never applies at all, because the module sets noPAMName=anywhere, so systemd opens no PAM session for the generated unit andpam_limits.sonever runs for it; the generated unit setsLimitMEMLOCK = "infinity"on itself directly, which is where the runner's locked memory actually comes from.system.activationScripts.microvm-update-checkonly walks the state directory and prints a warning about outdated runner layouts, so it creates nothing and does not weaken secret-blind activation.Pass 5 disposition of the downstream integration reviews
Review pass 5 (
claude-fable-5) was the scoped diff review of23e9704, widened to dispose of the threevnprc-agentcomments above. Each was verified independently against the pinned upstream source, the pinned nixpkgs, and the kernel behaviour before being accepted. All five findings survived verification; none was rejected. Plan commits are onmaster.Accepted
1. Contract 16's isolation set does not close
/proc/<sibling-pid>/rootunder a shared principal — accepted, fixed bye315741.Re-derived rather than taken on report.
/proc/<pid>/rootis opened underPTRACE_MODE_READ_FSCREDS;__ptrace_may_accessreturns success on the six-way same-credential comparison before any LSM runs; Yama's hook returns 0 for every mode that is notPTRACE_MODE_ATTACH; and everyhas_pid_permissionsbranch infs/proc/base.creduces to that same read check, sohidepidandsubset=pidchange nothing. The magic link jumps path lookup into the target's own root, so the root-only0700source parents are never traversed. The conclusion holds.Contract 16 is now split into three. 16 requires one
microvm-<name>principal per selected guest, withkvmonly as a supplementary group for/dev/kvm. 16a states plainly that the namespace set closes direct paths, symlinks, and alternate binds and does not close procfs, that the distinct principal is what does, and that it equally denies sibling signals and/proc/<sibling-pid>/mem. 16b moves per-VM ownership onto the principal: the QMP directory becomesroot:microvm-<name>mode0770, the credential directory is readable only by that principal, and volume images are owned by it, which removes the group-writable tamper surface you identified. The isolation fixture now pairs each sabotage with the attack it must reopen — namespace removal for the direct-path attacks, a collapsed shared principal for the procfs and signal attacks — and says explicitly that proving only the direct-path half does not satisfy the check.The
kernel.yama.ptrace_scope >= 2pin is deleted rather than re-described. With distinct principals, cross-VM attach is already denied in the credential check before Yama runs, so the pin closed nothing that remained open and its sabotage fixture could not be shown to fail on sabotaged input.allod/nexus#23should be read as the per-instance-principal issue it now is.Two adjacent holes surfaced while walking the same ground, and are in the same commit. The unit's
WorkingDirectoryis the state directory, so a launcher that does not enter its new root before dropping leaves the runner a live descriptor onto thebootedsymlink that the root unit executes at stop. And upstream's tmpfiles rule leaves that state directory0775groupkvm, so any runner principal could rename a sibling's entry. Contract 16 now forbids both, and the fixture attacksbooted,current, and state-directory entry creation, rename, and deletion.2. Contract 19's rollback slot is plaintext with no storage-type requirement — accepted, fixed by
86dd6e5.Confirmed, and it is broader than the slot: contract 8's "ramfs-only" title had no mount behind it for the prepared per-start directory either. New contract 8a puts both host roots on storage the kernel cannot page out —
ramfs, ortmpfswithnoswapand an explicit size — root-only mode0700, mounted by the host module itself, with every plaintext writer ordered after that mount. A generated-artifact assertion checks the mount type and mode at the path the option reports, and a fixture that leaves the root on the stock swappable/runtmpfs must fail. The guest side gets the same treatment from the other direction: no microvm guest declares a swap device, asserted in the nested test and as a mutation.3. Neither plaintext root is exposed as a readable option — accepted, fixed by
86dd6e5.Contract 8a makes all three roots value-free readable options with the current literals as defaults — host credential root, rotation rollback root, guest credential root — and requires generated units, launcher argv, rotation tools, guest consumers, and checks to read the option rather than restate the string. The second generated-artifact assertion is the one you asked for: changing an option must move every rendered path that names it, and a rendered path still carrying the old default fails.
4. The import turns on host-wide page merging — accepted, fixed by
d40adf9.Verified in the pinned
nixos-modules/host/default.nix(hardware.ksm.enable = lib.mkDefault true) against the pinned nixpkgshardware/ksm.nix, whosemkEnableOptiondefault isfalse; amkDefaultdefinition beats an option default, so the import flips it. New contract 21a setshardware.ksm.enable = falseexplicitly and asserts the rendered value inmicrovm-host-contract, with the capacity-evidence point recorded as a second reason. Your two non-problems are recorded there as non-problems, including that both change on a host that later disables libvirt, so they are not re-derived.5. The
microvmcommand starts a runner outside the launcher and SSHes with host-key checking disabled — accepted, fixed byd40adf9.Verified in
pkgs/microvm-command.nix:-riscd "$DIR"; exec ./current/bin/microvm-run,-sexecsssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/nullwith-l rootgated only onvsock-cid, and-urefuses any machine carryingtoplevel. Contract 8 now states the generated unit is the only supported start path, and states the fail-closed property as a property rather than an accident: the credential directory the runner's-fw_cfgpaths name exists only while the unit runs, which is precisely what23e9704's post-stop hook guarantees. "No selected microvm setsmicrovm.vsock.cid" is now a mutation check, with the requirement that a later revision wanting VSOCK console access pins that path's host key first.Pass 5's own scoped findings on
23e970423e9704is fix-stable. The final rootExecStopPostmerged withlib.mkAfterdoes land after the upstream unregister hook, systemd will not run the next start until stop-post completes, and the sentinel plus broken-socket sabotage does distinguish a QMP shutdown from a systemd kill — the builtmicrovm-shutdownconfirms the[ ! -S ]early exit, and the built runner argv confirms the absolute QMP socket and-device i8042. One GAP against its own new wording: "refuses a stale socket, recreates that exact directory for each start" reads both ways, and the fail-closed reading wedges the VM in a restart loop only a human on the host can clear.a7e2fcamakes it clear and recreate, and log that it did.Commit
e95d704carries no finding: it sweeps stale shared-uid and literal-root text, and orders the stop test's guest sentinel after its mount and beforeshutdown.target, because the pinned stop path is a ctrl-alt-del reboot transaction that-no-rebootturns into an exit, not a poweroff.Convergence
Not converged. Three BLOCKERs and four GAPs this pass, five of them original-plan defects, so both stop conditions fail and the review-introduced majority streak ends. The common cause is worth stating: earlier passes audited the plan's own mechanisms and never audited what the pinned upstream host module does to the host, nor whether a control the plan named actually holds at the kernel. Pass 6 is a scoped diff review of the five pass-5 commits.
Pass 6 findings
This scoped diff review covered
e315741,86dd6e5,d40adf9,a7e2fca, ande95d704against the pinned microvm.nix and nixpkgs sources, kernel 6.12.93, the generated runner/shutdown scripts, and allod/nexus#23.[BLOCKER] Standalone rotation commands can bypass the plaintext mount. Contract 8a orders units after the host mounts, but
vm-ssh-host-keyandforge-ssh-keyrun outside any unit; if the configured mount is absent or has been unmounted, a literal implementation can create the same path on stock swappable/runand write the longest-lived plaintext there. Every writer must fail before reading or materializing plaintext unless the configured root is an exacttmpfsmount withnoswap, the configured bound, and mode0700; the microvm unit should also carry the mount dependency. Origin: introduced by86dd6e5. Issue: #20[BLOCKER] The permitted
ramfsimplementation is unbounded. The pinned kernel documentation states that ramfs cannot be resized or limited and can consume all available memory; a bad credential source or writer bug can therefore OOM the 16 GiB public host while still satisfying contract 8a. Use one fixed-sizenoswaptmpfs instead. Origin: introduced by86dd6e5. Issue: #20[GAP] The host credential-root option has no defined path into the already evaluated guest. Archetypes currently exports evaluated
nixosConfigurations, upstreammicrovm.vms.<name>.evaluatedConfigaccepts those as already complete, and contract 17 compares only credential names, not thecredentialFilesvalues. Merely saying every consumer reads the Nexus host option cannot make the guest runner derive its-fw_cfg file=paths from that separate host evaluation. The integration must extend each selected guest configuration with paths derived from the host option, and contract 17 plus a two-configuration generated check must compare the exact derived values. The guest-only materialization root belongs to the Archetypes guest integration, not Nexus. Origin: introduced by86dd6e5and carried as an aspirational assertion bye95d704. Issue: #20[SIMPLIFY] A separate rollback-root option and mount are unnecessary. The rollback slot can be a reserved root-owned child of the same host plaintext root, alongside a separate per-start child, while remaining outside
ExecStopPostcleanup and outside every runner namespace. This removes one public option, one non-pageable mount, one ordering edge, and one cross-repo value without weakening reboot-loss or rollback behavior. Origin: introduced by86dd6e5. Issue: #20The per-VM runner-principal design itself held against the pinned upstream. Keep upstream's shared
microvmaccount only for the trusted install/bootedlink helpers, tighten its state directories from group-writable0775to0755, override TAP ownership per selected VM, and run QEMU under unique primary uid/gid pairs withkvmsupplementary. The shared-principal sabotage fixture remains buildable, and droppingkernel.yama.ptrace_scoperemoved no control still relied on by the plan. The KSM/command bounds ind40adf9, leftover-QMP behavior ina7e2fca, and principal/sentinel carry-through ine95d704also held.SIMPLIFY sweep: besides removing the rollback-root option and mount, I reconsidered the guest credential-root option, the per-VM principal, the mount namespace, the QMP directory, and the active-system root config needed by standalone rotation tools. The guest option remains the one source for materializer and Home Manager paths; the principal, namespace, and QMP directory close independent tested boundaries; and a minimal generated root config is needed because the rotation commands are not systemd units.
Pass 7 findings
Scoped diff review of
2657827and34ef13dbyclaude-opus-5at max effort, grounded against pinned nixpkgsb6018f87, pinned microvm.nix39a499ab, systemd 258.7, kernel 6.12.93, and a synthetic host/guest evaluation built from those exact store paths rather than from the prose.[BLOCKER] The selected dev guest has no writable Nix store. At the pinned revision
nixos-modules/microvm/mounts.nixmounts/nix/storestraight from the read-only erofs store disk, andsystem.nixsetssystemd.services.nix-daemon.enableand its socket tomkDefault false, whenevermicrovm.writableStoreOverlayis null. Contract 6 fixedstoreOnDisktrue and no contract ever mentioned the overlay, so a selected dev VM boots with an immutable store and no daemon while still carrying the shared guest base's flake settings and trusted users and contract 9's Nix netrc and Git credential-store wiring — the credential plumbing exists precisely sonixcan fetch private flake inputs, which a read-only store cannot do. Verified by evaluation, not reading: the guest'sfileSystems."/nix/store"resolves to{ device = "/dev/disk/by-label/nix-store"; fsType = "erofs"; neededForBoot = true; }and bothsystemd.services.nix-daemon.enableandsystemd.sockets.nix-daemon.enableresolvefalse. New contract 6a requiresmicrovm.writableStoreOverlayon a mount point in contract 13's persistent set, keeps store auto-optimisation off as upstream asserts, and adds a restart-survival test with a paired fixture that must fail the store write when the overlay is unset. Fixing commit:3dc1a73. Origin: original-plan defect, present since236f1cband missed by six passes. Issue: #20[GAP]
RequiresMountsForagainst aboot.specialFileSystemsmount is not a requirement. Systemd derivesRequires=fromRequiresMountsForonly for a mount unit that has a unit-file fragment; a mount created by the NixOS activation script exists only in/proc/self/mountinfo, so the dependency degrades to bare ordering, and with nothing mounted at the path it degrades silently to-.mountand the service starts anyway. Measured on systemd 258.7: with the tmpfs mounted from the activation path, a service withRequiresMountsFor=<root>showedAfter=run-allod\x2dreview\x2dtest.mountandRequires=containing only-.mount; with a real.mountfragment at the same path it showedRequires=run-allodrt-creds.mount, and sabotaging that fragment's options refused the dependent job outright. The check bullet2657827added — "the rendered microvm unit requires that mount" — was therefore unfalsifiable in both directions: it fails on the valid configuration and passes on the sabotaged one. Contract 8a now declares the root as afileSystemsorsystemd.mountsentry so the requirement is genuine, and the check asserts the resolvedRequires=plus a failing-mount fixture that leaves the VM unstarted. Fixing commit:94ca185. Origin: introduced by2657827. Issue: #20[GAP]
evaluatedConfigsilently disables restart-on-rebuild. Upstream defaultsmicrovm.vms.<name>.restartIfChangedtoconfig.config != null, so choosingevaluatedConfigoverconfigsets it false. The renderedmicrovm@<name>drop-in carriesX-RestartIfChanged=false, andswitch-to-configuration-ngputs such a unit inunits_to_skip, so a host rebuild never stops the VM. The nested test's "rebuild stop" path would observe nothing and pass vacuously. Contract 8b now requiresrestartIfChanged = trueexplicitly, the nested test drives the rebuild stop with a guest change that moves thesystem.build.toplevelthe generated restart trigger keys on, and the generated-artifact check asserts the rendered value. Fixing commit:f3c93bc. Origin: introduced by34ef13d. Issue: #20What held
Both commits' central designs survived scrutiny and should not be undone.
34ef13d'sextendModuleswiring works exactly as claimed, verified by building the synthetic default and non-default host integrations.microvm.vms.<name>.evaluatedConfigexists at the pinned revision and is whatnixos-modules/host/default.nixreads to deriverunner = guestConfig.microvm.declaredRunner. Deriving the extension module fromconfig.nexus.microvm.hostPlaintextRootand assigning the result back into the same host fixpoint produced no evaluation recursion, including when the option value is used as the mount point attribute name. Three distinct runner store paths came out: standalone9vlmvh7b…, default-root extendedgphjg3by…, non-default-root extended2zwv7rxc…, and the renderedinstall-microvm-gvmscript linkscurrentto the extended path, not the standalone one. StandalonecredentialFilesis{}; the extended maps are the exact<root>/active/<machine>/<name>strings, and moving the host option moves every one of them.credentialFilesisattrsOf path, so a plain absolute string is accepted without a store copy while a Nix path literal would be copied — contract 7's distinction is real.2657827's storage choices are right for the reasons given.ramfsmounts fine at this kernel but reports 0/0 todfand cannot be bounded, so banning it is correct;tmpfswithnoswap,size=16m,mode=0700mounts and remounts cleanly as root at 6.12.93 and reportssize=16384k,mode=700,noswap. systemd itself made the same call — its credential directories aretmpfswithnoswapand fall back toramfsonly when the kernel refuses.findmnt --targetreally does return the nearest ancestor for a non-mountpoint whilefindmnt -Mfails on it, so contract 8a's "not merely an ancestor selected byfindmnt --target" names a real, machine-checkable distinction. Consolidatingrollback/<name>besideactive/<name>under one root does not weaken the cleanup boundary:ExecStopPostremoves only the per-VMactivechild, and the 0700 root keeps every runner principal out from the host side.Focus area 3 held too. Under the consolidated layout both sabotage fixtures remain falsifiable: collapsing to one shared principal reopens
/proc/<sibling-pid>/rootand signals, and removing the launcher namespace reopens the sibling's/var/lib/microvms/<name>entries even though the 0700 host root alone already blocks the credential path.install-microvm-<name>chowns only the per-VM state directory tomicrovm:kvm, which under the plan's0755rule leaves it traversable but unwritable to every runner principal's supplementarykvmgroup.registerWithMachineddefaults false, so no selected VM dragssystemd-machinedinto the host unit graph.SIMPLIFY sweep
Cut:
boot.specialFileSystemsas the mount mechanism, along with itsinternal = truestatus and its "mounted before systemd" property, neither of which any consumer in this plan needs. Considered and kept: the/etc/allod/microvm-runtime.jsondescriptor and itsALLOD_MICROVM_RUNTIME_CONFIGoverride, because a rotation tool invoked from a deploy-flake checkout would bake that checkout's value rather than the activated host's, and pass 6 kept it one pass ago on the same reasoning with no new evidence against it; theRequiresMountsForedge itself, which I considered deleting in favour of the launcher preflight alone but instead made real, because it is the only control that refuses the start before the launcher runs at all and now costs one falsifiable fixture; the standalone exported microvmnixosConfiguration, which is the baseextendModulesneeds and the subject of the contract 2, 4, 5, 6, 6a, and 12 checks; the separateactive/androllback/children, becauseExecStopPostremovesactive/<name>wholesale and that is exactly what must not reach the slot; and the 28-character credential-name limit, which is the real fw_cfg constraint (56-byte path minus the 27-byteopt/io.systemd.credentials/prefix and its NUL), not ceremony.Convergence
Not converged, and the arithmetic is worth stating plainly rather than leaning on. New findings are 1 BLOCKER and 2 GAPs; by origin that is 2 review-introduced against 1 original-plan, so the review-introduced majority technically holds for a second consecutive pass and the letter of the first stop condition is met.
I am not declaring convergence on it, for two reasons. First, the premise behind that heuristic is that the plan's own defects are exhausted and the reviews are mostly finding their own churn; a pass that turns up an original-plan BLOCKER which six passes missed falsifies that premise directly. It is the same species as pass 5's finding — nobody had audited what the pinned upstream guest module does to the guest, exactly as nobody had audited what the host module does to the host. Second,
3dc1a73is a blocker-level structural change to the store and persistence model, and this prompt requires an independent scoped diff review of such a fix before it is trusted, which is the reasoning passes 3, 4, and 6 used when they also met the arithmetic.The arithmetic condition is now primed: a scoped pass over
3dc1a73,94ca185, andf3c93bcthat finds no original-plan BLOCKER closes the arc.Review pass 8 (
gpt-5.5, xhigh), scoped to pass 7 commits3dc1a73,94ca185, andf3c93bc.Findings new to this pass:
Origin: introduced by pass-7 commit
3dc1a73.3dc1a73made the writable store overlay persistent, but upstream keeps the Nix validity DB, profiles, and gcroots under/nix/var/nix. Paired nested boots showed a store path physically survived when only overlay upper/work persisted whilenix-store --verify-pathfailed after reboot.Repair commit:
35e49dc, which requires/nix/var/nixas the persistent Nix state volume andmicrovm.writableStoreOverlay = "/nix/var/nix".Issue: #20
Focus area 3 guest-module audit:
microvm.*: assertions/warnings; bootloader disablement; initrd kernel modules and store fs modules; blacklisted kernel modules; nix-daemon/socket defaults; pstore and GPT-generator disablement; optional machine-id and hostId; root/store/overlay/volume/sharefileSystems; closure-registrationboot.postBootCommands; optimization defaults for docs, initrd systemd, TPM2, serial UART params, swraid, networkd, wait-online, system switch, and the stratovirt overlay; optional graphics modules/packages; optional Rosetta; and optional VSOCK SSH enabling OpenSSH./nix/var/nixNix state volume. Future graphics, PCI, VSOCK, share, or non-qemu use needs explicit review.Pass-7 commit stability:
3dc1a73: not stable as written. Overlay rendering, daemon enablement, and auto-optimise guards held, but persistence needed35e49dc.94ca185: stable. The real mount unit makesRequiresMountsFora real dependency; launcher and rotation preflight still gate all writers; host mount loss is already modeled as lost active and rollback bytes.f3c93bc: stable. ExplicitrestartIfChanged = truerendersX-RestartIfChanged=trueand toplevel restart triggers for evaluated guests; the upstream default remains false.SIMPLIFY sweep considered deleting
restartIfChanged, deleting the real host mount unit, deleting the standalone guest export beforeextendModules, deletingmicrovm.registerClosure, and keeping overlay-only state separate from/nix/var/nix. Only the overlay-only state surface was cut, as part of the blocker repair.Convergence: not converged. This pass found no original-plan defect and the guest-module audit came back clean after the Nix-state repair, but it produced a review-introduced BLOCKER. Next pass should be a scoped diff review of
35e49dcplus a final convergence check; recommended model:gpt-5.6-solat xhigh.Plan review pass 9 — scoped
35e49dcreviewFindings new to this pass: 1 BLOCKER, 0 GAP, 0 SIMPLIFY, 0 QUESTION.
/nix/var/nixrepair works on its happy path. The A-to-B-to-A fixture exposed the missing lifecycle: under B, A's toplevel was physically absent whilenix path-infostill reported it valid from the persistent DB andnix-store --verify-pathrejected it. Guest GC also whiteouted each lower store's registration file because that payload is outside its own registration. Afternix-store --verifyremoved the stale A entries, rollback booted A with its current toplevel invalid and its registration source hidden. Origin: introduced by35e49dc. Repair commit:2c847c4. Tracking issue: allod/strategy#20.35e49dctherefore did not hold overall. Its/nix/var/nixpersistence boundary, overlay layout, daemon enablement, repeated-boot validity for upper-store paths, initrd failure boundary, relay coupling, and credential exclusion held; its use of upstreammicrovm.registerClosureand its persistent-DB/changing-lower assumption did not.The SIMPLIFY sweep reconsidered the writable overlay, guest GC, a separate registration volume, a generic registration manager, a separate service, and retained registration metadata. It cut the upstream
microvm.registerClosurerequirement as part of the blocker repair. The existing Nix-state volume owns small keyed registration copies, and one pre-systemd replay is the narrowest mechanism that preserves GC and rollback.Convergence: NOT CONVERGED. No new original-plan defect was found, but this pass found a review-introduced BLOCKER and
2c847c4is a structural replacement requiring independent verification. Next target: a scoped diff review of2c847c4, including fresh boot, same-toplevel restart after GC, DB reconciliation on a toplevel move, explicit pre-systemd failure, cache sabotage, and A-to-B-to-A rollback.Review pass 10: scoped diff review of
2c847c4and terminal convergence checkScope: the pass-9 repair
2c847c4(persistent registration replay) plus the convergence determination. Run byclaude-fable-5at max effort; interrupted once by a provider session limit and resumed, with no work lost.What held in
2c847c4Verified against the pinned microvm.nix
39a499aband nixpkgsb6018f87sources, a synthetic evaluation of the selected dev guest shape withregisterClosure = false, and the pass-9 built artifacts:storeDisk.passthru.regInforeally exposes the registration derivation, and the payload is packed into the erofs store disk gated only onnix.enable, so disablingregisterClosurekeeps the payload in the lower store (store-disk.nixlines 36-40, confirmed in the built pass-9 store disk's contents listing).registerClosure = false, the upstreamregInfo=kernel parameter andload-dbpostBootCommands hook disappear, a framework parameter reaches the QEMU-appendline throughmicrovm.kernelParams, the nix daemon stays enabled through the overlay, and the overlay layout is exactly upper/nix/var/nix/store, work/nix/var/nix/work, ro-store by label,/nix/var/nixneededForBoot.gcroots/booted-system(tmpfiles) andgcroots/current-system(activation) whenevernix.enable, so guest GC cannot whiteout the booted closure. Across A-to-B-to-A, the only hidden lower object is exactly what the cache covers.@shell@ @postBootCommands@as a child whose exit status is ignored in both initrd flavors (classic and systemd-initrdprepare-root), so only an active halt (not an exit status) stops systemd, the daemon, and dev sessions. Volume-before-overlay mount ordering is handled by nixpkgs overlayfsdependswiring.Findings new to this pass (both introduced by
2c847c4)nix-store --verifyrun. Contract 6a claimed verify "removes database entries whose paths are absent from the current merged store". Measured on Nix 2.31.5 (the pinnednixVersions.stableline) with a scratch store: an absent path with no surviving referrer is removed cleanly with exit 0, but an absent path that still has a valid referrer is kept, reporteddisappeared, but it still has valid referrers!, and produces exit 1 on that and every later run, never converging. After an ordinary toplevel move this referenced-absent state is the normal one, because paths built into the persistent upper overlay under the previous toplevel reference old lower-only paths (the dev guest builds against the same pinned nixpkgs its system closure comes from). The fail-closed reading of "reconciliation failure terminates the boot" would therefore brick the guest in a tight restart loop (Restart=always,panic=-1,-no-reboot) after the first host rebuild that follows real dev work; a fail-open reading silently preserves the stale validity pass 9 set out to remove. Fixed in8a21d50: reconciliation is defined as invalidating absent paths together with the valid entries whose reference closures reach them (surviving files stay on disk as unregistered upper data), a clean repeated verify is the post-condition, test 9 gains the referenced-absent fixture, and rooted-path survival promises are scoped to reference-complete paths.init=understoreOnDiskfor exactly this reason. Fixed inc784e32: the context is discarded; the erofs payload is what makes that safe.No original-plan defect was found, and nothing outside the store-persistence subsystem.
Convergence: CONVERGED, with contract 6a flagged as an empirical specification
This is the fourth consecutive pass in which review-introduced findings outnumber original-plan findings (7: 2-1, 8: 1-0, 9: 1-0, 10: 2-0), and the fourth consecutive pass to find exactly one defect in store persistence by static analysis. That pattern is the evidence: this subsystem cannot be settled by plan-text review, and each structural repair has seeded the next finding. Pass 10 therefore does not author a fourth structural rewrite or hand off another plan-text verification pass. Instead,
8a21d50corrects the one disproven factual claim to measured semantics, andd20de3brecords contract 6a as an explicitly empirical specification: the named store-lifecycle boots in acceptance test 9 (fresh volume; GC whiteout plus same-toplevel restart; toplevel move with and without a surviving upper referrer; A-to-B-to-A rollback; cache sabotage; replay termination) are the evidence that settles it, and the archetypes milestone that first selects a dev microvm does not land until they pass in the real nested boot.The full prioritized handoff list for implementation review is in the review prompt's Focus Areas: store persistence first, then the test-10 lifecycle paths, store-volume initrd failures, two-runner namespace isolation, rotation failure boundaries, and the host plaintext mount dependency.
Limits of this pass
No nested boot exercised the replay, because the replay exists only as plan text; the pass-9 spike guests embody the superseded
registerClosure = truedesign, and implementing the replay to boot it would be implementation, not review. The BLOCKER found here is a store-semantics fact measured directly against the pinned Nix version; a boot would reproduce it, not resolve it. The payload-in-erofs claim was verified from pinned source and the pass-9 built store disk rather than a freshregisterClosure = falsebuild, because packing is provably independent of that option.Commits:
8a21d50,c784e32,d20de3b,07fa46a.