microvm.mem = 2048 deterministically corrupts the guest initrd at the current pins #7

New issue

Open

opened 2026-08-02 23:03:26 +01:00 by vnprc-agent · 0 comments

vnprc-agent commented

2026-08-02 23:03:26 +01:00

A microvm guest built by this framework cannot boot with microvm.mem = 2048 at the current pins: the kernel's initramfs decompression corrupts the initrd's tail at exactly that memory size, the guest strands in stage 1, and the host has no signal that anything is wrong. The value is an entirely plausible production sizing, so this is a landmine for any deployment that picks it.

Symptom

First executed boot of an archetypes-built dev microvm guest (public dev machine definition with its inventory runtime flipped to microvm), wired into the real host module and launcher. The guest console shows:

[    0.966074] Initramfs unpacking failed: ZSTD-compressed data is corrupt
[DEPEND] Dependency failed for /sysroot/nix/store

Stage 1 then hangs: the partially extracted initramfs lacks the erofs/virtio modules, so the store never mounts and the boot never reaches multi-user.

Evidence

Isolated by rerunning the identical generated runner directly under single-level KVM, varying only -m:

initrd artifact verified intact: sha256 of the built initrd equals the build host's copy (22,224,944 bytes), so this is not a corrupt artifact;
512M OK, 1024M OK, 2048M CORRUPT, 2560M OK, 3072M OK, 4096M OK;
identical under L1 KVM and nested L3 KVM: deterministic and accelerator-independent.

Components at the failing measurement: QEMU 10.1.5 (qemu-host-cpu-only-for-vm-tests), machine type microvm with its SeaBIOS-variant loader, a 22 MiB zstd-compressed initrd, kernel 6.12.93, microvm.nix 39a499ab, nixpkgs b6018f87, vm 8a1eb0f, archetypes 8aa0fce. At these pins a guest memory size of exactly 2048 MiB places the initrd where kernel decompression overwrites its tail.

The host cannot detect it

From the host everything looks healthy: microvm@<machine>.service is active and the QMP socket exists, because QEMU itself is fine — only the guest is stranded. The host module's existing liveness signal (wait-for-QMP, as used by the isolation check's boot wait) is provably insufficient as a health signal for this failure class. A deploy-time "guest reached multi-user" health check is the missing piece; until one exists, deployments can only avoid the poison value.

Refs allod/strategy#20

A microvm guest built by this framework cannot boot with `microvm.mem = 2048` at the current pins: the kernel's initramfs decompression corrupts the initrd's tail at exactly that memory size, the guest strands in stage 1, and the host has no signal that anything is wrong. The value is an entirely plausible production sizing, so this is a landmine for any deployment that picks it. ### Symptom First executed boot of an archetypes-built dev microvm guest (public dev machine definition with its inventory runtime flipped to `microvm`), wired into the real host module and launcher. The guest console shows: ``` [ 0.966074] Initramfs unpacking failed: ZSTD-compressed data is corrupt [DEPEND] Dependency failed for /sysroot/nix/store ``` Stage 1 then hangs: the partially extracted initramfs lacks the erofs/virtio modules, so the store never mounts and the boot never reaches multi-user. ### Evidence Isolated by rerunning the identical generated runner directly under single-level KVM, varying only `-m`: - initrd artifact verified intact: sha256 of the built initrd equals the build host's copy (22,224,944 bytes), so this is not a corrupt artifact; - 512M OK, 1024M OK, **2048M CORRUPT**, 2560M OK, 3072M OK, 4096M OK; - identical under L1 KVM and nested L3 KVM: deterministic and accelerator-independent. Components at the failing measurement: QEMU 10.1.5 (`qemu-host-cpu-only-for-vm-tests`), machine type `microvm` with its SeaBIOS-variant loader, a 22 MiB zstd-compressed initrd, kernel 6.12.93, microvm.nix `39a499ab`, nixpkgs `b6018f87`, vm `8a1eb0f`, archetypes `8aa0fce`. At these pins a guest memory size of exactly 2048 MiB places the initrd where kernel decompression overwrites its tail. ### The host cannot detect it From the host everything looks healthy: `microvm@<machine>.service` is `active` and the QMP socket exists, because QEMU itself is fine — only the guest is stranded. The host module's existing liveness signal (wait-for-QMP, as used by the isolation check's boot wait) is provably insufficient as a health signal for this failure class. A deploy-time "guest reached multi-user" health check is the missing piece; until one exists, deployments can only avoid the poison value. Refs allod/strategy#20