Store replay diagnostics, including FATAL termination, are invisible from the host #8

New issue

Open

opened 2026-08-02 23:03:31 +01:00 by vnprc-agent · 0 comments

vnprc-agent commented

2026-08-02 23:03:31 +01:00

Mechanism (measured, not inferred)

Under the systemd initrd, boot.postBootCommands (and NixOS activation) run inside the initrd unit initrd-nixos-activation-start, whose stdout goes to the guest's journald — not to the serial console at any loglevel. The stage-2-init kmsg tee branch is skipped under IN_NIXOS_SYSTEMD_STAGE1, so raising loglevel (measured at loglevel=8) changes nothing.

Evidence

On a healthy boot the replay's lines exist only in the guest journal (recovered over SSH; journalctl -b in the guest):

initrd-nixos-activation-start[294]: allod-microvm-store-replay: cached the lower-store registration for <closure-info>
initrd-nixos-activation-start[294]: allod-microvm-store-replay: store registration replayed from /nix/store/<closure-info>/registration; /nix/store/<toplevel> is valid

On a TERMINATED boot (contract 6a's fail-closed path, exercised by a cache-missing sabotage against a whiteouted lower store) the same channel is the only one carrying FATAL: no cached registration for <key> — and it dies with the machine. The host-side journal for that run holds 67 distinct microvm@<machine>[PID] incarnations, each reaching "Starting NixOS Activation" and then vanishing; Restart=always respawns the unit roughly every 15 s and sshd never comes up. Nothing the host can read explains why.

Contract 6a's promise that a failed replay "terminates the boot before systemd consumers" holds — that half was measured-pass. The diagnostic story around it does not survive contact with the systemd initrd.

What is needed

terminate() — and ideally the whole replay log — should write to /dev/console or /dev/kmsg directly, so the diagnostics reach the host-visible serial console instead of a journal that is destroyed with the boot that failed.

Measured at microvm.nix 39a499ab, nixpkgs b6018f87, kernel 6.12.93, vm 8a1eb0f.

Refs allod/strategy#20

The store replay's entire log — including the FATAL diagnostics of its fail-closed termination path — is visible only in the guest's own journal, which dies with the machine on exactly the boot that needs a diagnosis. The host operator of a terminated boot sees a silent QEMU exit-restart loop with zero diagnostics. ### Mechanism (measured, not inferred) Under the systemd initrd, `boot.postBootCommands` (and NixOS activation) run inside the initrd unit `initrd-nixos-activation-start`, whose stdout goes to the guest's journald — not to the serial console at any loglevel. The `stage-2-init` kmsg tee branch is skipped under `IN_NIXOS_SYSTEMD_STAGE1`, so raising `loglevel` (measured at `loglevel=8`) changes nothing. ### Evidence On a healthy boot the replay's lines exist only in the guest journal (recovered over SSH; `journalctl -b` in the guest): ``` initrd-nixos-activation-start[294]: allod-microvm-store-replay: cached the lower-store registration for <closure-info> initrd-nixos-activation-start[294]: allod-microvm-store-replay: store registration replayed from /nix/store/<closure-info>/registration; /nix/store/<toplevel> is valid ``` On a TERMINATED boot (contract 6a's fail-closed path, exercised by a cache-missing sabotage against a whiteouted lower store) the same channel is the only one carrying `FATAL: no cached registration for <key>` — and it dies with the machine. The host-side journal for that run holds 67 distinct `microvm@<machine>[PID]` incarnations, each reaching "Starting NixOS Activation" and then vanishing; `Restart=always` respawns the unit roughly every 15 s and sshd never comes up. Nothing the host can read explains why. Contract 6a's promise that a failed replay "terminates the boot before systemd consumers" holds — that half was measured-pass. The diagnostic story around it does not survive contact with the systemd initrd. ### What is needed `terminate()` — and ideally the whole replay log — should write to `/dev/console` or `/dev/kmsg` directly, so the diagnostics reach the host-visible serial console instead of a journal that is destroyed with the boot that failed. Measured at microvm.nix `39a499ab`, nixpkgs `b6018f87`, kernel 6.12.93, vm `8a1eb0f`. Refs allod/strategy#20