rebuild-host: rebuild the hypervisor from DEPLOY_FLAKE, not the working directory #59

New issue

Closed

opened 2026-09-19 15:27:25 +01:00 by vnprc-agent · 0 comments

vnprc-agent commented

2026-09-19 15:27:25 +01:00

Give the hypervisor a rebuild-host script that rebuilds the host itself from DEPLOY_FLAKE, so the operator's working directory can no longer select which flake the host is built from.

Primary goals:

One target — rebuild-host runs nixos-rebuild switch --flake "$DEPLOY_FLAKE#$(hostname)", reading the same pointer nexus.provisioning.deployFlake already renders into the operator session for the VM scripts (nix/configuration.nix:22-34).
No silent fallback — an unset DEPLOY_FLAKE is a refusal with a message naming the option, not a fallback to $HOME/work/allod/deploy. For the VM scripts that fallback is the public template's convenience; for the host it is exactly the failure this issue exists to stop.
Say what it is about to do — print the flake path, the attribute, and the evaluated system.build.toplevel out path before switching, so an operator can compare it against nix eval "$DEPLOY_FLAKE#nixosConfigurations.<host>.config.system.build.toplevel.outPath" and a rebuild that changes hundreds of packages on a lock bump that moved one input is visible before it lands.
Same conventions as rebuild-vm-from-host — set -euo pipefail, die, a NIXOS_REBUILD override so the test suite can substitute a recorder, packaged unwrapped by nix/provisioning-scripts.nix like every other host script.

Current state and specifics

The host is rebuilt by hand with sudo nixos-rebuild switch --flake .#<host> from whatever directory the shell is in. A deployment keeps two checkouts that both evaluate that attribute: its own composition root and the public template allod/deploy, whose nixosConfigurations carries the same host name so that the template stays a working example. Run from the template checkout, the command evaluates, builds, and switches cleanly, then removes the desktop, the operator's home-manager unit, the tailnet and VPN units, and the deployment's secrets, and installs the template's allod account instead. Switching stops the display manager, which kills the session running the rebuild, and the next boot lands on a text console. To the operator this is indistinguishable from a bricked host; recovery was booting the previous generation from the boot menu and deleting the bad one by hand.

This has now happened twice on one deployment. The second time the diagnosis took the closure diff between the two generations (nix store diff-closures) plus an out-of-band evaluation of both flakes to match the bad generation's store path to the template. Nothing on the host said which flake it was built from.

The VM scripts already solved this for guests: rebuild-vm-from-host and provision-vm-from-host read DEPLOY_FLAKE (scripts/rebuild-vm-from-host:6, scripts/rebuild-vm-from-host:67), and nexus.provisioning.deployFlake renders it with an absolute-path type guard so a malformed pointer fails evaluation. The host rebuild is the one path that still takes its target from the shell.

Default shape:

rebuild-host [switch|boot|test]

with switch the default and the mode passed through to nixos-rebuild unchanged. No target-machine argument: the host rebuilds itself and the attribute is $(hostname).

Validation

Test suite: with NIXOS_REBUILD pointing at a recorder, rebuild-host invokes it with switch --flake <DEPLOY_FLAKE>#<hostname>; with DEPLOY_FLAKE unset it exits non-zero without invoking it; the printed toplevel path equals what nix eval reports for the same attribute.
On a deployment host: run rebuild-host after a lock bump that moved one input, confirm the new generation's readlink /nix/var/nix/profiles/system-<N>-link equals the printed path, and confirm nix store diff-closures against the previous generation shows only that input's consequences.

Scope

In: the script, its tests, the packaging test line in nix/provisioning-scripts.nix, and the README sentence that tells an operator how to rebuild the host. Out: rolling a host or guest back to its previous generation, which is allod/nexus#39; the VM rebuild path, which already reads DEPLOY_FLAKE; and any guard inside nixos-rebuild itself.

Give the hypervisor a `rebuild-host` script that rebuilds the host itself from `DEPLOY_FLAKE`, so the operator's working directory can no longer select which flake the host is built from. Primary goals: - **One target** — `rebuild-host` runs `nixos-rebuild switch --flake "$DEPLOY_FLAKE#$(hostname)"`, reading the same pointer `nexus.provisioning.deployFlake` already renders into the operator session for the VM scripts (`nix/configuration.nix:22-34`). - **No silent fallback** — an unset `DEPLOY_FLAKE` is a refusal with a message naming the option, not a fallback to `$HOME/work/allod/deploy`. For the VM scripts that fallback is the public template's convenience; for the host it is exactly the failure this issue exists to stop. - **Say what it is about to do** — print the flake path, the attribute, and the evaluated `system.build.toplevel` out path before switching, so an operator can compare it against `nix eval "$DEPLOY_FLAKE#nixosConfigurations.<host>.config.system.build.toplevel.outPath"` and a rebuild that changes hundreds of packages on a lock bump that moved one input is visible before it lands. - **Same conventions as `rebuild-vm-from-host`** — `set -euo pipefail`, `die`, a `NIXOS_REBUILD` override so the test suite can substitute a recorder, packaged unwrapped by `nix/provisioning-scripts.nix` like every other host script. ### Current state and specifics The host is rebuilt by hand with `sudo nixos-rebuild switch --flake .#<host>` from whatever directory the shell is in. A deployment keeps two checkouts that both evaluate that attribute: its own composition root and the public template `allod/deploy`, whose `nixosConfigurations` carries the same host name so that the template stays a working example. Run from the template checkout, the command evaluates, builds, and switches cleanly, then removes the desktop, the operator's home-manager unit, the tailnet and VPN units, and the deployment's secrets, and installs the template's `allod` account instead. Switching stops the display manager, which kills the session running the rebuild, and the next boot lands on a text console. To the operator this is indistinguishable from a bricked host; recovery was booting the previous generation from the boot menu and deleting the bad one by hand. This has now happened twice on one deployment. The second time the diagnosis took the closure diff between the two generations (`nix store diff-closures`) plus an out-of-band evaluation of both flakes to match the bad generation's store path to the template. Nothing on the host said which flake it was built from. The VM scripts already solved this for guests: `rebuild-vm-from-host` and `provision-vm-from-host` read `DEPLOY_FLAKE` (`scripts/rebuild-vm-from-host:6`, `scripts/rebuild-vm-from-host:67`), and `nexus.provisioning.deployFlake` renders it with an absolute-path type guard so a malformed pointer fails evaluation. The host rebuild is the one path that still takes its target from the shell. Default shape: ``` rebuild-host [switch|boot|test] ``` with `switch` the default and the mode passed through to `nixos-rebuild` unchanged. No target-machine argument: the host rebuilds itself and the attribute is `$(hostname)`. ### Validation - Test suite: with `NIXOS_REBUILD` pointing at a recorder, `rebuild-host` invokes it with `switch --flake <DEPLOY_FLAKE>#<hostname>`; with `DEPLOY_FLAKE` unset it exits non-zero without invoking it; the printed toplevel path equals what `nix eval` reports for the same attribute. - On a deployment host: run `rebuild-host` after a lock bump that moved one input, confirm the new generation's `readlink /nix/var/nix/profiles/system-<N>-link` equals the printed path, and confirm `nix store diff-closures` against the previous generation shows only that input's consequences. ### Scope In: the script, its tests, the packaging test line in `nix/provisioning-scripts.nix`, and the README sentence that tells an operator how to rebuild the host. Out: rolling a host or guest back to its previous generation, which is allod/nexus#39; the VM rebuild path, which already reads `DEPLOY_FLAKE`; and any guard inside `nixos-rebuild` itself.