nexus-host-key: rotation must re-key the installer ISO (silent provisioning break) #8

New issue

Open

opened 2026-07-15 15:20:36 +01:00 by vnprc-agent · 1 comment

vnprc-agent commented

2026-07-15 15:20:36 +01:00

User story: So that I can rotate the host identity without silently breaking future provisioning, the rotation workflow must either refresh the installer image or stop with exact rebuild steps before it reports success.

Part of the "Host-key rotation lifecycle & safety" arc.

Context

nexus-host-key rotates the host SSH/age identity at ~/.ssh/host. The custom installer ISO (nixosConfigurations.installer) bakes the host pubkey into the installer's root authorized_keys — users.users.root.openssh.authorizedKeys.keys = secrets.lib.identity.hostPublicKeys — and provision-vm-from-host SSHes into the freshly-booted installer as root with -i ~/.ssh/host (AGE_IDENTITY).

The gap

nexus-host-key stage/activate/retire re-key the consumers they know about — agenix secrets (dual-recipient then new-only), VM authorized_keys, and the Forgejo identity checks — but they never rebuild the installer ISO, nor flag that it must be rebuilt. So after a rotation the ISO in the operator's ISO directory still authorizes the pre-rotation pubkey, and every provision-vm-from-host --replace <vm> fails at the nixos-anywhere step with Permission denied (publickey) / kex_exchange_identification: Connection reset by peer — potentially long after the rotation, with no hint that the stale ISO is the cause.

Ask

Make the installer ISO a first-class consumer of the host key in the rotation lifecycle. On activate (and/or retire), nexus-host-key should either:

(a, preferred) rebuild the installer ISO from the deploy flake and place it in the ISO directory new-vm reads, or
(b) if building the ISO is out of scope for the rotation step, fail loud / print a blocking post-rotation reminder with the exact nix build <deploy>#nixosConfigurations.installer.config.system.build.isoImage command and the copy-into-the-ISO-directory step — so a rotation cannot silently leave provisioning broken.

Notes

The ISO directory and the deploy-flake ref are operator/deployment values; the script change should be parameterized on them (env/config) rather than hardcoding a path. A post-activate end-to-end check would catch this whole class of break at rotation time — tracked in the sibling blast-radius issue.

**User story:** So that I can rotate the host identity without silently breaking future provisioning, the rotation workflow must either refresh the installer image or stop with exact rebuild steps before it reports success. _Part of the "Host-key rotation lifecycle & safety" arc._ --- ## Context `nexus-host-key` rotates the host SSH/age identity at `~/.ssh/host`. The custom installer ISO (`nixosConfigurations.installer`) bakes the host pubkey into the installer's root `authorized_keys` — `users.users.root.openssh.authorizedKeys.keys = secrets.lib.identity.hostPublicKeys` — and `provision-vm-from-host` SSHes into the freshly-booted installer as `root` with `-i ~/.ssh/host` (`AGE_IDENTITY`). ## The gap `nexus-host-key` `stage`/`activate`/`retire` re-key the consumers they know about — agenix secrets (dual-recipient then new-only), VM `authorized_keys`, and the Forgejo identity checks — but they never rebuild the installer ISO, nor flag that it must be rebuilt. So after a rotation the ISO in the operator's ISO directory still authorizes the pre-rotation pubkey, and every `provision-vm-from-host --replace <vm>` fails at the `nixos-anywhere` step with `Permission denied (publickey)` / `kex_exchange_identification: Connection reset by peer` — potentially long after the rotation, with no hint that the stale ISO is the cause. ## Ask Make the installer ISO a first-class consumer of the host key in the rotation lifecycle. On `activate` (and/or `retire`), `nexus-host-key` should either: - **(a, preferred)** rebuild the installer ISO from the deploy flake and place it in the ISO directory `new-vm` reads, or - **(b)** if building the ISO is out of scope for the rotation step, fail loud / print a blocking post-rotation reminder with the exact `nix build <deploy>#nixosConfigurations.installer.config.system.build.isoImage` command and the copy-into-the-ISO-directory step — so a rotation cannot silently leave provisioning broken. ## Notes The ISO directory and the deploy-flake ref are operator/deployment values; the script change should be parameterized on them (env/config) rather than hardcoding a path. A post-`activate` end-to-end check would catch this whole class of break at rotation time — tracked in the sibling blast-radius issue.

vnprc-agent commented

2026-07-15 15:20:51 +01:00

Author

Systemic follow-up (blast-radius verification that would catch this at rotation time): allod/nexus#9