Make VM SSH host-key rotation runtime-aware for microVMs #21
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
allod/nexus#21
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Keep stable VM SSH host-key rotation working when a guest receives its private host key per boot instead of storing it on the guest filesystem.
Primary goals:
machine-host-keys.json, staged/active public keys, machine-facts coherence, and strict known-host pinning as the authoritative model.Current state and specifics
vm-ssh-host-key activatecurrently decrypts the staged key in host tmpfs, copies it to/etc/ssh/<target>in the guest, reloads sshd over SSH, and verifies the new presented key. Its stage and retire guidance also assumes rebuilding the target deploys secrets encrypted to the staged VM host key. Those behaviors match the libvirt/agenix runtime, but they cannot activate a microVM whose stable host key is delivered from a host ramfs on every start and whose guest has no age identity.The stable public-key model does not need to change. The activation mechanism does: for a microVM target, the tool must refresh the host-side plaintext credential from the staged encrypted backup, restart the target through the declared microVM host unit, verify the presented key against staged state, and retain the same fail-closed anti-TOFU checks. Generated output and tests must not claim the microVM receives age-recipient secrets.
Scope
Part of allod/strategy#20.
In scope: runtime-aware behavior in the public host-key rotation tool and fixtures, using the runtime fact and host credential-delivery interface supplied by that arc.
Out of scope: changing SSH host-key identity to ephemeral keys, changing the registry schema beyond the settled runtime fact, TPM sealing, private machine values, and implementing the deployment-specific cutover.
Integration review found a migration-window interaction the runtime-aware rework should pin down.
vm-ssh-host-keyupdates known-host pins by rewriting only the lines for the single address it resolves from the generated inventory view, andstagereplaces the machine's one encrypted backup with the staged key. During a runtime migration a deployment can legitimately hold pins for two endpoints of the same machine at once — the selected runtime's address plus the old runtime's address kept for rollback. Every stage/activate/retire then touches only the selected endpoint. Rotated before cutover, per-boot delivery hands the staged key to a guest whose validation endpoint is still pinned to the pre-rotation key, so strict SSH raises a host-key alarm exactly where the anti-TOFU property is being demonstrated. Rotated after cutover, the rollback endpoint keeps trusting the retired key while the stopped old-runtime guest still holds that key on disk, so the rollback path silently diverges from the registry's active/staged state.Suggest the runtime-aware behavior state this boundary explicitly and prove it in fixtures: either the tool refuses routine rotation while a machine's runtime state marks a migration in progress (if the runtime fact can express that), or the contract documents that only the selected endpoint's pin is managed and any second rollback pin is the deployment's to freeze or reconcile — with a fixture showing whichever behavior was chosen. Without one of the two, fixtures for "retain the same fail-closed anti-TOFU checks" can pass while a mid-migration rotation breaks strict SSH on one of the machine's two endpoints.
Integration review found a second thing the runtime-aware activate branch has to pin down: which copy of the encrypted backup counts as "the staged encrypted backup" it refreshes from.
For a microVM whose stable host key is delivered per boot, the host-side plaintext is produced by a pre-start hook that decrypts ciphertext reached through the deployment's pinned framework/secrets inputs, and that hook necessarily reruns on every start of the VM unit — including every automatic restart, since the upstream host unit is
Restart=always.stage, meanwhile, writes the newly generated key over the machine's single encrypted backup in the secrets working checkout:vm-ssh-host-key's stage path encrypts to the host recipient and writes the one<target>-ssh.agein place, andforge-ssh-key's stage path does the same to the secret named by the Forge key registry. Neither changes what a deployed host will decrypt at the next start.That produces a rotation that appears to succeed and then silently reverts. If
activaterefreshes the host-side plaintext directly from the working-checkout backup and restarts the unit, the guest presents the staged key — until the next automatic restart re-runs the pre-start hook against the still-old pinned ciphertext and the guest goes back to the pre-rotation key. By thenretirehas promoted staged to active and rewritten the pin to the new key only, so the regression does not show up at rotation time. It shows up later as a strict-SSH host-key alarm on the first VM crash or restart, which is exactly the anti-TOFU property these fixtures are meant to prove.Two ways to state the contract, either is fine as long as one is chosen and shown in a fixture:
retirerefuses until it has verified the key survives a restart.The second option is weaker but at least honest; the first is what the delivery model already implies.
Concretely, the fixture that catches this is a restart cycle: after activating on a microVM target, restart the unit a second time and require the guest to still present the staged key before allowing
retire. A single post-activate verification cannot distinguish a durable rotation from an out-of-band write that the next start undoes.This is separate from the two-endpoint migration window in my earlier comment, and applies to steady-state microVM rotation as well.