nexus: enable linger so the backup timer runs unattended #3

Merged

vnprc merged 1 commit from agent/nexus-backup-linger into master

2026-07-29 20:36:19 +01:00

vnprc-agent commented

2026-07-25 19:10:16 +01:00

Collaborator

Summary

Enable systemd user lingering for the nexus operator so home-manager user timers run unattended. Sets users.users.<user>.linger = true on the nexus host. Without linger the systemd --user manager runs only during an active login session, so the 03:00 backup-forgejo.timer never fires when nobody is logged in.

Refs vnprc/forgejo-config#5

Risk

R2 (low end). A single NixOS option on the existing operator user; reversible by a straight revert. Linger keeps the user manager alive across logout and boot, which is the intent. No secrets, no provisioning path, no persistent data touched.

Validation

nix eval of nixosConfigurations.nexus.config.users.users.<user>.linger, with this branch overridden into vnprc/deploy, returns true (was null/unmanaged on master).

Follow-up

Reaches nexus after a vnprc/deploy lock bump plus a host nixos-rebuild switch --flake ~/work/deploy#nexus (host-only). Pairs with vnprc/forgejo-config#6, which packages the backup tools onto $PATH; together they make the backup runnable from anywhere and running automatically.

## Summary Enable systemd user lingering for the nexus operator so home-manager user timers run unattended. Sets `users.users.<user>.linger = true` on the nexus host. Without linger the systemd `--user` manager runs only during an active login session, so the 03:00 `backup-forgejo.timer` never fires when nobody is logged in. Refs vnprc/forgejo-config#5 ## Risk R2 (low end). A single NixOS option on the existing operator user; reversible by a straight revert. Linger keeps the user manager alive across logout and boot, which is the intent. No secrets, no provisioning path, no persistent data touched. ## Validation `nix eval` of `nixosConfigurations.nexus.config.users.users.<user>.linger`, with this branch overridden into `vnprc/deploy`, returns `true` (was `null`/unmanaged on master). ## Follow-up Reaches nexus after a `vnprc/deploy` lock bump plus a host `nixos-rebuild switch --flake ~/work/deploy#nexus` (host-only). Pairs with vnprc/forgejo-config#6, which packages the backup tools onto `$PATH`; together they make the backup runnable from anywhere and running automatically.

vnprc-agent added 1 commit

2026-07-25 19:10:16 +01:00

nexus: enable linger for the operator user c928cb8d38

Set users.users.<user>.linger = true on nexus so the systemd --user manager
persists without an active login session, letting home-manager user timers
(the 03:00 backup-forgejo.timer) fire unattended.

Refs vnprc/forgejo-config#5

vnprc commented

2026-07-26 01:59:20 +01:00

Owner

what are the security implications of this change? should we just automate backups on the forge server instead? this would skip a download -> upload step and prevent hypervisor network issues from interrupting backups. not to mention the hypervisor might be powered down at cron time.

vnprc-agent commented

2026-07-29 20:17:30 +01:00

Author

Collaborator

Security implications of linger

Enabling linger grants no new privilege. user@<uid>.service starts at boot instead of at first login and survives logout; units still run as the unprivileged operator user, with the same file permissions, no new capabilities, no new listening port, no sudo rule. Three things do change, in descending order of how much they matter:

The backup's credentials get exercised with no human present. backup-forgejo sshes to forge-vps as root and to rsync-net, both with ~/.ssh/host (secrets/identity.nix). That key already sits on nexus unattended whether or not linger is on, and ssh never hands it to the far side, so linger changes when it is used, not what it can reach. If nexus is compromised, that key is the prize either way.
User units now persist across logout and boot. ~/.config/systemd/user/ plus linger is a textbook non-root persistence foothold. It is not a real boundary here — the operator is in wheel on their own hypervisor, so anything that can write that directory already has a shorter path — but it is the honest delta.
/run/user/<uid> exists from boot. Immaterial.

Worth more attention than linger itself: linger alone may not make the 03:00 run succeed. A systemd user service has no TTY, no ssh-agent and no askpass, so if ~/.ssh/host carries a passphrase, or forge-vps's host key is not already in the operator's known_hosts, the unattended run fails closed at the first ssh. notes/allod/forgejo-nixos-vps.md records the current state as "automated timer has linger/SSH issues" — plural — so treat this PR as fixing one of two. Verify with journalctl --user -u backup-forgejo after a real unattended fire; a manual systemctl --user start from a logged-in session can inherit session environment that the boot-time manager will not have.

Should the backup just run on the forge server

Yes, and that decision is already on record. notes/allod/forgejo-nixos-vps.md target state: "Backup runs on the VPS itself (no linger, no laptop dependency)" and "nexus drops the homeModules.forgejoBackup import (backup is server-side now)". It is Phase 4, gated behind migrating the VPS off Docker-on-Ubuntu to NixOS. All three of your arguments are in the plan's reasoning, and the network one is the strongest: the dump comes down over scp, which is not resumable, so a single home-link drop fails the whole run.

Why it is sequenced after the migration rather than done now: on today's imperative Ubuntu box, VPS-side automation is a hand-installed cron job plus a key that no repo describes, so the machine stops being a function of its config (principles 8 and 9 — one source of truth, everything rebuilds from a clone). After the migration it is a declarative timer in forgejo-config, which is also what makes that flake importable by strangers.

What the move costs, and what Phase 4 has to do to pay for it:

The pull model's security property is capability absence: the VPS holds nothing that can reach the backup store, so root on it gets the live data and nothing else (principle 4 — a boundary that depends on cooperation is not a boundary). Push inverts that; the machine being backed up gains write access to its own backups.
The pushing credential must be a new, dedicated key. Never ~/.ssh/host: per secrets/identity.nix that single key is root on forge-vps, nowhr-vps and hashpool.dev, git at the forge, and the rsync-net login. Putting it on the most internet-exposed box in the stack collapses the entire set (principle 7 — per-purpose credentials bound any compromise to that machine's own access).
rsync.net's snapshots are what make push acceptable: seven free daily snapshots, immutable, and per their docs not destroyable even by someone holding full account credentials (https://www.rsync.net/resources/howto/snapshots.html, https://www.rsync.net/products/ransomware.html). Worst case from a VPS compromise then becomes corrupted or deleted live files with a week to notice it, not loss of history. Confirm the retention actually in effect on this account before leaning on it, and consider paying for a longer window.
Encrypting on the VPS is fine — the age recipient is a public key. The private key stays at ~/.age/forgejo-backup.key plus 1Password, so the offsite host still sees only ciphertext (principle 1 — own the root).
The VPS has a 40GB disk with a documented history of filling (backup.md, Disk Management). Dumping and then encrypting on that same disk roughly doubles peak residency; stream it, or keep the delete-immediately-after-transfer discipline the current script has.
The real functional loss is the 8-week plaintext window in ~/backups/forgejo on nexus, which is what makes a fast restore possible without the age key at hand. Either keep a nexus-side pull as a second consumer, or accept that every restore starts with restore-forgejo --from-cloud.

Recommendation

Merge this as the stopgap. It is three lines plus a comment, reverts cleanly, and it makes the current design do what it already claims: the timer is OnCalendar=03:00 with Persistent = true, so today a missed run waits for you to log in, and with linger it fires shortly after the next boot instead. That covers the powered-down-at-cron case inside the existing model at roughly zero cost, and it does not compete with Phase 4 — it expires into it.

Then let Phase 4 delete it. The actionable next step there is the VPS survey and the NixOS migration, not a hand-rolled cron on the Ubuntu box; server-side backup is the reward for that migration, not a shortcut around it. Note that once Phase 4 lands, this line has no consumer left unless some other nexus user timer wants one, so revisit it then rather than keeping it on principle.

Neither design notifies anyone when the 03:00 run fails, on nexus or on the VPS. That gap outlives this PR and is probably the higher-value follow-up; say the word and I will open an issue for an OnFailure= handler.

Happy to hold this PR open until Phase 4 instead if you would rather not carry the stopgap.

## Security implications of linger Enabling linger grants no new privilege. `user@<uid>.service` starts at boot instead of at first login and survives logout; units still run as the unprivileged operator user, with the same file permissions, no new capabilities, no new listening port, no sudo rule. Three things do change, in descending order of how much they matter: 1. The backup's credentials get exercised with no human present. `backup-forgejo` sshes to `forge-vps` as root and to `rsync-net`, both with `~/.ssh/host` (`secrets/identity.nix`). That key already sits on nexus unattended whether or not linger is on, and ssh never hands it to the far side, so linger changes *when* it is used, not what it can reach. If nexus is compromised, that key is the prize either way. 2. User units now persist across logout and boot. `~/.config/systemd/user/` plus linger is a textbook non-root persistence foothold. It is not a real boundary here — the operator is in `wheel` on their own hypervisor, so anything that can write that directory already has a shorter path — but it is the honest delta. 3. `/run/user/<uid>` exists from boot. Immaterial. Worth more attention than linger itself: linger alone may not make the 03:00 run *succeed*. A systemd user service has no TTY, no ssh-agent and no askpass, so if `~/.ssh/host` carries a passphrase, or `forge-vps`'s host key is not already in the operator's `known_hosts`, the unattended run fails closed at the first ssh. `notes/allod/forgejo-nixos-vps.md` records the current state as "automated timer has linger/SSH issues" — plural — so treat this PR as fixing one of two. Verify with `journalctl --user -u backup-forgejo` after a real unattended fire; a manual `systemctl --user start` from a logged-in session can inherit session environment that the boot-time manager will not have. ## Should the backup just run on the forge server Yes, and that decision is already on record. `notes/allod/forgejo-nixos-vps.md` target state: "Backup runs on the VPS itself (no linger, no laptop dependency)" and "`nexus` drops the `homeModules.forgejoBackup` import (backup is server-side now)". It is Phase 4, gated behind migrating the VPS off Docker-on-Ubuntu to NixOS. All three of your arguments are in the plan's reasoning, and the network one is the strongest: the dump comes down over `scp`, which is not resumable, so a single home-link drop fails the whole run. Why it is sequenced after the migration rather than done now: on today's imperative Ubuntu box, VPS-side automation is a hand-installed cron job plus a key that no repo describes, so the machine stops being a function of its config (principles 8 and 9 — one source of truth, everything rebuilds from a clone). After the migration it is a declarative timer in `forgejo-config`, which is also what makes that flake importable by strangers. What the move costs, and what Phase 4 has to do to pay for it: - The pull model's security property is capability absence: the VPS holds nothing that can reach the backup store, so root on it gets the live data and nothing else (principle 4 — a boundary that depends on cooperation is not a boundary). Push inverts that; the machine being backed up gains write access to its own backups. - The pushing credential must be a new, dedicated key. Never `~/.ssh/host`: per `secrets/identity.nix` that single key is root on `forge-vps`, `nowhr-vps` and `hashpool.dev`, `git` at the forge, and the `rsync-net` login. Putting it on the most internet-exposed box in the stack collapses the entire set (principle 7 — per-purpose credentials bound any compromise to that machine's own access). - rsync.net's snapshots are what make push acceptable: seven free daily snapshots, immutable, and per their docs not destroyable even by someone holding full account credentials (<https://www.rsync.net/resources/howto/snapshots.html>, <https://www.rsync.net/products/ransomware.html>). Worst case from a VPS compromise then becomes corrupted or deleted *live* files with a week to notice it, not loss of history. Confirm the retention actually in effect on this account before leaning on it, and consider paying for a longer window. - Encrypting on the VPS is fine — the age recipient is a public key. The private key stays at `~/.age/forgejo-backup.key` plus 1Password, so the offsite host still sees only ciphertext (principle 1 — own the root). - The VPS has a 40GB disk with a documented history of filling (`backup.md`, Disk Management). Dumping and then encrypting on that same disk roughly doubles peak residency; stream it, or keep the delete-immediately-after-transfer discipline the current script has. - The real functional loss is the 8-week plaintext window in `~/backups/forgejo` on nexus, which is what makes a fast restore possible without the age key at hand. Either keep a nexus-side pull as a second consumer, or accept that every restore starts with `restore-forgejo --from-cloud`. ## Recommendation Merge this as the stopgap. It is three lines plus a comment, reverts cleanly, and it makes the current design do what it already claims: the timer is `OnCalendar=03:00` with `Persistent = true`, so today a missed run waits for you to log in, and with linger it fires shortly after the next boot instead. That covers the powered-down-at-cron case inside the existing model at roughly zero cost, and it does not compete with Phase 4 — it expires into it. Then let Phase 4 delete it. The actionable next step there is the VPS survey and the NixOS migration, not a hand-rolled cron on the Ubuntu box; server-side backup is the reward for that migration, not a shortcut around it. Note that once Phase 4 lands, this line has no consumer left unless some other nexus user timer wants one, so revisit it then rather than keeping it on principle. Neither design notifies anyone when the 03:00 run fails, on nexus or on the VPS. That gap outlives this PR and is probably the higher-value follow-up; say the word and I will open an issue for an `OnFailure=` handler. Happy to hold this PR open until Phase 4 instead if you would rather not carry the stopgap.

vnprc force-pushed agent/nexus-backup-linger from c928cb8d38 to 46d5c1ed30

2026-07-29 20:35:58 +01:00

Compare

vnprc approved these changes

2026-07-29 20:36:12 +01:00