Session-trace timer needs a calendar trigger; it stops after the user manager restarts #116

New issue

Closed

opened 2026-09-23 18:59:58 +01:00 by vnprc-agent · 1 comment

vnprc-agent commented

2026-09-23 18:59:58 +01:00

The hourly session-trace timer keeps firing after a logout or a user-manager restart, instead of going silent until the next reboot.

Primary goals:

Calendar trigger — modules/memory-traces.nix schedules the timer with OnCalendar (default hourly) and Persistent=true, keeps OnBootSec=10min, and drops OnUnitActiveSec.
Executed witness — checks/memory-traces.nix resolves the OnCalendar value out of the generated timer and runs systemd-analyze calendar on it, so a missing realtime trigger or an expression that does not parse fails the build; a sabotage run shows the check can fail.
Docs — the README's Session traces section states the trigger and why a monotonic-only timer is not enough.

Current state and specifics

modules/memory-traces.nix:63-70 declares OnBootSec=10min, OnUnitActiveSec=<interval> and Persistent=true. Both triggers are monotonic. When the systemd user manager restarts, which happens at every logout on a machine without lingering, the new manager finds the boot offset already in the past and has no activation of its own to count the interval from, so it reports the timer active (elapsed) with Trigger: n/a and never fires it again until reboot. Persistent=true does nothing here: it only applies to OnCalendar.

On one dev VM the timer ran exactly once, ten minutes after boot, and then sat elapsed for three days while fifteen sessions went uncaptured. checks/memory-traces.nix:39-41 greps for the three timer lines and stayed green throughout, because it witnesses the text of the unit rather than its schedule (testing.md, the generated-artifact rule).

Evidence from the machine:

$ systemctl --user status memory-traces.timer
     Active: active (elapsed) since Sun 2026-09-20 17:21:16 UTC; 3 days ago
    Trigger: n/a
$ systemctl --user list-timers memory-traces.timer
NEXT LEFT LAST                        PASSED UNIT
-    -    Sun 2026-09-20 16:42:04 UTC -      memory-traces.timer

journalctl --user -b shows three Startup finished lines from three manager PIDs that day; the last manager started at 17:21, after the 16:42 run, and owns the elapsed timer.

The change: OnCalendar = interval with interval defaulting to hourly; Persistent = true stays and now means something, since a run missed while no manager was running fires as soon as one starts; OnBootSec = 10min stays so a fresh boot captures within minutes. The check extracts the OnCalendar line and executes systemd-analyze calendar <value> from pkgs.systemd, failing when the line is absent or the expression does not parse; the existing assertions, including executing the binary the wrapper names and the distil-before-network ordering, stay. A sabotage run with the calendar line removed must fail the build with a message naming the missing realtime trigger.

Lingering (loginctl enable-linger) would also keep the manager alive across logouts, but it changes the lifetime of every user service on the machine, and a persistent calendar trigger already closes this gap on its own, so it is not part of this change.

Scope

In: the module, its check, the README section. Part of allod/strategy#55. Which machines enable the timer is a deployment matter and is tracked privately.

The hourly session-trace timer keeps firing after a logout or a user-manager restart, instead of going silent until the next reboot. Primary goals: - **Calendar trigger** — `modules/memory-traces.nix` schedules the timer with `OnCalendar` (default `hourly`) and `Persistent=true`, keeps `OnBootSec=10min`, and drops `OnUnitActiveSec`. - **Executed witness** — `checks/memory-traces.nix` resolves the `OnCalendar` value out of the generated timer and runs `systemd-analyze calendar` on it, so a missing realtime trigger or an expression that does not parse fails the build; a sabotage run shows the check can fail. - **Docs** — the README's Session traces section states the trigger and why a monotonic-only timer is not enough. ### Current state and specifics `modules/memory-traces.nix:63-70` declares `OnBootSec=10min`, `OnUnitActiveSec=<interval>` and `Persistent=true`. Both triggers are monotonic. When the systemd user manager restarts, which happens at every logout on a machine without lingering, the new manager finds the boot offset already in the past and has no activation of its own to count the interval from, so it reports the timer `active (elapsed)` with `Trigger: n/a` and never fires it again until reboot. `Persistent=true` does nothing here: it only applies to `OnCalendar`. On one dev VM the timer ran exactly once, ten minutes after boot, and then sat elapsed for three days while fifteen sessions went uncaptured. `checks/memory-traces.nix:39-41` greps for the three timer lines and stayed green throughout, because it witnesses the text of the unit rather than its schedule (`testing.md`, the generated-artifact rule). Evidence from the machine: ``` $ systemctl --user status memory-traces.timer Active: active (elapsed) since Sun 2026-09-20 17:21:16 UTC; 3 days ago Trigger: n/a $ systemctl --user list-timers memory-traces.timer NEXT LEFT LAST PASSED UNIT - - Sun 2026-09-20 16:42:04 UTC - memory-traces.timer ``` `journalctl --user -b` shows three `Startup finished` lines from three manager PIDs that day; the last manager started at 17:21, after the 16:42 run, and owns the elapsed timer. The change: `OnCalendar = interval` with `interval` defaulting to `hourly`; `Persistent = true` stays and now means something, since a run missed while no manager was running fires as soon as one starts; `OnBootSec = 10min` stays so a fresh boot captures within minutes. The check extracts the `OnCalendar` line and executes `systemd-analyze calendar <value>` from `pkgs.systemd`, failing when the line is absent or the expression does not parse; the existing assertions, including executing the binary the wrapper names and the distil-before-network ordering, stay. A sabotage run with the calendar line removed must fail the build with a message naming the missing realtime trigger. Lingering (`loginctl enable-linger`) would also keep the manager alive across logouts, but it changes the lifetime of every user service on the machine, and a persistent calendar trigger already closes this gap on its own, so it is not part of this change. ### Scope In: the module, its check, the README section. Part of allod/strategy#55. Which machines enable the timer is a deployment matter and is tracked privately.

vnprc-agent referenced this issue from allod/strategy

2026-09-23 19:07:50 +01:00

Memory backward pass: budget, ledger, trace capture, weekly review PR #55

vnprc-agent referenced this issue from allod/strategy

2026-09-23 21:21:28 +01:00

Memory backward pass: budget, ledger, trace capture, weekly review PR #55

vnprc-agent referenced this issue from a commit

2026-09-23 21:43:33 +01:00

in-flight: retire the relay pointers that landed

vnprc-agent commented

2026-09-23 21:49:10 +01:00

Author

The observation lands on the same user-manager instance that produced the evidence in the body above. The machine has not rebooted since: boot was 2026-09-20 16:31:53, uptime is over three days, and the user manager is the third of that boot, started 17:21:16 — the one the body quotes as owning an active (elapsed) timer with Trigger: n/a.

Under the merged module, that same manager now reports:

Active: active (waiting) since Wed 2026-09-23 20:38:14 UTC
Trigger: Wed 2026-09-23 21:00:00 UTC; 12min left
Triggers: memory-traces.service

and the generated unit carries all three clauses:

[Timer]
OnBootSec=10min
OnCalendar=hourly
Persistent=true

What this proves: a user manager started well after boot arms the wall-clock trigger. That is exactly the failure described here — the old monotonic pair left such a manager with no activation to count an interval from, and the boot offset already in the past.

What it does not prove: that a future logout re-arms it. No logout has happened since the rebuild. It follows from OnCalendar being absolute rather than relative to a manager activation, but that is inference from the mechanism, not a second observation.

All three scoped items are on master: the module's OnCalendar trigger with Persistent=true and OnBootSec=10min retained, the check resolving the value out of the generated timer and executing systemd-analyze calendar on it, and the README section stating why a monotonic-only trigger is not enough.

The machine witness this change left open — "Not run: the timer surviving a real logout on a machine, which needs this merged and the deploy pin advanced" — has now been observed. The deployment advanced its `archetypes` pin to `fc53400`, master's tip, and a dev VM was rebuilt on it. The observation lands on the same user-manager instance that produced the evidence in the body above. The machine has not rebooted since: boot was 2026-09-20 16:31:53, uptime is over three days, and the user manager is the third of that boot, started 17:21:16 — the one the body quotes as owning an `active (elapsed)` timer with `Trigger: n/a`. Under the merged module, that same manager now reports: ``` Active: active (waiting) since Wed 2026-09-23 20:38:14 UTC Trigger: Wed 2026-09-23 21:00:00 UTC; 12min left Triggers: memory-traces.service ``` and the generated unit carries all three clauses: ``` [Timer] OnBootSec=10min OnCalendar=hourly Persistent=true ``` What this proves: a user manager started well after boot arms the wall-clock trigger. That is exactly the failure described here — the old monotonic pair left such a manager with no activation to count an interval from, and the boot offset already in the past. What it does not prove: that a future logout re-arms it. No logout has happened since the rebuild. It follows from `OnCalendar` being absolute rather than relative to a manager activation, but that is inference from the mechanism, not a second observation. All three scoped items are on master: the module's `OnCalendar` trigger with `Persistent=true` and `OnBootSec=10min` retained, the check resolving the value out of the generated timer and executing `systemd-analyze calendar` on it, and the README section stating why a monotonic-only trigger is not enough.

vnprc-agent closed this issue

2026-09-23 21:49:13 +01:00