flake-update-cascade: read branch heads over the git protocol instead of spending GitHub API calls on unchanged inputs #174

New issue

Closed

opened 2026-09-08 22:16:44 +01:00 by vnprc-agent · 1 comment

vnprc-agent commented

2026-09-08 22:16:44 +01:00

Contributor

Make a flake-update-cascade run that finds nothing to move cost no GitHub API calls, so a workspace-wide update fits the unauthenticated budget that every machine behind one public address shares.

Primary goals:

Heads over git, not REST — before asking Nix anything, read each named input's branch head with git ls-remote, once per branch per run, and compare it to the locked revision.
Nothing moved, nothing spent — a repository whose inputs all sit at their branch heads prints already up to date with no nix invocation, in every mode.
Moved inputs pinned to what was read — an input whose head differs is written with nix flake lock --override-input <path> <prefix><rev>, which fetches that revision and never resolves the branch again.
Nix keeps the rest — an input the tool cannot resolve itself (a flake.nix reference that is not a GitHub or git branch, a remote it cannot reach) is still updated by nix flake update, so the fallback is today's behavior, not a skip.
No change to what lands — the resulting flake.lock, commit message, PR title and body, and every skip and failure path stay as they are; the existing suites hold unchanged.

Current state

nix flake update <inputs> resolves every named GitHub input through https://api.github.com/repos/<owner>/<repo>/commits/<ref>, once per input per repository, even when the locked revision already is the branch head. GitHub allows sixty unauthenticated requests an hour per public address, and every machine on a LAN behind one address draws on the same sixty. A cascade over a workspace of about twenty flake repositories with nixpkgs nixpkgs-unstable home-manager forge-nixpkgs nixpkgs-tailscale spends roughly forty-six calls when nothing has moved; a dry run costs the same, so a dry run followed by the real run exhausts the budget on its own. Once it is exhausted every remaining resolution fails and Nix falls back to its cache:

warning: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/nixos-unstable': HTTP error 403

response body:

{"message":"API rate limit exceeded for <address>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}; using cached version
  already up to date

The already up to date that follows is Nix reporting its cached answer, not a comparison against the branch, so the run's report is a guess for every repository after the budget runs out.

The tool never asks whether anything moved: cmd/flake-update-cascade/main.go hands each repository's update paths straight to nixFlakeUpdate from dryRunRepo and updateAndCheck, and internal/flakelock reads inputs and locked.rev but not original, so nothing in the program knows which repository and branch a pin came from.

Measured: git ls-remote https://github.com/<owner>/<repo> refs/heads/<ref> is a git-protocol ref advertisement, not a REST request, and curl -s https://api.github.com/rate_limit shows used unchanged across it. nix flake lock --override-input <path> github:<owner>/<repo>/<rev> keeps the lock entry's original intact and costs no API call either, even for a revision not yet in the store: Nix 2.34 fetches a named GitHub revision over the git protocol, and used stays unchanged across a dry run that pins one.

Design

Classify each update path by its lock node's original. A node whose original names a rev is fixed and never moves. A github node with only owner, repo and an optional ref reads its head at https://github.com/<owner>/<repo> and pins with the prefix github:<owner>/<repo>/; a git node with only url and an optional ref reads its head at url and pins with git+<url>?rev=. Any other shape — an indirect registry reference, a tarball, a path, a dir, host or submodules attribute the override would have to carry — is left to nix flake update. A missing ref is the default branch, HEAD; a ref under refs/ is used as given; any other ref is tried as a branch and then as a tag.

Heads are read with git ls-remote --exit-code, stdin from /dev/null and GIT_TERMINAL_PROMPT=0 so a remote that wants credentials fails instead of prompting, bounded at sixty seconds, and cached per URL and ref for the run so a branch three repositories pin is read once. A failed read is reported on the repository's own lines — <path>: could not resolve <ref> at <url>; asking nix instead — and that path joins the ones Nix resolves.

After the pull, before the temporary lock copy, the plan for a repository is: paths whose head equals the lock need nothing; paths whose head differs become --override-input <path> <prefix><head> arguments to one nix flake lock; paths the tool cannot resolve go to one nix flake update, with --reference-lock-file and --output-lock-file in dry-run mode so both commands write the same temporary lock. When both lists are empty the repository prints already up to date and nothing else runs. The evaluation gate, the revision report, and every commit and PR step are unchanged.

Validation

A new suite, tests/flake/flake-update-cascade/resolve-heads.sh, with a mock git ls-remote and a mock nix flake lock added to testlib.sh: nothing moved makes one ls-remote for a branch two repositories pin and no nix call; a moved GitHub input is pinned with nix flake lock --override-input and no nix flake update runs, in direct, --pr and --dry-run modes; a moved git input pins with the git+<url>?rev= prefix; an unreachable remote prints the fallback line and runs nix flake update; a rev-pinned input is neither read nor updated; a repository with one resolvable and one opaque input runs both commands against the same temporary lock in dry-run mode; a failed nix flake lock restores the lock and ends the repository with status 1 like a failed update does. Unit tests cover the original classification and the ls-remote output parsing. Every existing suite passes unchanged: their fixtures carry no original, so they take the nix flake update path as before.

Live: a dry run over a real workspace with the built program, rate_limit read before and after, expecting used unchanged whether or not anything moved; and a direct-mode run against a disposable fixture whose input is a local git+file:// repository one commit ahead of the lock, including a transitive mid/dep path, expecting the head to be pinned, evaluated and committed.

Scope

In scope: the classification, the head cache, the two-command apply, the early already up to date, the tests above, and a flake/README.md paragraph on what a run costs.

Tracked elsewhere: processing order and pin propagation, allod/tools#171; flake-status --upstream comparing to the default branch, allod/tools#170. Authenticating Nix with access-tokens would raise the budget to five thousand an hour and stays available as a fallback on any one machine, but it is not a change to this tool.

Make a `flake-update-cascade` run that finds nothing to move cost no GitHub API calls, so a workspace-wide update fits the unauthenticated budget that every machine behind one public address shares. Primary goals: - **Heads over git, not REST** — before asking Nix anything, read each named input's branch head with `git ls-remote`, once per branch per run, and compare it to the locked revision. - **Nothing moved, nothing spent** — a repository whose inputs all sit at their branch heads prints `already up to date` with no `nix` invocation, in every mode. - **Moved inputs pinned to what was read** — an input whose head differs is written with `nix flake lock --override-input <path> <prefix><rev>`, which fetches that revision and never resolves the branch again. - **Nix keeps the rest** — an input the tool cannot resolve itself (a `flake.nix` reference that is not a GitHub or git branch, a remote it cannot reach) is still updated by `nix flake update`, so the fallback is today's behavior, not a skip. - **No change to what lands** — the resulting `flake.lock`, commit message, PR title and body, and every skip and failure path stay as they are; the existing suites hold unchanged. ### Current state `nix flake update <inputs>` resolves every named GitHub input through `https://api.github.com/repos/<owner>/<repo>/commits/<ref>`, once per input per repository, even when the locked revision already is the branch head. GitHub allows sixty unauthenticated requests an hour per public address, and every machine on a LAN behind one address draws on the same sixty. A cascade over a workspace of about twenty flake repositories with `nixpkgs nixpkgs-unstable home-manager forge-nixpkgs nixpkgs-tailscale` spends roughly forty-six calls when nothing has moved; a dry run costs the same, so a dry run followed by the real run exhausts the budget on its own. Once it is exhausted every remaining resolution fails and Nix falls back to its cache: ``` warning: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/nixos-unstable': HTTP error 403 response body: {"message":"API rate limit exceeded for <address>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}; using cached version already up to date ``` The `already up to date` that follows is Nix reporting its cached answer, not a comparison against the branch, so the run's report is a guess for every repository after the budget runs out. The tool never asks whether anything moved: `cmd/flake-update-cascade/main.go` hands each repository's update paths straight to `nixFlakeUpdate` from `dryRunRepo` and `updateAndCheck`, and `internal/flakelock` reads `inputs` and `locked.rev` but not `original`, so nothing in the program knows which repository and branch a pin came from. Measured: `git ls-remote https://github.com/<owner>/<repo> refs/heads/<ref>` is a git-protocol ref advertisement, not a REST request, and `curl -s https://api.github.com/rate_limit` shows `used` unchanged across it. `nix flake lock --override-input <path> github:<owner>/<repo>/<rev>` keeps the lock entry's `original` intact and costs no API call either, even for a revision not yet in the store: Nix 2.34 fetches a named GitHub revision over the git protocol, and `used` stays unchanged across a dry run that pins one. ### Design Classify each update path by its lock node's `original`. A node whose `original` names a `rev` is fixed and never moves. A `github` node with only `owner`, `repo` and an optional `ref` reads its head at `https://github.com/<owner>/<repo>` and pins with the prefix `github:<owner>/<repo>/`; a `git` node with only `url` and an optional `ref` reads its head at `url` and pins with `git+<url>?rev=`. Any other shape — an `indirect` registry reference, a `tarball`, a `path`, a `dir`, `host` or `submodules` attribute the override would have to carry — is left to `nix flake update`. A missing `ref` is the default branch, `HEAD`; a `ref` under `refs/` is used as given; any other `ref` is tried as a branch and then as a tag. Heads are read with `git ls-remote --exit-code`, stdin from `/dev/null` and `GIT_TERMINAL_PROMPT=0` so a remote that wants credentials fails instead of prompting, bounded at sixty seconds, and cached per URL and ref for the run so a branch three repositories pin is read once. A failed read is reported on the repository's own lines — `<path>: could not resolve <ref> at <url>; asking nix instead` — and that path joins the ones Nix resolves. After the pull, before the temporary lock copy, the plan for a repository is: paths whose head equals the lock need nothing; paths whose head differs become `--override-input <path> <prefix><head>` arguments to one `nix flake lock`; paths the tool cannot resolve go to one `nix flake update`, with `--reference-lock-file` and `--output-lock-file` in dry-run mode so both commands write the same temporary lock. When both lists are empty the repository prints `already up to date` and nothing else runs. The evaluation gate, the revision report, and every commit and PR step are unchanged. ### Validation A new suite, `tests/flake/flake-update-cascade/resolve-heads.sh`, with a mock `git ls-remote` and a mock `nix flake lock` added to `testlib.sh`: nothing moved makes one `ls-remote` for a branch two repositories pin and no `nix` call; a moved GitHub input is pinned with `nix flake lock --override-input` and no `nix flake update` runs, in direct, `--pr` and `--dry-run` modes; a moved `git` input pins with the `git+<url>?rev=` prefix; an unreachable remote prints the fallback line and runs `nix flake update`; a `rev`-pinned input is neither read nor updated; a repository with one resolvable and one opaque input runs both commands against the same temporary lock in dry-run mode; a failed `nix flake lock` restores the lock and ends the repository with status 1 like a failed update does. Unit tests cover the `original` classification and the `ls-remote` output parsing. Every existing suite passes unchanged: their fixtures carry no `original`, so they take the `nix flake update` path as before. Live: a dry run over a real workspace with the built program, `rate_limit` read before and after, expecting `used` unchanged whether or not anything moved; and a direct-mode run against a disposable fixture whose input is a local `git+file://` repository one commit ahead of the lock, including a transitive `mid/dep` path, expecting the head to be pinned, evaluated and committed. ### Scope In scope: the classification, the head cache, the two-command apply, the early `already up to date`, the tests above, and a `flake/README.md` paragraph on what a run costs. Tracked elsewhere: processing order and pin propagation, allod/tools#171; `flake-status --upstream` comparing to the default branch, allod/tools#170. Authenticating Nix with `access-tokens` would raise the budget to five thousand an hour and stays available as a fallback on any one machine, but it is not a change to this tool.

vnprc-agent referenced this issue

2026-09-08 22:36:26 +01:00

flake-update-cascade: update paths come from the pre-pull lock, so a stale checkout reports 'already up to date' without examining its current inputs #175

vnprc referenced this issue from a commit

2026-09-09 02:23:37 +01:00

flake-update-cascade: read branch heads over git before asking nix

vnprc-agent commented

2026-09-09 02:32:38 +01:00

Author

Contributor

Landed on master as 090ceab. Measured with the deployed binary on a development machine over fourteen workspace repositories and five GitHub inputs: the dry run finished in 42 s, seven repositories reported a moved nixos-unstable revision through nix flake lock, and rate_limit.used stayed at 0 before and after.

Landed on master as 090ceab. Measured with the deployed binary on a development machine over fourteen workspace repositories and five GitHub inputs: the dry run finished in 42 s, seven repositories reported a moved nixos-unstable revision through nix flake lock, and rate_limit.used stayed at 0 before and after.

vnprc-agent closed this issue

2026-09-09 02:32:38 +01:00