Bound the FTP connections allod site deploy opens

allod-agent commented

2026-09-10 19:08:24 +01:00

Member

Cap the simultaneous FTP connections allod site deploy opens so a sync to shared hosting finishes instead of tripping the host's per-client connection limit partway through (the failure in allod/tools#184).

Primary goals:

Bounded connections — the rclone sync that deploy runs passes --transfers, --checkers, and --ftp-concurrency, with the concurrency one more than the sum of the other two, as rclone's FTP documentation requires for a sync to avoid deadlock.
Small, fixed defaults — two transfers, two checkers, five connections. Shared hosts commonly allow a handful of connections per client; the value that matters is that the total is bounded, not that it is tuned.
Pinned, not documented — a test asserts the exact flags in the sync argv, so a later change that drops one fails a test rather than a deploy.
Dry run identical — --dry-run passes the same flags, so its listing exercises the same connection budget.

Current state and specifics

siteDeploy in cmd/allod/site.go builds syncArgs as sync <store path> <docroot>, the profile's --filter-from, --backup-dir, and --verbose, then --dry-run when asked. rclone's defaults fill the rest: four transfers, eight checkers, and the FTP backend's concurrency of 0, unlimited. Directory listing during the check phase is what opens connections in parallel; with the store-path timestamps honoured by the host, the transfer set of a typical deploy is small and the listings still exceed the limit (allod/tools#184).

The flags belong in the deploy's own argv, next to --backup-dir, rather than in the rclone config stanza: the stanza is the credential, placed by agenix and rotated by rotate-token, and its parser accepts exactly the five keys allod site config writes.

Scope

In: the three flags in siteDeploy's sync argv, the test that pins them, and one sentence in the deploy detail prose saying the sync is connection-bounded.

Out: making the numbers a hosting-profile field (trigger: a host whose limit the fixed values do not fit), and check, which opens one connection.

Cap the simultaneous FTP connections `allod site deploy` opens so a sync to shared hosting finishes instead of tripping the host's per-client connection limit partway through (the failure in allod/tools#184). Primary goals: - **Bounded connections** — the `rclone sync` that deploy runs passes `--transfers`, `--checkers`, and `--ftp-concurrency`, with the concurrency one more than the sum of the other two, as rclone's FTP documentation requires for a sync to avoid deadlock. - **Small, fixed defaults** — two transfers, two checkers, five connections. Shared hosts commonly allow a handful of connections per client; the value that matters is that the total is bounded, not that it is tuned. - **Pinned, not documented** — a test asserts the exact flags in the sync argv, so a later change that drops one fails a test rather than a deploy. - **Dry run identical** — `--dry-run` passes the same flags, so its listing exercises the same connection budget. ### Current state and specifics `siteDeploy` in `cmd/allod/site.go` builds `syncArgs` as `sync <store path> <docroot>`, the profile's `--filter-from`, `--backup-dir`, and `--verbose`, then `--dry-run` when asked. rclone's defaults fill the rest: four transfers, eight checkers, and the FTP backend's `concurrency` of 0, unlimited. Directory listing during the check phase is what opens connections in parallel; with the store-path timestamps honoured by the host, the transfer set of a typical deploy is small and the listings still exceed the limit (allod/tools#184). The flags belong in the deploy's own argv, next to `--backup-dir`, rather than in the rclone config stanza: the stanza is the credential, placed by agenix and rotated by `rotate-token`, and its parser accepts exactly the five keys `allod site config` writes. ### Scope In: the three flags in `siteDeploy`'s sync argv, the test that pins them, and one sentence in the deploy detail prose saying the sync is connection-bounded. Out: making the numbers a hosting-profile field (trigger: a host whose limit the fixed values do not fit), and `check`, which opens one connection.

allod-agent referenced this issue from a commit

2026-09-10 19:13:08 +01:00

Cap allod site deploy's rclone sync to a small, bounded number of simultaneous FTP connections: --transfers 2, --checkers 2, --ftp-concurrency 5, following rclone's documented rule that concurrency be one more than transfers plus checkers so a sync does not deadlock. A shared host's per-client connection limit tripped mid-deploy before this, per allod/tools#184.

allod-agent referenced this issue from a commit

2026-09-10 19:20:50 +01:00

Correct the FTP connection cap for allod site deploy: the bound is per rclone filesystem instance, and --backup-dir on the same remote is a second instance with its own connection pool, so a deploy's worst case is two instances, not one. Switch to rclone's cheaper --check-first budget, max(transfers, checkers) + 1 = 3 instead of transfers + checkers + 1 = 5, which also runs the listing phase - the phase that tripped the limit - to completion before any file moves. Worst case across both instances is now 6 connections instead of 10. The constants' comment quotes rclone's docs and states the per-instance rule; the deploy help prose says the same.

allod-agent referenced this issue from a pull request that will close it,

2026-09-10 19:23:31 +01:00

Bound the FTP connections allod site deploy opens #187

allod-agent referenced this issue from a commit

2026-09-10 19:23:32 +01:00

site deploy: note what --check-first costs