rotate-token test suite enforces only the last assertion of each test #46

New issue

Closed

opened 2026-09-10 15:57:53 +01:00 by allod-agent · 0 comments

allod-agent commented

2026-09-10 15:57:53 +01:00

Member

tests/rotate-token.sh enforces only the last command of each test function; every earlier [[ ... ]] assertion is a no-op, so most of the suite cannot fail on the properties it names.

Evidence

run_test (tests/rotate-token.sh:11-21) invokes each test as if "$@"; then. Bash ignores set -e inside an if condition, and that extends into any function called there, so a failed [[ ... ]] mid-function continues to the next line and the function's status is whatever its last command returns.

Minimal reproduction of the semantics:

$ bash -c 'set -euo pipefail; f(){ [[ 1 == 2 ]]; echo reached; true; }; if f; then echo PASS-vacuous; fi'
reached
PASS-vacuous

Running the suite on master with an enforcing harness (set +e; ( set -e; "$@" ); status=$?; set -e in place of if "$@") stops at the sixth test:

PASS: unknown option is rejected
PASS: help flag prints group options
PASS: extra positional is rejected
PASS: group dry-run lists shared secrets, targets, and commands
FAIL: in-place continuity dry-run orders refresh before network steps

The suite exits on the first failure, so the total number of tests whose non-final assertions do not hold on master is unknown; at least one does not. nix build .#checks.x86_64-linux.provisioning-contract runs this suite and reports success today.

Affected: every test in tests/rotate-token.sh with more than one assertion; flake.nix provisioning-contract. Observed on the master at 0f0fa2b. Found while adding tests for allod/nexus#42; those new tests carry || return 1 on each assertion so they hold under the current harness, but the rest of the suite is unchanged.

Labels are unavailable to the filing token; this is a bug report.

`tests/rotate-token.sh` enforces only the last command of each test function; every earlier `[[ ... ]]` assertion is a no-op, so most of the suite cannot fail on the properties it names. ### Evidence `run_test` (`tests/rotate-token.sh:11-21`) invokes each test as `if "$@"; then`. Bash ignores `set -e` inside an `if` condition, and that extends into any function called there, so a failed `[[ ... ]]` mid-function continues to the next line and the function's status is whatever its last command returns. Minimal reproduction of the semantics: ``` $ bash -c 'set -euo pipefail; f(){ [[ 1 == 2 ]]; echo reached; true; }; if f; then echo PASS-vacuous; fi' reached PASS-vacuous ``` Running the suite on `master` with an enforcing harness (`set +e; ( set -e; "$@" ); status=$?; set -e` in place of `if "$@"`) stops at the sixth test: ``` PASS: unknown option is rejected PASS: help flag prints group options PASS: extra positional is rejected PASS: group dry-run lists shared secrets, targets, and commands FAIL: in-place continuity dry-run orders refresh before network steps ``` The suite exits on the first failure, so the total number of tests whose non-final assertions do not hold on `master` is unknown; at least one does not. `nix build .#checks.x86_64-linux.provisioning-contract` runs this suite and reports success today. Affected: every test in `tests/rotate-token.sh` with more than one assertion; `flake.nix` `provisioning-contract`. Observed on the `master` at `0f0fa2b`. Found while adding tests for allod/nexus#42; those new tests carry `|| return 1` on each assertion so they hold under the current harness, but the rest of the suite is unchanged. Labels are unavailable to the filing token; this is a `bug` report.

allod-agent referenced this issue

2026-09-10 16:16:57 +01:00

Teach rotate-token the rclone remote stanza format #47

allod-agent referenced this issue

2026-09-10 19:10:23 +01:00

Make rotate-token's test runner enforce every assertion #48

allod-agent referenced this issue from a commit

2026-09-10 19:24:35 +01:00

tests/rotate-token: make run_test enforce every assertion in a test body

allod-agent referenced this issue from a pull request that will close it,

2026-09-10 19:28:42 +01:00

Make rotate-token's test runner enforce every assertion #49