Skip to content

Unblock the Dependabot queue: four CI failures and an MSRV decision #358

Open
#360
Open
Unblock the Dependabot queue: four CI failures and an MSRV decision#358
#360
Labels
bugSomething isn't workingenhancementNew feature or requestmediumRoadmap items to schedule within the current quarter. Clear scope, normal review cycles.testingTest coverage, test infrastructure, and verification tooling work.

Description

@leynos
leynos
opened on Oct 23, 2026

Problem

Every open Dependabot PR here has red CI, and none can merge. This is the oldest PR in the queue:

The rest are:

Main's last recorded green run was Coverage (main) on 11 October (99cbad9).

The failures have four separate causes, plus an upstream MSRV change that needs a decision. Each is listed below with its evidence (logs read on 23 October).

1. New nightly lints (#357, the toolchain bump)

-D warnings turns two new nightly diagnostics into errors.

  • Linux (job): use of deprecated method std::sync::atomic::Atomic::<usize>::fetch_update: renamed to try_update at src/worker_process/privileges.rs:240.
  • macOS (job): trailing semicolon in macro used in expression position at tests/bootstrap_overrides.rs:111 and :112. Those are the color_eyre::eyre::bail!(...) match arms.
  • Windows: Build binaries fails the same way.

Fix:

  1. Replace fetch_update with try_update.
  2. Wrap each bail! arm in a block (or use return Err(eyre!(...))).
  3. Fix any other new lints that make lint reports on the new toolchain. Fix the code; don't add #[allow].

Dependabot PRs don't get CodeRabbit reviews, so do this on a leynos branch with the bump included. The PR says "Supersedes #357", and #357 is closed once it merges.

2. Deprecated makefile_lossless API (#354, the minor-and-patch group)

The group bumps makefile-lossless, which deprecates Parse::ok. The error is use of deprecated method makefile_lossless::Parse::<T>::ok: use is_ok instead, at tests/makefile_version_guard.rs:18 (job).

Fix: use is_ok(), and check the group's other four updates for further deprecations. Like #357, this needs a superseding leynos-branch PR.

3. A hard-coded install-whitaker pin (#336, the shared-actions group)

tests/whitaker_install_pin.rs:126 asserts that install-whitaker is pinned to exactly 6cec89bac47a21cf756d68d638a9a510998e57f8, so any shared-actions bump fails it (job):

install-whitaker must pin the revision QG-002 lists as compliant
  left: "e693d81d176d2305e2b736a0be0f9038085dec7f"
 right: "6cec89bac47a21cf756d68d638a9a510998e57f8"

QG-002 no longer approves a single revision. Since leynos/concordat#324 (23 October), compliant_install_whitaker_refs lists 77 approved commits, up to d5fded1107ee6822fdbf9539d81ff18ce6ce073a, which installs whitaker-installer 0.2.10.

Fix: the test should check that the pin is a full 40-character SHA and belongs to that list. A local copy of the list is acceptable if a comment names its source and the test fails clearly when the pin isn't on it. Every routine shared-actions bump should then pass. Prove it with a mutation: set the pin to a SHA that isn't on the list, and show the test fails.

4. invalid value for parameter "TimeZone": "UTC" (#351 and possibly others)

tests/password_reuse_e2e.rs:115 (a_second_bootstrap_adopts_a_password_that_authenticates) fails with the adopted password did not log in: invalid value for parameter "TimeZone": "UTC" (job, 21 October).

This is the same error as leynos/mxd#665, where mxd's Postgres coverage lane began failing after 12 October. That makes it a property of the embedded PostgreSQL bundle or its runtime environment, not of mxd. Ruled out on mxd: the runner has /usr/share/zoneinfo/UTC, and setting TZ or TZDIR makes no difference.

Probes, in order:

  1. The bundle's timezone data. PostgreSQL reads its own compiled-in share/timezone directory, not TZDIR, unless it was built with --with-system-tzdata. List share/timezone in the installation that postgresql_embedded unpacks, and record the PostgreSQL version and the bundle's release date. A missing or pruned timezone directory, or a new upstream bundle published after 12 October, would explain the error.
  2. Which PostgreSQL version resolves. Find out whether postgresql_embedded picks the newest bundle matching its version requirement at runtime. If so, a new upstream binary release changes behaviour without any lockfile change. Compare a passing run (before 12 October) with a failing one.
  3. The server's own log from a failing run.

Fix: depends on what the probes find.

  • If the bundle lacks timezone data: pin the bundle version, fall back to timezone = 'UTC' with system tzdata, or report it upstream. Reporting upstream needs the user's approval, because it means posting outside leynos/.
  • Add a regression test that starts a cluster and runs SET TimeZone = 'UTC'.

The fix should go out in a pg-embed-setup-unpriv release, because mxd's migration (leynos/mxd#667) depends on it.

5. Upstream raised its MSRV to 1.94 (#207 and #269)

postgresql_embedded and postgresql_archive 0.20.3 and later, including 0.21.0 and 0.22.0 (released 21 October), declare rust-version = 1.94. This crate declares rust-version = "1.92" and holds both at ^0.20.2 deliberately: see the Cargo.toml comment and the 29 September 0.6.1 regression. So the Build at the declared rust-version job fails on both PRs.

Decision for the user: either keep 1.92 and ignore these updates (an ignore entry in .github/dependabot.yml for >= 0.20.3, with the reason in a comment), or raise rust-version to 1.94 in a minor release and take 0.22.0.

Known consumers: axinite (0.6.3) and mxd (0.5.2, migrating). Check each consumer's toolchain before raising it.

#207 and #245 also conflict with main. Recreate them or close them once their replacements land.

Order of work

  1. Probe 4 first, because it blocks the mxd migration and may be what makes Run Dependabot pull requests on GitHub-hosted runners #351 red.
  2. Then fixes 3, 1 and 2, each as its own leynos-branch PR. Each PR names the Dependabot PR it supersedes; that PR is closed after the merge.
  3. Then put decision 5 to the user.

Once main is green, re-run #351 (Dependabot pull requests on GitHub-hosted runners).

Constraints

Done when

  • Main CI is green on Linux (root and unprivileged), macOS and Windows.
  • Every Dependabot PR listed above is merged, superseded and closed, or ignored with a recorded reason.
  • The TimeZone failure has a confirmed cause and a regression test.

Activity

  1. coderabbitai commented on Oct 23, 2026

    @coderabbitai
    coderabbitaiBot
    on Oct 23, 2026 – with coderabbitai
    Contributor
    🔗 Related PRs

    leynos/podbot#267 - Run Dependabot pull requests on GitHub-hosted runners [open]
    leynos/cuprum#646 - Run Dependabot pull requests on GitHub-hosted runners [open]
    leynos/repovec-appliance#187 - Run Dependabot pull requests on GitHub-hosted runners [open]
    #351 - Run Dependabot pull requests on GitHub-hosted runners [open]

    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
  auto_enrich:
    enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. coderabbitai
    added
    bugSomething isn't working
    mediumRoadmap items to schedule within the current quarter. Clear scope, normal review cycles.
    testingTest coverage, test infrastructure, and verification tooling work.
    enhancementNew feature or request
    on Oct 23, 2026

  3. leynos commented on Oct 23, 2026

    @leynos
    leynos
    on Oct 23, 2026
    OwnerAuthor

    @CodeRabbit plan

  4. coderabbitai commented on Oct 23, 2026

    @coderabbitai
    coderabbitaiBot
    on Oct 23, 2026 – with coderabbitai
    Contributor

    Coding Plan

    Summary

    • Use the regression test as the probe. Confirm the cause on every CI lane before you change behaviour. The evidence so far is the hard-linked tzdata in the Theseus 18.6.1 bundle and the hard-link skip in postgresql_archive versions before 0.22.0.
    • Ship an MSRV-neutral interim fix. Pin the default bundle to an exact working version, keep the PG_VERSION_REQ override, and make installation inference honour the requirement.
    • Fix each remaining CI cause on its own leynos branch that supersedes the matching Dependabot PR. Preserve the behavioural contracts and do not suppress lints.
    • Put decision 5 to the user with commit-pinned consumer evidence. Recommend MSRV 1.94 with 0.22.0, which contains the real extraction fix. Then clean up the queue.
    Design Choices
    Design Choice 1: How to fix the TimeZone failure while decision 5 is open

    Options Considered:

    1. Pin the default PostgreSQL bundle to =18.6.0 when PG_VERSION_REQ is unset. Keep the PG_VERSION_REQ override.
    2. Upgrade both crates to 0.22.0 and raise rust-version to 1.94.
    3. Set timezone = 'UTC' in the server configuration. This does not work because clients such as Diesel still send SET TIME ZONE 'UTC', and every other hard-linked zone (for example Europe/London) stays missing.
    4. Recreate missing aliases after extraction. This is not feasible because the archive's link list is not available after extraction.
    5. Report the bug upstream. This is not needed because 0.22.0 already handles hard links.

    Chosen Option: 1

    Rationale: Use Option 1 now, with Option 2 as the recommendation for decision 5. The full upstream fix (postgresql_archive 0.22.0) needs MSRV 1.94, and the ticket reserves that decision for the user. Main must still turn green. The pin is MSRV-neutral, deterministic and reversible. It also prevents cache and installation-directory reuse of broken 18.6.1 trees.

    Design Choice 2: Where to keep the QG-002 approved list

    Options Considered:

    1. Put a local constant list in the test, with a comment that names leynos/concordat commit f46a4848ad5e62cfd170360074cddc3a792f3ea7 and the rule.yaml path.
    2. Fetch the list from concordat at test time.

    Chosen Option: 1

    Rationale: Option 1. The ticket permits it, the tests stay hermetic, and the test fails clearly when the pin is not on the list.

    Design Choice 3: How to replace `fetch_update` without breaking MSRV 1.92

    Options Considered:

    1. Use try_update.
    2. Use an explicit compare_exchange loop.

    Chosen Option: 1

    Rationale: try_update can be newer than Rust 1.92. Use Option 1 if make msrv passes at the declared rust-version. Otherwise use Option 2. Both remove the deprecated call without #[allow].

    💡 User Tips

    Regenerate the plan with different choices with @coderabbitai <feedback>.

    Implementation Steps

    Phase 1: TimeZone failure — probe, confirm, regression test, interim fix

    This phase comes first because it blocks the mxd migration (#667). It also turns every other PR red. The regression test also serves as the probe: it gathers safe diagnostics before cleanup.

    Task 1: Task 1.1: Add a live-cluster TimeZone regression test with safe diagnostics

    Add a test that reproduces the failure and records the evidence that the probes need.

    • Create tests/timezone_regression.rs. Follow the tests/password_reuse_e2e.rs pattern:
      • Use TestSandbox, env_without_timezone() and TestCluster::new_split().
      • Map skips through cluster_skip_message.
      • Use the same feature gating and the Diesel PgConnection.
    • Use one cluster. Run SET TimeZone = '<zone>' and then SHOW TimeZone for each of UTC, Etc/UTC and Europe/London. These are all hard-linked aliases in 18.6.1. Use an rstest case table that iterates inside the single cluster.
    • On failure, add only allowlisted diagnostics to the error. Collect them before the guard drops:
      • the SELECT version() output, if a connection exists;
      • the installation path from handle.settings().installation_dir;
      • whether share/timezone, share/timezone/UCT and share/timezone/UTC exist, and their file types;
      • a bounded tail of <data_dir>/start.log.
    • Never print Settings, passwords, connection URLs or unrestricted command-debug logs.
    Task 2: Task 1.2: Run the probes and record the confirmed cause

    Confirm the cause with run evidence, not with release dates.

    • Push the test without a fix to a leynos branch. Run CI on Linux root, Linux unprivileged, macOS and Windows.
    • Record the following for each lane:
      • the resolved bundle version;
      • the cache-hit status, from the find_matching_cached_version tracing;
      • the share/timezone evidence;
      • the start.log tail.
    • Run tar -tvzf offline on the 18.6.0 and 18.6.1 assets for each CI target, including Windows and macOS. Confirm the hard-link entries and the tzdata configuration.
    • Reconcile the date gap. mxd reports failures after 12 October, but 18.6.1 was published on 18 October. Check whether mxd's 0.5.2 resolution or another cause explains the earlier failures. Do not claim the cause for those runs without evidence.
    • Search .github/workflows/ for PG_VERSION_REQ, PG_RUNTIME_DIR and any PostgreSQL cache steps. A preset requirement or a persisted cache changes what the pin affects.
    • Record the confirmed cause in the PR description and on the ticket. Upstream reporting is not needed. If the probes show a defect that 0.22.0 does not fix, ask the user for approval before you post outside leynos/.
    Task 3: Task 1.3: Pin the default PostgreSQL bundle

    Stop runtime drift to the broken bundle while MSRV stays at 1.92.

    • Update apply_version in src/lib.rs. When PG_VERSION_REQ is unset, set settings.version to a named constant with the value =18.6.0.
    • Add a comment to the constant. It must cite the hard-link extraction gap and the postgresql_archive 0.22.0 fix.
    • Keep PG_VERSION_REQ as the override. Update the variable documentation in src/main.rs:16–24.
    • If Task 1.2 shows that a CI platform has no usable 18.6.0 asset, choose the nearest working exact version for that platform. Document the choice.
    • Add a unit test next to the existing apply_version tests. It asserts the default requirement and the override precedence.
    Task 4: Task 1.4: Make installation inference honour the requested version

    Prevent the root path from selecting a stale 18.6.1 directory under a reused installation root.

    • Update the resolver in src/cluster/installation.rs (~174–218). Filter the candidate subdirectories with the settings' VersionReq before you select the highest version.
    • Keep the lexical fallback only for names that do not parse as semver. Keep the trusted-directory behaviour.
    • Extend the existing resolver tests. Add a root that contains 18.6.0/bin and 18.6.1/bin, and assert that =18.6.0 selects 18.6.0.
    Task 5: Task 1.5: Document the fix and stale installations

    Tell consumers what changed and how to recover.

    • Add an Unreleased entry to CHANGELOG.md. It must cover:
      • the cause;
      • the default pin;
      • the PG_VERSION_REQ override;
      • the advice to delete 18.6.1 trees from PG_BINARY_CACHE_DIR and from installation roots. Cache lookup accepts these trees without validation.
    • Add a short note on stale or incomplete cache entries to docs/cache.md.
    • Gate each commit in this order: make check-fmt, make lint, make test, make markdownlint. Merge only after a CodeRabbit review at the exact head with no ❌ rows. Ask the user to release, because mxd #667 depends on this fix.
    🤖 Prompt for AI agents 
    Implement the TimeZone regression test and the interim fix. The goal is to
confirm the cause, pin the default PostgreSQL bundle to `=18.6.0`, and keep MSRV
at 1.92.

Regression test (`tests/timezone_regression.rs`):
- Follow the `tests/password_reuse_e2e.rs` pattern. Use `TestSandbox`,
`env_without_timezone()`, `TestCluster::new_split()`, `cluster_skip_message`,
the same feature gating and the Diesel `PgConnection`.
- Use one cluster. Run `SET TimeZone = '<zone>'` then `SHOW TimeZone` for `UTC`,
`Etc/UTC` and `Europe/London`, using an rstest case table that iterates inside
the single cluster.
- On failure, attach only allowlisted diagnostics, collected before the guard
drops: `SELECT version()` output, `handle.settings().installation_dir`,
existence and file types of `share/timezone`, `share/timezone/UCT` and
`share/timezone/UTC`, and a bounded tail of `<data_dir>/start.log`.
- Never print `Settings`, passwords, connection URLs or unrestricted
command-debug logs.

Probes:
- Run the test without a fix on a leynos branch across Linux root, Linux
unprivileged, macOS and Windows. Record the evidence listed in the plan. Search
`.github/workflows/` for `PG_VERSION_REQ`, `PG_RUNTIME_DIR` and PostgreSQL cache
steps.

Pin:
- In `src/lib.rs` `apply_version`, when `PG_VERSION_REQ` is unset, set
`settings.version` to a named constant `=18.6.0`. Comment the constant with the
hard-link extraction gap and the `postgresql_archive` 0.22.0 fix.
- Keep the `PG_VERSION_REQ` override. Update the docs in `src/main.rs:16–24`.
- Add a unit test for the default and the override precedence.

Installation inference:
- In `src/cluster/installation.rs` (~174–218), filter candidate subdirectories
by the settings' `VersionReq` before selecting the highest. Keep the lexical
fallback only for non-semver names. Keep trusted-directory behaviour.
- Add a resolver test with `18.6.0/bin` and `18.6.1/bin` that asserts `=18.6.0`
selects `18.6.0`.

Docs:
- Add an Unreleased `CHANGELOG.md` entry (cause, pin, override, advice to delete
`18.6.1` trees from `PG_BINARY_CACHE_DIR` and installation roots). Add a
stale-cache note to `docs/cache.md`.
- Gate each commit: `make check-fmt`, `make lint`, `make test`, `make
markdownlint`.

    Phase 2: QG-002 install-whitaker pin (supersedes #336)

    Change the Whitaker contract from one exact SHA to membership of the approved list. Include the shared-actions bump in the same branch.

    Task 1: Task 2.1: Replace the exact-SHA assertion with list membership
    • In tests/whitaker_install_pin.rs (~15–23, 113–130), add a local constant with the 77 approved SHAs.
    • Add a comment above the constant. It must name leynos/concordat commit f46a4848ad5e62cfd170360074cddc3a792f3ea7 (PR #324) and the path platform-standards/canon/lint-rules/whitaker-provisioning/rule.yaml.
    • Assert that the pin is a full 40-character lowercase hexadecimal SHA. Then assert that it is on the list. The failure message must name the pin and the list source.
    • Add guard assertions on the list itself:
      • the list has exactly 77 entries;
      • every entry is unique;
      • every entry is a valid 40-character SHA.
        These follow the explicit-count pattern in tests/build_standard_contract.rs.
    • Add parameterized negative cases for the membership helper: one valid SHA that is not on the list and one short SHA.
    • Keep the existing contracts:
      • step selection;
      • the installer-version input, with a minimum of 0.2.9;
      • rejection of legacy cargo install and cargo binstall commands.
    Task 2: Task 2.2: Apply the shared-actions bump and prove it with a mutation
    • Apply the #336 group revisions to .github/workflows/*.yml. The new install-whitaker revision e693d81d176d2305e2b736a0be0f9038085dec7f is on the list.
    • Make a local mutation. Set the pin in .github/workflows/ci.yml:90 to an off-list SHA, run the test, record the failure output, and revert the change. Put the output in the PR description.
    • Update docs/developers-guide.md:219–229. Describe the list-membership policy and how to refresh the local copy from concordat.
    • Title the PR "Supersedes #336". Gate the PR, get the review, merge, and then close #336.
    🤖 Prompt for AI agents 
    Replace the exact-SHA install-whitaker assertion with list membership, and apply
the #336 shared-actions bump.

In `tests/whitaker_install_pin.rs`:
- Add a local constant with the 77 approved SHAs. Comment it with
`leynos/concordat` commit `f46a4848ad5e62cfd170360074cddc3a792f3ea7` (PR #324)
and path `platform-standards/canon/lint-rules/whitaker-provisioning/rule.yaml`.
- Assert the pin is a 40-character lowercase hexadecimal SHA, then assert it is
on the list. The failure message names the pin and the list source.
- Add list guards: exactly 77 entries, all unique, all valid 40-character SHAs.
Follow the explicit-count pattern in `tests/build_standard_contract.rs`.
- Add parameterized negative cases: one valid off-list SHA and one short SHA.
- Keep step selection, the installer-version input (minimum `0.2.9`) and the
rejection of legacy `cargo install` and `cargo binstall` commands.

Workflows and docs:
- Apply the #336 revisions to `.github/workflows/*.yml`. The install-whitaker
revision is `e693d81d176d2305e2b736a0be0f9038085dec7f`.
- Mutate the pin at `.github/workflows/ci.yml:90` to an off-list SHA, run the
test, record the failure, then revert.
- Update `docs/developers-guide.md:219–229` with the list-membership policy and
the refresh procedure.
- Title the PR "Supersedes #336".

    Phase 3: New nightly lints (supersedes #357)

    Move to nightly-2026-10-23 and fix the new diagnostics in the code.

    Task 1: Task 3.1: Bump the toolchain and fix the reported lints
    • Set rust-toolchain.toml to nightly-2026-10-23.
    • Replace fetch_update in src/worker_process/privileges.rs:238–251 as decided in Assumption 3. Keep these properties:
      • SeqCst ordering;
      • saturating decrement;
      • the debug assertion against a drop at zero.
        The existing serial nested-guard tests must still pass.
    • In tests/bootstrap_overrides.rs:98–113, wrap each bail! arm in a block. Both unexpected outcomes must still fail.
    Task 2: Task 3.2: Remove the remaining new-toolchain fallout
    • Run make lint and make test on the new nightly. Fix every new warning in the code. Do not add #[allow].
    • If trybuild UI tests exist, regenerate the .stderr snapshots for the new compiler diagnostics. Review each diff.
    • Run make msrv. This confirms that the fixes still compile on the declared rust-version.
    • Confirm that the Windows and macOS Build binaries steps pass. Title the PR "Supersedes #357", merge it, and close #357.
    🤖 Prompt for AI agents 
    Move to `nightly-2026-10-23` and fix the new lints without suppressing them.

- Set `rust-toolchain.toml` to `nightly-2026-10-23`.
- Replace `fetch_update` in `src/worker_process/privileges.rs:238–251`. Use
`try_update` if `make msrv` passes at the declared `rust-version`. Otherwise use
an explicit `compare_exchange` loop. Keep `SeqCst` ordering, the saturating
decrement and the debug assertion against a drop at zero. Keep the serial
nested-guard tests passing.
- In `tests/bootstrap_overrides.rs:98–113`, wrap each `bail!` arm in a block.
Both unexpected outcomes must still fail.
- Run `make lint` and `make test`. Fix every new warning. Do not add `#[allow]`.
- Regenerate and review any trybuild `.stderr` snapshots.
- Run `make msrv`. Confirm the Windows and macOS `Build binaries` steps pass.
- Title the PR "Supersedes #357".

    Phase 4: Minor-and-patch group (supersedes #354)

    Apply the Dependabot minor-and-patch group and fix the deprecations.

    Task 4.1: Apply the group and fix the deprecations
    • Apply the five bumps:
      • serde_json to 1.0.154;
      • sha2 to 0.11.1;
      • makefile-lossless to 0.4.8;
      • proptest to 1.12.0;
      • trybuild to 1.0.122.
    • In tests/makefile_version_guard.rs:13–22, replace Parse::ok with is_ok(). Keep the strict no-recovery check and the positioned_errors() report.
    • Keep the make --dry-run VERSION= lint subprocess contract unchanged.
    • Check the other four updates for deprecations. Check trybuild snapshot changes in particular.
    • Run the gates and make msrv. Title the PR "Supersedes #354", merge it, and close #354.
    🤖 Prompt for AI agents 
    Apply the five dependency bumps and fix the `makefile_lossless` deprecation.

- Bump `serde_json` to 1.0.154, `sha2` to 0.11.1, `makefile-lossless` to 0.4.8,
`proptest` to 1.12.0 and `trybuild` to 1.0.122.
- In `tests/makefile_version_guard.rs:13–22`, replace `Parse::ok` with
`is_ok()`. Keep the strict no-recovery check and the `positioned_errors()`
report.
- Keep the `make --dry-run VERSION= lint` subprocess contract unchanged.
- Check the other four updates for deprecations. Check trybuild snapshot changes
in particular.
- Run the gates and `make msrv`. Title the PR "Supersedes #354".

    Phase 5: MSRV decision and queue cleanup

    Put decision 5 to the user with evidence. Then carry out the chosen option.

    Task 1: Task 5.1: Present the decision to the user
    • State that 0.22.0 is the upstream fix for the TimeZone extraction gap. Without it, the Phase 1 pin stays in place.
    • Give the consumer evidence at commit-pinned revisions:
      • axinite 0553ee5: rust-version = "1.95", stable CI, requirement ^0.6.3;
      • mxd da9c7da: no rust-version, toolchain nightly-2026-03-26, requirement 0.5.2 and migrating.
    • Note that both consumers meet Rust 1.94. Note also that an earlier branch snapshot showed axinite at 1.92.
    • Recommend Option B: raise rust-version to 1.94 in a minor release and take 0.22.0.
    Task 2: Task 5.2: Option B — raise MSRV to 1.94 and take 0.22.0
    • Update the following together, so that the retry classification still shares one archive error type:
      • in Cargo.toml, set postgresql_embedded and postgresql_archive to 0.22.0, and set rust-version to 1.94;
      • rewrite the comment at Cargo.toml:145–152.
    • Fix API changes from 0.21 and 0.22. Check src/test_support/bootstrap_retry/tests.rs in particular.
    • Add an Unreleased entry to CHANGELOG.md that records the deliberate MSRV raise.
    • Run the Phase 1 regression test against 18.6.1. If it passes, remove or relax the default pin. Keep it if the user prefers reproducible bundles.
    • Run make msrv against 1.94. Close #207 and #269 as superseded.
    Task 3: Task 5.3: Option A — keep 1.92 and ignore the updates
    • Add ignore entries to .github/dependabot.yml for postgresql_embedded and postgresql_archive with >= 0.20.3. Add a comment that cites the 1.94 MSRV, the 0.6.1 regression, and the dependency of the TimeZone pin on this choice.
    • Close #207 and #269 with the recorded reason. The Phase 1 pin stays in place.
    Task 4: Task 5.4: Clear the rest of the queue
    • After main is green on all lanes, comment @dependabot recreate on #245. Merge it after a green run and a review.
    • Re-run #351 on GitHub-hosted runners.
    • Do not change #216.
    • Confirm that every PR in the list is merged, superseded and closed, or ignored with a recorded reason.
    🤖 Prompt for AI agents 
    Put the MSRV decision to the user, then carry out the chosen option. Do not
choose an option without the user's decision.

Decision:
- State that 0.22.0 is the upstream fix. Without it, the Phase 1 pin stays.
- Give the evidence: axinite `0553ee5` (`rust-version = "1.95"`, stable CI,
requirement `^0.6.3`) and mxd `da9c7da` (no `rust-version`, toolchain
`nightly-2026-03-26`, requirement `0.5.2`, migrating). Both meet Rust 1.94. An
earlier branch snapshot showed axinite at 1.92.
- Recommend Option B.

Option B:
- In `Cargo.toml`, set `postgresql_embedded` and `postgresql_archive` to
`0.22.0` together, so retry classification shares one archive error type. Set
`rust-version` to `1.94`. Rewrite the comment at `Cargo.toml:145–152`.
- Fix API changes from 0.21 and 0.22. Check
`src/test_support/bootstrap_retry/tests.rs`.
- Add an Unreleased `CHANGELOG.md` entry for the MSRV raise.
- Run the Phase 1 regression test against `18.6.1`. If it passes, remove or
relax the default pin, unless the user prefers reproducible bundles.
- Run `make msrv` against 1.94. Close #207 and #269 as superseded.

Option A:
- Add `ignore` entries in `.github/dependabot.yml` for both crates with `>=
0.20.3`. Comment the 1.94 MSRV, the 0.6.1 regression and the pin dependency.
- Close #207 and #269 with the recorded reason. Keep the Phase 1 pin.

Queue cleanup:
- When main is green on all lanes, comment `@dependabot recreate` on #245 and
merge after a green run and review.
- Re-run #351 on GitHub-hosted runners.
- Do not change #216.
- Confirm every PR is merged, superseded and closed, or ignored with a recorded
reason.
    Research

    Ticket Summary

    Unblock the Dependabot queue. Fix four CI failure causes and put the MSRV decision to the user. The four causes are new nightly lints (#357), a deprecated makefile_lossless API (#354), an exact install-whitaker pin assertion (#336), and the PostgreSQL invalid value for parameter "TimeZone": "UTC" failure. Confirm the TimeZone cause with evidence. Add a regression test. Ship the fix in a release that the user publishes.

    Codebase Summary

    pg-embed-setup-unpriv bootstraps embedded PostgreSQL through postgresql_embedded and postgresql_archive. Both have the requirement 0.20.2 and resolve to 0.20.4 in the lockfile. src/lib.rs (apply_version, apply_paths) maps PG_* environment variables onto upstream Settings. When PG_VERSION_REQ is unset, the upstream version is VersionReq::STAR. This requirement resolves to the newest Theseus release at install time.

    src/cluster/startup.rs runs cache_integration::try_use_binary_cache before setup. Cache lookup in src/cache/operations/lookup.rs requires only .complete and bin/. src/cluster/installation.rs (refresh_worker_installation_dir) picks the highest versioned subdirectory. It does not filter by the requirement. Upstream pg_ctl start writes the server log to <data_dir>/start.log. Upstream drop deletes that directory when temporary=true.

    Investigation established these facts:

    • Theseus 18.6.1 was published on 18 October 2026. It bundles share/timezone. It was built without --with-system-tzdata.
    • In that bundle, UTC, Etc/UTC and 255 other zones are tar hard links.
    • The extractors in postgresql_archive 0.20.2, 0.20.4 and 0.21.0 skip hard links. 0.22.0 extracts hard links but declares rust-version = 1.94.0.
    • Theseus 18.6.0 uses system tzdata at /usr/share/zoneinfo.

    Diesel's PgConnection::establish issues SET TIME ZONE 'UTC'. For this reason, the error appears as a login failure.

    Supporting contracts:

    • tests/whitaker_install_pin.rs asserts one exact SHA for install-whitaker.
    • tests/makefile_version_guard.rs uses Parse::ok.
    • src/worker_process/privileges.rs uses fetch_update.
    • tests/bootstrap_overrides.rs has bail! match arms.
    • make msrv (scripts/msrv_check.py) checks a fresh resolution on the declared rust-version.

    File-level Change Summary

    • tests/timezone_regression.rs (new): live-cluster SET TimeZone test with allowlisted diagnostics.
    • src/lib.rs: default exact bundle requirement in apply_version; unit test.
    • src/main.rs: document the default and the PG_VERSION_REQ override.
    • src/cluster/installation.rs: filter inferred installation directories by VersionReq; tests.
    • CHANGELOG.md, docs/cache.md: TimeZone cause, stale-tree recovery, and later the MSRV entry.
    • tests/whitaker_install_pin.rs: membership of the QG-002 list, list guards, negative cases.
    • .github/workflows/*.yml: shared-actions group revisions (#336).
    • docs/developers-guide.md: QG-002 list policy and refresh procedure.
    • rust-toolchain.toml: nightly-2026-10-23.
    • src/worker_process/privileges.rs: replace fetch_update and keep its semantics.
    • tests/bootstrap_overrides.rs: put the bail! arms in blocks.
    • tests/makefile_version_guard.rs: replace Parse::ok with is_ok().
    • Cargo.toml, Cargo.lock: group bumps; Option B: 0.22.0 and rust-version = "1.94".
    • .github/dependabot.yml: Option A only: ignore entries with the reason.

    Follow-up Agent Prompts

    Add tests/timezone_regression.rs to pg-embed-setup-unpriv. Copy the cluster start, skip and Diesel connection pattern in tests/password_reuse_e2e.rs.
Use env_without_timezone(). In one cluster, run SET TimeZone = '<zone>' and then SHOW TimeZone for UTC, Etc/UTC and Europe/London.
On failure, report only these items:
- the output of SELECT version();
- the installation_dir path;
- whether share/timezone, share/timezone/UCT and share/timezone/UTC exist, and their file types;
- a bounded tail of <data_dir>/start.log, read before the guard drops.
Never print Settings, passwords or connection URLs.
Gate each commit in this order: make check-fmt, make lint, make test, make markdownlint.

    In tests/whitaker_install_pin.rs, replace the exact-SHA assertion with membership of the QG-002 compliant_install_whitaker_refs list.
Copy the 77 SHAs from leynos/concordat at f46a4848ad5e62cfd170360074cddc3a792f3ea7, path platform-standards/canon/lint-rules/whitaker-provisioning/rule.yaml. Put a comment that cites this source above the list.
Assert that the list has 77 unique 40-character hexadecimal entries.
Assert that the pin is a 40-character SHA and that it is on the list.
Add negative rstest cases for an off-list SHA and a short SHA.
Keep the step-selection, installer-version and legacy-command contracts.
Apply the #336 revisions. Prove the check with a local mutation to an off-list SHA, then revert it.

    Bump rust-toolchain.toml to nightly-2026-10-23.
Replace fetch_update in src/worker_process/privileges.rs. Keep SeqCst ordering, the saturating decrement and the debug assertion. Use try_update only if make msrv passes. Otherwise use a compare_exchange loop.
Put each bail! arm in tests/bootstrap_overrides.rs in a block.
Fix all other new lints in the code. Do not add #[allow].
Regenerate and review any trybuild snapshots.

    🚀 Next Steps

    🤖 All AI agent prompts combined 
    Task: 1

Implement the TimeZone regression test and the interim fix. The goal is to
confirm the cause, pin the default PostgreSQL bundle to `=18.6.0`, and keep MSRV
at 1.92.

Regression test (`tests/timezone_regression.rs`):
- Follow the `tests/password_reuse_e2e.rs` pattern. Use `TestSandbox`,
`env_without_timezone()`, `TestCluster::new_split()`, `cluster_skip_message`,
the same feature gating and the Diesel `PgConnection`.
- Use one cluster. Run `SET TimeZone = '<zone>'` then `SHOW TimeZone` for `UTC`,
`Etc/UTC` and `Europe/London`, using an rstest case table that iterates inside
the single cluster.
- On failure, attach only allowlisted diagnostics, collected before the guard
drops: `SELECT version()` output, `handle.settings().installation_dir`,
existence and file types of `share/timezone`, `share/timezone/UCT` and
`share/timezone/UTC`, and a bounded tail of `<data_dir>/start.log`.
- Never print `Settings`, passwords, connection URLs or unrestricted
command-debug logs.

Probes:
- Run the test without a fix on a leynos branch across Linux root, Linux
unprivileged, macOS and Windows. Record the evidence listed in the plan. Search
`.github/workflows/` for `PG_VERSION_REQ`, `PG_RUNTIME_DIR` and PostgreSQL cache
steps.

Pin:
- In `src/lib.rs` `apply_version`, when `PG_VERSION_REQ` is unset, set
`settings.version` to a named constant `=18.6.0`. Comment the constant with the
hard-link extraction gap and the `postgresql_archive` 0.22.0 fix.
- Keep the `PG_VERSION_REQ` override. Update the docs in `src/main.rs:16–24`.
- Add a unit test for the default and the override precedence.

Installation inference:
- In `src/cluster/installation.rs` (~174–218), filter candidate subdirectories
by the settings' `VersionReq` before selecting the highest. Keep the lexical
fallback only for non-semver names. Keep trusted-directory behaviour.
- Add a resolver test with `18.6.0/bin` and `18.6.1/bin` that asserts `=18.6.0`
selects `18.6.0`.

Docs:
- Add an Unreleased `CHANGELOG.md` entry (cause, pin, override, advice to delete
`18.6.1` trees from `PG_BINARY_CACHE_DIR` and installation roots). Add a
stale-cache note to `docs/cache.md`.
- Gate each commit: `make check-fmt`, `make lint`, `make test`, `make
markdownlint`.
===============================================================================

Task: 2

Replace the exact-SHA install-whitaker assertion with list membership, and apply
the #336 shared-actions bump.

In `tests/whitaker_install_pin.rs`:
- Add a local constant with the 77 approved SHAs. Comment it with
`leynos/concordat` commit `f46a4848ad5e62cfd170360074cddc3a792f3ea7` (PR #324)
and path `platform-standards/canon/lint-rules/whitaker-provisioning/rule.yaml`.
- Assert the pin is a 40-character lowercase hexadecimal SHA, then assert it is
on the list. The failure message names the pin and the list source.
- Add list guards: exactly 77 entries, all unique, all valid 40-character SHAs.
Follow the explicit-count pattern in `tests/build_standard_contract.rs`.
- Add parameterized negative cases: one valid off-list SHA and one short SHA.
- Keep step selection, the installer-version input (minimum `0.2.9`) and the
rejection of legacy `cargo install` and `cargo binstall` commands.

Workflows and docs:
- Apply the #336 revisions to `.github/workflows/*.yml`. The install-whitaker
revision is `e693d81d176d2305e2b736a0be0f9038085dec7f`.
- Mutate the pin at `.github/workflows/ci.yml:90` to an off-list SHA, run the
test, record the failure, then revert.
- Update `docs/developers-guide.md:219–229` with the list-membership policy and
the refresh procedure.
- Title the PR "Supersedes #336".
===============================================================================

Task: 3

Move to `nightly-2026-10-23` and fix the new lints without suppressing them.

- Set `rust-toolchain.toml` to `nightly-2026-10-23`.
- Replace `fetch_update` in `src/worker_process/privileges.rs:238–251`. Use
`try_update` if `make msrv` passes at the declared `rust-version`. Otherwise use
an explicit `compare_exchange` loop. Keep `SeqCst` ordering, the saturating
decrement and the debug assertion against a drop at zero. Keep the serial
nested-guard tests passing.
- In `tests/bootstrap_overrides.rs:98–113`, wrap each `bail!` arm in a block.
Both unexpected outcomes must still fail.
- Run `make lint` and `make test`. Fix every new warning. Do not add `#[allow]`.
- Regenerate and review any trybuild `.stderr` snapshots.
- Run `make msrv`. Confirm the Windows and macOS `Build binaries` steps pass.
- Title the PR "Supersedes #357".
===============================================================================

Task: 4

Apply the five dependency bumps and fix the `makefile_lossless` deprecation.

- Bump `serde_json` to 1.0.154, `sha2` to 0.11.1, `makefile-lossless` to 0.4.8,
`proptest` to 1.12.0 and `trybuild` to 1.0.122.
- In `tests/makefile_version_guard.rs:13–22`, replace `Parse::ok` with
`is_ok()`. Keep the strict no-recovery check and the `positioned_errors()`
report.
- Keep the `make --dry-run VERSION= lint` subprocess contract unchanged.
- Check the other four updates for deprecations. Check trybuild snapshot changes
in particular.
- Run the gates and `make msrv`. Title the PR "Supersedes #354".
===============================================================================

Task: 5

Put the MSRV decision to the user, then carry out the chosen option. Do not
choose an option without the user's decision.

Decision:
- State that 0.22.0 is the upstream fix. Without it, the Phase 1 pin stays.
- Give the evidence: axinite `0553ee5` (`rust-version = "1.95"`, stable CI,
requirement `^0.6.3`) and mxd `da9c7da` (no `rust-version`, toolchain
`nightly-2026-03-26`, requirement `0.5.2`, migrating). Both meet Rust 1.94. An
earlier branch snapshot showed axinite at 1.92.
- Recommend Option B.

Option B:
- In `Cargo.toml`, set `postgresql_embedded` and `postgresql_archive` to
`0.22.0` together, so retry classification shares one archive error type. Set
`rust-version` to `1.94`. Rewrite the comment at `Cargo.toml:145–152`.
- Fix API changes from 0.21 and 0.22. Check
`src/test_support/bootstrap_retry/tests.rs`.
- Add an Unreleased `CHANGELOG.md` entry for the MSRV raise.
- Run the Phase 1 regression test against `18.6.1`. If it passes, remove or
relax the default pin, unless the user prefers reproducible bundles.
- Run `make msrv` against 1.94. Close #207 and #269 as superseded.

Option A:
- Add `ignore` entries in `.github/dependabot.yml` for both crates with `>=
0.20.3`. Comment the 1.94 MSRV, the 0.6.1 regression and the pin dependency.
- Close #207 and #269 with the recorded reason. Keep the Phase 1 pin.

Queue cleanup:
- When main is green on all lanes, comment `@dependabot recreate` on #245 and
merge after a green run and review.
- Re-run #351 on GitHub-hosted runners.
- Do not change #216.
- Confirm every PR is merged, superseded and closed, or ignored with a recorded
reason.
    💡 Iterate on the plan with: @coderabbitai <feedback> 
    Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For design choice 1 go ahead with option 3 and replan.

    Start Coding →

  5. leynos commented on Oct 24, 2026

    @leynos
    leynos
    on Oct 24, 2026
    OwnerAuthor

    Independent reproduction of the hard-link cause

    mxd (leynos/mxd#665) reproduced this from the bundle side while preparing its
    0.6.3 migration, and the result matches the evidence in the Coding Plan here.

    • The 18.6.1 Linux asset contains 257 tar hard links, including
      share/timezone/UTC -> share/timezone/UCT and Etc/UTC -> UCT.
    • postgresql_archive before 0.22.0 has no hard-link branch in its tar
      extractor, so those entries are skipped silently and share/timezone/UTC
      never exists.
    • Extracting 18.6.1 with hard links skipped and starting that tree reproduces
      ERROR: invalid value for parameter "TimeZone": "UTC" verbatim;
      SHOW timezone reports Europe/Copenhagen. The intact tree accepts
      SET TIME ZONE 'UTC'.
    • The August 18.6.0 asset was created and last updated on 2026-08-15, so it
      was not replaced in the October wave. It was built
      --with-system-tzdata=/usr/share/zoneinfo, ships no share/timezone, has no
      hard links, and accepts the setting.
    • Onset at mxd: the 18.6.1 asset was published 2026-10-18 and mxd's failures
      attributed to this cause start 2026-10-19. mxd's 2026-10-14 and 2026-10-16
      coverage failures are codescene_uploader_contract and are not evidence for
      this cause.

    One correction to the interim-fix framing, for whoever implements it: an exact
    pin is required, not a range. PG_VERSION_REQ is parsed straight into a
    VersionReq and release selection takes the highest matching semver across all
    pages, so ^18.6.0 or ~18.6.0 would still resolve to 18.6.1. mxd is using
    PG_VERSION_REQ="=18.6.0" and it makes the PostgreSQL lane pass on the current
    0.5.2 harness.

    Also worth recording on the MSRV question: the fix is not reachable from any
    published release even ignoring MSRV. pg-embed-setup-unpriv 0.6.3 requires
    postgresql_embedded 0.20.2, which requires postgresql_archive ^0.20.4,
    and no 0.20.x release carries the hard-link extraction. Reaching 0.22.0
    needs both a postgresql_embedded bump and the Rust 1.94 floor.

  6. leynos
    added 5 commits that reference this issue on Oct 26, 2026

    Repair CI and retain Rust 1.92 (#358)

    6c8ecf5

    Address review of dependency queue fixes (#358)

    ee0f7c5

    Simplify review test scenarios (#358)

    3e565d1

    Document internal setup contracts (#358)

    c4c1902

    Clarify and verify version selection (#358)

    b102a9f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requestmediumRoadmap items to schedule within the current quarter. Clear scope, normal review cycles.testingTest coverage, test infrastructure, and verification tooling work.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions