release-linux: sccache sees zero compilations and manual runs are cache READ_ONLY — no build-time win from #343 #392

Activity

1. ik0zy commented on Oct 9, 2026

   

   ik0zy

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Update from the Oct 8 run (37785742972, 0.4.1 + #366 + #375):

   Step timings (kills one hypothesis): the 44-minute step is the Tauri build itself (Finished release profile in 42m00s, ~1210 Compiling units — a full cold compile). The Arch Docker step took only 33s, so it repackages rather than compiles; the runner-side build is where all the time goes.

   rust-cache also saves nothing on manual runs. The restore step took 2s (miss — hence the cold compile), and save-if is gated on the same READ_WRITE flag, which is false for branch dispatches. So on a manual branch run, neither cache layer persists anything.

   sccache bypass, narrowed further. Verified from the repo side that the env chain looks intact: workflow-level RUSTC_WRAPPER: sccache, scripts/with-posthog-env.mjs spreads process.env into the tauri build child, and .cargo/config.toml has only an [env] section (no build.rustc-wrapper to take precedence). Yet end-of-job stats are all zeros. What still needs a runner-side probe: echo "RUSTC_WRAPPER=$RUSTC_WRAPPER"; which sccache immediately before the build step, to see whether the variable survives the bun run → node → tauri CLI → cargo chain on the runner (works when cargo is invoked directly, per the CI Windows job's 88% hit rate, so the suspect is something in that chain).
2. ik0zy commented on Oct 9, 2026

   

   ik0zy

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Further findings from deeper investigation (all verified, none fix it yet):

   1. Tauri CLI is exonerated. Read the actual 2.11 source (crates/tauri-cli/src/interface/rust/desktop.rs, cargo_command): plain Command::new("cargo"), no env_clear(). Environment — including RUSTC_WRAPPER — is inherited by cargo. Combined with the earlier checks (workflow env is additive, with-posthog-env.mjs spreads process.env, no build.rustc-wrapper in .cargo/config.toml), every link of the env chain is confirmed clean.

   2. The stats are trustworthy. Read the action source (show_stats.ts in sccache-action v0.0.11): the post step only runs --show-stats, it never stops/restarts the server. And CI's Windows job reports real numbers with the same action version. All-zero end stats mean the queried server genuinely recorded zero requests.

   3. The cache service tells a stranger story. gh cache list on the fork shows the Oct 7 release run (37567733819) did write ~50MB of sccache/* objects (03:40:08–03:42:10) and read older ones — then went silent for the remaining 40 minutes, with end stats at zero. The Oct 8 run wrote nothing at all (only the setup connectivity ping). So on Oct 7 sccache served briefly and then stopped being used mid-build, with the stats lost along the way; on Oct 8 it never engaged. No CI run overlaps the Oct 7 window, so those writes came from the release build itself.

   4. SCCACHE_GHA_RW_MODE is honored by the sccache binary (verified in sccache's own config.rs: read-only GHA backend skips writes but still counts requests). So READ_ONLY explains no-saving, never zero-requests.

   Net: the wrapper is bypassed or abandoned in the release job specifically — direct-cargo CI works, the tauri chain is clean by source inspection, yet the release build compiles 1200+ units with zero sccache contact (Oct 8) or a few minutes of contact then silence (Oct 7). Prime suspect is now the sccache server dying mid-build (OOM killer is a candidate — the workflow adds swap for exactly that) with stats dying alongside it.

   Blocked next step: I wrote a ~10-minute throwaway probe workflow (direct-cargo control + fake-cargo env interception through the real tauri chain) to settle it, but all manual dispatches on the fork currently fail with 422: Actions has been disabled for this repository despite settings showing enabled — being resolved separately. Will run the probe once dispatch works again.
3. ik0zy commented on Oct 9, 2026

   

   ik0zy

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Decisive local probe result — the env chain is exonerated empirically, not just by source inspection.

   Ran the real chain locally with a fake cargo first on PATH (no compilation involved — it logs and exits 1, so the build fails fast before compiling anything):

   RUSTC_WRAPPER=probe-sentinel-value node scripts/with-posthog-env.mjs tauri build --bundles deb
   

   The fake cargo, spawned by the real Tauri CLI 2.11.4 through the real wrapper script, logged:

   FAKECARGO RUSTC_WRAPPER=probe-sentinel-value
   

   So RUSTC_WRAPPER provably reaches cargo through bun run → node → tauri → cargo. (Repo files verified unmutated after the probe.) Combined with the earlier verifications — Tauri CLI source (Command::new("cargo"), no env_clear), no build.rustc-wrapper in .cargo/config.toml, valid stats mechanism — cargo in the release job had the wrapper and honors it, yet 1200+ units compiled with zero sccache contact on Oct 8.

   That inverts the suspect ranking: the drop is not in env delivery. What fits all facts (Oct 7: service 03:40–03:42 then silence + zeroed stats; Oct 8: never engaged + zeroed stats; builds succeed throughout) is sccache server loss mid-build with the post-step querying a fresh/restarted server — the one piece I could not verify remotely. Still open: whether clients silently fall back to direct compile in that window (check SCCACHE_IGNORE_SERVER_IO_ERROR semantics in 0.18), and what kills the server (OOM killer is a candidate — the workflow adds swap for exactly that reason; needs runner-side evidence such as the server PID surviving the build or dmesg).

   The throwaway GHA probe workflow for that half is written and ready on ik0zy/probe-sccache, but all dispatches on the fork currently 422 (Actions reports enabled everywhere — tracked separately).
4. ik0zy commented on Oct 9, 2026

   

   ik0zy

   on Oct 9, 2026

   ContributorAuthor

   More actions

   Root-cause class confirmed via upstream precedent — and it corrects something I claimed earlier.

   Upstream #2069 (mozilla/sccache) is our exact symptom: server shuts down mid-build → warning sccache: warning: The server looks like it shut down unexpectedly, compiling locally instead → build succeeds slowly → post-build --show-stats reports all zeros (their paste matches ours field for field). Key point: this fallback needs no opt-in — my earlier claim that silent fallback requires SCCACHE_IGNORE_SERVER_IO_ERROR=1 was incomplete. That var governs connection-failure behavior (where the default is indeed fail-the-build, per docs.rs); a server that was reachable and then exited takes the graceful local-compile path by default.

   Maintainer (Xuanwo) diagnosed that case as the 600s default idle timeout (slow task > 10 min with no requests → server exits) and the reporter confirmed SCCACHE_IDLE_TIMEOUT=0 fixed it with stats reporting correctly afterwards (4911 requests, 4428 hits). Related: #1025 confirms the OOM killer as another server-killer in practice.

   Consistent story for our runs now:

   • Oct 8: server gone before/during early build → all 1210 units local → 42 min success → zero stats. (Setup ping only proves backend credentials, not a live server; the action never pre-starts one — verified in its source.)
   • Oct 7: served ~2 min (the 03:40 writes/reads), server died, rest local. Different code order between the two runs plausibly moves when the killer strikes, which is why the timing differs.
   • Env chain was never the problem (proven end-to-end with the fake-cargo probe).

   Fixes, in order of leverage:

   1. SCCACHE_IDLE_TIMEOUT=0 in release-linux.yml — eliminates the idle-shutdown path entirely (upstream-verified fix for this symptom class).
   2. OOM-proofing — the workflow already adds swap for OOM; if the killer is memory, consider fewer parallel units (CARGO_BUILD_JOBS) or a larger runner. Needs runner evidence (dmesg / SCCACHE_ERROR_LOG) to confirm OOM vs crash.
   3. Observability for recurrence: SCCACHE_ERROR_LOG in the release job so the next incident leaves traces instead of zeros.
   4. Optionally bump sccache past 0.18.0 if a newer release fixes server-death bugs.

   Unchanged and still required: even with engagement fixed, manual branch runs persist nothing (READ_ONLY + save-if: false), so cross-run reuse needs the policy decision too. The two layers are independent.

   Still open (needs a runner — blocked on the fork dispatch 422): which killer struck our runs, and a measured post-fix duration to replace #343's expectation with data.
5. ik0zy commented on Oct 11, 2026

   

   ik0zy

   on Oct 11, 2026

   ContributorAuthor

   More actions

   Decisive probe results — the env question is now settled on the runner itself, and one sub-mystery is fully solved.

   1. Env reaches cargo on the runner (proven). Probe 2 now runs correctly (--bundles deb): a fake cargo first on PATH, invoked through the real node scripts/with-posthog-env.mjs tauri build chain on ubuntu-latest, logged FAKECARGO RUSTC_WRAPPER=sccache. (Tauri then fails at cargo metadata as designed — fail-fast, no compilation.) Combined with the identical local result, env delivery is proven twice. The env-drop theory is dead.

   2. Solved: what the probe's 6 non-cacheable calls are. Full stats gave the reason map: "missing input": 4, "crate-type": 1, "-": 1. Against the 0.18.0 source: missing input = cargo's probe invocations with no input file; crate-type = the hello-world bin crate — sccache declines any bin target without an explicit --crate-type flag (rust.rs: inferred binaries are refused as 'we don't know what to do'). Cargo only passes --crate-type explicitly for lib targets. So sccache 0.18.0 never caches bin-crate compilations by design. In this workspace (19 lib targets + the final atlas bin), that costs the final binary compile, not the bulk — it does NOT explain the 1210-unit zero.

   3. Remaining, narrowed to one suspect. Release cargo provably has the wrapper, honors it, yet records zero contact while succeeding slowly — the only story left fitting all evidence is sccache server loss with the documented graceful local fallback (upstream #2069: identical zero-stats + slow-success signature; #1025: OOM killer as a confirmed server-killer). The exact killer on these runs (OOM vs crash) still needs runner evidence: SCCACHE_ERROR_LOG + server liveness in a release run.

   Suggested fix stack (unchanged, now with proof for each layer): SCCACHE_IDLE_TIMEOUT=0 (kills the idle-shutdown path), OOM-proofing, SCCACHE_ERROR_LOG for observability, plus the independent policy decision on manual-run cache writes (READ_ONLY/save-if), without which cross-run reuse stays impossible regardless.
6. ik0zy commented on Oct 11, 2026

   

   ik0zy

   on Oct 11, 2026

   ContributorAuthor

   More actions

   Diagnostic build results — mechanism confirmed, and it flips part of the story. Run 38136672033 (test-sccache-diag tag, 0.4.1 + watchdog + SCCACHE_ERROR_LOG):

   The server survived the whole build. Watchdog logged PID 4740 alive every 30s from 11:46 to past 12:16, with client PIDs churning beside it. No OOM lines in dmesg, error log empty.

   And sccache worked normally: 2214 requests, 1936 executed, 1927 misses (1358 Rust), 0 hits. Cold branch + READ_ONLY, so all misses and a 35-min build — but fully engaged, counting everything. Same workflow file (modulo the diagnostic additions), same action version, same env as the Oct 7/8 runs that recorded zero.

   So: nothing in the repo, workflow, env chain, or Tauri CLI prevents sccache from working — when the server stays alive, it serves. The Oct 7/8 zeros therefore mean the server was gone before/during those builds (Oct 7: died ~2 min in after serving; Oct 8: gone before first compile), with clients falling back to local compile per the documented #2069 behavior. The killer varies run to run (OOM prime suspect given the swap workaround already in this workflow); idle timeout is ruled out for Oct 8 (31s to first compile).

   What this changes about the fix stack:

   • Server survivability is now the confirmed crux: SCCACHE_IDLE_TIMEOUT=0 (belt and braces), OOM-proofing (swap exists; consider CARGO_BUILD_JOBS cap), and SCCACHE_ERROR_LOG permanently so the next incident is observable instead of zeros.
   • Independent and still required: writable cache for manual runs (READ_ONLY/save-if) — this run proves even a healthy server yields zero reuse on a cold branch.
   • Bonus data point: healthy-server cold build took 35 min vs 42–47 min for the disengaged ones — but different code each time, so treat as variance, not a measurement.