Track restoring real rust-runtime TSan coverage in nightly sanitizers

[slepp]

Summary

PR #765 made rust-runtime-tsan advisory because we do not currently have a clean repo-side way to run ThreadSanitizer against hew-runtime. We should restore real blocking TSan coverage once the underlying Rust/Cargo sanitizer + -Zbuild-std debt is resolved.

Current state

• rust-runtime-tsan in .github/workflows/nightly-sanitizers.yml is now advisory/non-blocking.
• docs/release-runbook.md documents the waiver and the re-evaluation trigger.
• This is not currently treated as a v0.3.0 release blocker.

What we learned

• Nightly run 23996664465 fails because cargo +nightly test with -Zsanitizer=thread is effectively building against an unsanitized/prebuilt sysroot.
• Without -Zbuild-std, we hit mixed-instrumentation / ABI-mismatch failures against core / compiler_builtins.
• With -Zbuild-std, current toolchains still hit duplicate core / alloc lang-item failures and panic-strategy issues.
• -Cunsafe-allow-abi-mismatch=sanitizer is not an acceptable workaround because it suppresses the safety guard and still leaves the sysroot story incorrect.

Desired outcome

• Re-enable a real rust-runtime-tsan lane without continue-on-error.
• Do it without unsafe ABI-mismatch flags.
• Remove the waiver from the workflow and release docs when the toolchain path is viable again.

Acceptance criteria

• A clean end-to-end rust-runtime-tsan run exists in .github/workflows/nightly-sanitizers.yml.
• The advisory wording / continue-on-error can be removed.
• docs/release-runbook.md is updated to drop the waiver.
• Any required upstream issue links or toolchain pinning are documented in the workflow comment or this issue.

[slepp]

Triage (2026-04-16)

Current state. The advisory wording from #765 is still in place and matches the issue body exactly. The rust-runtime-tsan job in .github/workflows/nightly-sanitizers.yml:150-153 still carries continue-on-error: true and the (advisory) name suffix, and the waiver/re-evaluation rationale is preserved in the surrounding comment at .github/workflows/nightly-sanitizers.yml:140-148. The release checklist still flags TSan as advisory in docs/release-runbook.md:10 and the known-gaps table at docs/release-runbook.md:126 / :138-139. No follow-up PR has touched the lane since #765 merged, and there are no new upstream toolchain signals recorded here.

Landed / in flight / pending.

• Landed: advisory marker, release-runbook waiver, root-cause note pointing at -Zsanitizer=thread + -Zbuild-std ABI/lang-item failures (all in ci(tsan): mark rust-runtime-tsan advisory; document upstream waiver #765).
• In flight: nothing — this issue is a pure tracker for the waiver.
• Pending: all three acceptance criteria (clean end-to-end run without continue-on-error, no -Cunsafe-allow-abi-mismatch=sanitizer, runbook waiver removed). Resolution depends on upstream Rust toolchain work, not Hew-side changes.

Recommendation: keep open, assign to v0.3.0 milestone. The issue is correctly scoped, recent, and still represents real deferred safety coverage we want to restore. It is explicitly not a release blocker per its own body, but it belongs on the v0.3.0 milestone as a tracked waiver so it shows up in the release review rather than drifting as an unmilestoned orphan. Closing it would lose the only durable pointer to the waiver's re-evaluation trigger.

Proposed next concrete capability. Add a lightweight upstream-watch step: capture a dated note in this issue (or link to a LESSONS.md row) each time the nightly TSan lane is reviewed, recording the nightly toolchain version tried and the specific failure mode. That keeps the waiver honest and gives us a trigger to attempt -Zbuild-std again when upstream lang-item/panic-strategy behaviour changes. Concretely: re-run cargo +nightly test -p hew-runtime --no-default-features --lib -- --test-threads=1 with RUSTFLAGS="-Zsanitizer=thread -Zbuild-std" on each quarterly release prep and paste the failure (or success) tail here.

Decision: keep open; suggest milestone v0.3.0.

[slepp]

Augmenting current state for orchestration scheduling

Bringing this issue up to current state so it can be picked up as a scoped lane.

Current state confirmed against main

• Workflow waiver still in place — .github/workflows/nightly-sanitizers.yml:140-150 keeps rust-runtime-tsan as continue-on-error: true, with the comment naming the upstream blocker as of 2026-04.
• Release-runbook waiver still in place — docs/release-runbook.md:126,138 lists TSan as Advisory (waived) in the gate matrix and in "Known gaps".
• No rust-toolchain pin — the workflow uses dtolnay/rust-toolchain@nightly, so any upstream fix in nightly should reach the lane automatically once it lands.
• No alternative concurrency-testing infrastructure — neither Loom nor Shuttle is in hew-runtime/Cargo.toml or anywhere under hew-runtime/src/. TSan is the only data-race signal the project has today.

Open questions to resolve before execution

1. Has the upstream blocker shifted? The waiver cites duplicate lang items / panic-strategy mismatch under -Zsanitizer=thread + -Zbuild-std. A periodic check on the matching rust-lang/rust and rust-lang/cargo issues — and a quick re-attempt on the latest nightly — would tell us whether the workaround still requires upstream movement or whether the toolchain has caught up. Decision: schedule a periodic re-evaluation cadence (quarterly?) or wait for an upstream issue subscription to fire?
2. Is a complementary path worthwhile? Loom or Shuttle could exercise the actor scheduler / mailbox / reply-channel concurrency surface deterministically, model-checker-style, even while TSan is waived. They don't replace TSan (different cost/coverage), but they would close the "we have no automated data-race signal at all" gap. Decision: scope a Loom/Shuttle adoption as a sibling lane (separate scope from the TSan restoration), or keep TSan as the only path?
3. Can we narrow TSan coverage to a subset that builds today? The current scope excludes tokio/quinn/profiler features for noise reasons. There may be a further-narrowed subset (e.g. only the synchronous reply-channel + mailbox tests, no cargo test of the workspace) that builds without -Zbuild-std against a prebuilt sanitizer-instrumented sysroot. Worth a 1-day spike before declaring the upstream blocker truly the only path. Decision: do the narrow-subset spike, or wait for upstream?
4. What's the migration plan when upstream lands? Concretely: (a) re-run the lane on a fresh nightly, (b) confirm clean exit on the affected tests, (c) remove continue-on-error: true, (d) remove the waiver text from release-runbook.md (lines 126 and 138), (e) close this issue. The plan is small but should be documented so the trigger is unambiguous.

Validation candidates

• gh run rerun <latest rust-runtime-tsan run-id> on a fresh nightly — confirms whether upstream has moved.
• cargo +nightly test -p hew-runtime --target x86_64-unknown-linux-gnu -Zsanitizer=thread --no-default-features --lib mailbox — narrowest possible reproduction; if this passes, the lane can be split into a working subset + a still-blocked superset.
• Searching rust-lang/rust issues for Zsanitizer=thread Zbuild-std to identify the live upstream tracker.

Out of scope for this issue

• Adding Loom or Shuttle (separate lane if the project decides to pursue it).
• Restructuring the broader sanitizer story (ASan/LSan/UBSan are separate lanes).
• Changing the release-gate policy on TSan being advisory (this issue only restores blocking when feasible).

Schedulability note

Milestone is currently unset. The user said v0.3.0 has shipped, so attaching this to a future release milestone (e.g. the new "Release pipeline and cross-compilation" milestone, where it fits the "release validation" theme) is one option; leaving it unmilestoned until the upstream blocker resolves is another. User decision.