Hosted boot canaries fail at random on unrelated PRs (claude fresh seat, opencode idle minute)

Two required-gate failures on hosted Linux in the same hour, on PRs that do not touch the boot or delivery paths, both in boot_canaries:

PR fix(ci): discover Bash before isolating Cargo tool test PATH #2718 (CI script test only), run 38502169301, attempt 1: boot_canaries::claude::a_fresh_seat_boots failed at crates/st3/tests/boot_canaries.rs:165 (missing or non-passing result.json, 13.3 s, empty stdout/stderr). The uploaded artifact had only terminal and trace files, no outbox summary, driver or daemon logs. A rerun of that job was authorized to see whether it reproduces.
PR Defer loop until scripts during preflight checks #2534 (rebased CLI change), run 38502646178, shard 2 job 115655543508: boot_canaries::opencode_idle_minute_makes_no_hold_calls_then_delivers_with_hold_release_and_expiry failed with "released did not reach consumption/read" (boot 9.2 s, idle 60.015 s, hold calls 8 before and 8 after, test 103.7 s). Other messaging, roster and sync tests in the same shard failed once and recovered on the existing retry.

Both look like load or timing sensitivity on shared runners rather than a defect in either PR, but that is not proved. Neither failure is waived; both stay adverse on their PRs. Needs: a causal classification (runner load, a real race, or a missing artifact upload), a decision whether these tests need a longer or condition-based wait, and a rule for how many retries a required gate may spend on them.

Evidence is kept in the fleet documents of the two builders. Priority: medium (it blocks required gates at random).

Third data point, from main's own merge group: run 38498736502 (queue entry for #2651, 02:00Z) failed linux-tests-shard-4 on st3::integration daemon_restart::claude_mcp_initializes_offline_then_attaches_and_reconnects_without_restarting (31.1 s, panicked at crates/st3/tests/daemon_restart.rs:2521), which cost that entry its place. That is a third, different harness-driven integration test failing on timing in a few hours, on three different changes. Intake's request (no blind rerun; evidence first): (1) the canary evidence directory for each failure, (2) whether these tests fail on main's scheduled and merge-queue runs over the last days (this one does), (3) the harness version and wait budget. Rerun policy until classified: a rerun of the failed job on the same head is allowed once, recorded as a rerun after an unexplained required failure, the first failure stays in the record, and a second failure at the same head is a stop.

Triage (curator): owner is mission run fleet/smalltalk/canary-diagnostics (builder on hetz2): step 1 make failures diagnosable (daemon and driver logs and outbox summary in the uploaded evidence directory, also on interrupted runs; the claude canary artifact had only terminal and trace files), step 2 condition-based waits where a fixed wait ran out. No retry loops in tests. Retry rule for required gates is a CI policy decision for cos.

Concrete evidence from the opencode idle-hold failure (run 38502646178, shard 2 artifact linux-tests-shard-2-logs, evidence directory target/boot-canaries/opencode-idle-hold-*/evidence), read by the cli-papercuts builder: the hold release was durably recorded (delivery.hold held=false until=0 at 1791947136959). The daemon then persisted harness.observed state=working, quiescent=false, blocking=[turn-in-flight] at 1791947137351 (observation timestamp 1791947132206, i.e. 5 s older than the persist). The provider receipts show the after-idle read completed at 1791947132247 but no held-message turn or read before the canary shut down. observation-outboxes.json has two pending harness-state rows and a latest local state of idle for the same driver incarnation. So the release is durable and an idle observation is pending: the delivery was gated on a stale working observation. The daemon log records /v1/messages 2317 ms, receipt 2513 ms, read-claim 3760 ms and WAL passive 3074 ms / truncate 213 ms, so the runner was slow. Open question for the canary-diagnostics mission: is a stale working observation allowed to block delivery after a hold release while the idle observation sits in the outbox (a real ordering race) or is this only slowness?

Another hosted data point, recorded for the rerun count: PR #2546 at 2233ae8, upgrade-impact (run 38524168533, job 115720482873) failed in a test's tempfile cleanup (OSError Errno 39 Directory not empty: .git), not an assertion. One recorded rerun issued by the curator.

Correction: the rerun noted above for #2546 was not issued yet (the run is in progress and cannot be rerun). I will record it when it is issued.

Another hosted failure for the count (different test, same class: timing-sensitive daemon-process fixture): PR #2699 at 64e7c92, linux-gate job 115716170681 (run 38520768509): st3::integration operation_audit::a_consistent_conflict_written_during_the_audit_is_never_reported failed all three attempts at crates/st3/tests/operation_audit.rs:577 (assertion 'observations > at': the mutation ran after the last snapshot). The PR's own diff does not touch the operation audit. Recorded here; one rerun of the failed job on that head is allowed by cos's rule and the first failure stays on record.

Recorded rerun (CI retry rule), PR #2648 head e67f4c82, run 38536455124, attempt 1.

First failure, kept as evidence: linux-test-build ended with "The operation was canceled", KeyboardInterrupt, exit 130, at about 08:41Z, when the runner got a shutdown signal. isolation-vm, linux-gate, linux-tests, linux-tests-shard-2 and mail-redelivery-canaries failed behind it because the shared build was missing. genie-freshness, typescript-client and clippy passed. This is an interruption of the hosted build, not a code fault. One rerun of the failed jobs started; a second failure at this head stops here.

Recorded rerun (CI retry rule), PR #2755 head 310bc6e0, run 38536629825, attempt 1.

First failure, kept as evidence: mail-redelivery-canaries failed st3::integration boot_canaries::claude_missing_channel_blocks_and_recovers_early_mail_without_replay (258 s, verdict fail, harness claude, panic at crates/st3/tests/boot_canaries.rs:165) (job). The PR changes iOS, generated clients and mission views, none of the boot canary path, so I treat it as an unexplained hosted failure. One rerun of the failed jobs started; a second failure at this head stops here and goes back to the author.

Diagnostics-only PR #2729 merged at d4db5347a263140b1ab40169c89bc380c44e37d6 from reviewed head 1076da17f. The process guardian now retains daemon/driver logs, outbox summaries, capture errors and atomic verdicts after interrupted boot-canary and daemon-process tests. Pre-shutdown and final outbox state are preserved separately, and the test requires the runner's actual exit status.

The original waits and product assertions are retained, per cos's split decision. All five required checks passed at this head, seven evidence regressions passed, and the three restored-wait cases passed locally with zero retries. The separate perf-cost failure remains adverse and unwaived, with the tracked #2733 usage-route signature.

The original Claude artifact shows collection stopped after terminal/trace capture; why its runner stopped remains unclassified. The OpenCode artifact is consistent with delayed ordered publication, but lacks acknowledgement/in-flight timing needed to establish its cause; no product race is established or fixed here. The longer-wait changes remain local only. Before proposing them, a new failure with the retained diagnostics must establish that a wait expired, and we must first investigate the owned API writes taking 2.3–3.8 seconds. This issue remains open for that diagnosis.

Recorded rerun (CI retry rule), PR #2466 head 19cdde65, run 38323177518 (2026-10-12/13), attempt 1.

First failure, kept as evidence: this older run at the same head failed linux-gate and cancelled isolation-vm and mail-redelivery-canaries. A later run, 38417193199, at the same head passed all five required checks (intake's GO rests on it). My queue helper refuses to queue while a failed entry of the same name remains in the rollup, so I rerun the failed jobs of the older run once so the rollup reads green. This is not a claim about the cause of the first failure. A second failure at this head stops here and goes back to the author.

Two first-attempt failures in required lanes on PR #2390 at head 644e56bf (run 38542343696), both passed on an inherited retry in the same run. Recorded here with their evidence so they count toward the flake record; neither is waived.

Roster summary: a roster-summary test failed published_at.is_some() after the roster published, in job 115792752209. The source it exercises is 0ec141e2d ("Serve a waiting summary as soon as the complete roster publishes"). Raw log 1,066,569 bytes, SHA256 34932e5fa5643e784d8db78dc60dd1e48cf16039f76316ee44400dd2365f744b.
messaging_faults::link_seconds in shard 2 (job 115792752036): after a 3,053 ms outage, the first request completed 24,069 ms after clear and the read 597 ms later (24,666 ms total, limit 10,000 ms); the sender still sent, with one copy and one graph read and the same provider and incarnation. Raw log 1,061,245 bytes, SHA256 34112dfd37f65e6b9a9c7aebe4d2e0146fa71ef2c3300f7294755078b3863cac. This is the link_seconds sibling of the quarantined link_minutes case in Exit issue for the link_minutes quarantine (#2101): message unread 25 s after a 120 s link fault; evidence dir not uploaded #2122, in a required lane without Exit issue for the link_minutes quarantine (#2101): message unread 25 s after a 120 s link fault; evidence dir not uploaded #2122's optional status.

#2390 changes signature judging and storage, not roster summaries or message delivery timing, so these look unrelated by path; that is not proven. Cause ownership: the roster-summary one belongs to the author of 0ec141e2d; link_seconds belongs with #2122. A second failure of the same test at the same head would stop here and go to its owner.

Row for #2467 (head 05830de2, run 38321969514): linux-gate red from st3-feed tests::sidebar_arrangements_follow_the_explicit_person_and_remote_register_updates, which panicked at lib.rs:1514 with database table is locked: batches. The in-run retry passed and the job was then cancelled, so this counts as the one recorded rerun for that head. The PR changes only a web test and a release note, so it cannot cause it. Intake classifies it as adverse and unexplained, not waived: a SQLite table lock in a store test is a product signal and matches none of the earlier rows here. If it fails on the same test again at the new head c569a416, that is the second failure, the rerun rule stops, and the test must be explained before an exact-head GO.

Baseline row for #2785 (head 38e0e39b, Nix release job 115819580853). Compared with main's scheduled Nix run 38520416703: 28 failures that main has no longer fail at this head, including both git_evidence tests that failed on the missing openssh. One name is in this head's list but not in main's: setup::uninstall_stops_an_instance_daemon_that_runs_without_a_service. It fails at setup.rs:107 with st daemon exited with exit status: 2, the same helper as the other login-shell setup tests, which #2790 is adding to the exclusion list as its 14th. The remaining failures are the login-shell setup::* cases, the 12 otel_export::*, reconcile::tests::start_preemption::native_late_pty_publication_is_retired_by_exact_operation_and_incarnation, and one sandbox-network case: git_evidence::protected_ci_cli_fetches_selectors_and_exports_verified_objects_without_a_key, which fails at line 117 with GitHub observation unavailable because the Nix sandbox has no network and no gh. All of these are adverse and unwaived, none is caused by #2785, and none is a required check.

Row for #2452 (ready head ea251fcc, run 38561739796, shard-2 job 115859801236): otel_export::daemon_request_id_and_roster_stages_match_response failed on its first nextest try (fresh_requested=true, requested_cut 3, publication_cut 2, response 200, 13:59:19Z) and passed on the in-run retry at 13:59:50Z. Both Linux jobs then concluded SUCCESS. Nothing in #2452's diff points at roster freshness, and nobody has shown whether the test is flaky on main. Recorded as adverse and unwaived: a request-id/roster-stage mismatch is a product signal, not a runner problem. A second occurrence on any head stops the retry rule for this test. The immutable raw packet is doc/fleet/factory/changes-in-git/mutation/ea251-roster-freshness-adverse@afe9c5c8…; the factory owner has a repair request open (message/cff6772681b310d9).

Row for #2540 (head 71ddd98a, run 38572773993, isolation-vm job 115905568309): failed with RuntimeError: exact producer artifact transfer failed after two bounded attempts (the artifact ZIP request failed twice, "private partial input discarded; no tests started"). No test ran, so it says nothing about the PR; it is an unexplained hosted failure in a path the PR does not touch. Under the retry rule I will rerun the failed jobs once for this head as soon as the run finishes (the canary is still queued, so the run cannot be rerun yet), and record the result here. A second failure on this head stops the rule.

Recorded timing flake on #3670 head f74fb17, natural Workspace CI run 39859157536, linux-tests-shard-2 job 120107767753.

fractal-api-core api::client_messages_selector_tests::client_messages_selector_desired_delete_seeks_only_subject_and_affected_reminder_group failed TRY 1 (2.433 s) and TRY 2 (11.894 s), then passed TRY 3 (2.770 s). Both failures were at client_messages_selector_tests.rs:189: “selector maintenance transaction exceeded 100ms (not page p95)”. The longest transactions including COMMIT were 149.771837 ms and 180.386589 ms; work portions were 1.302413 ms and 0.323217 ms, respectively. The log includes scheduler and fsync time; this record does not establish their cause.

Shard summary: 4375 run, 4375 passed, 1 flaky; nextest flaky: 1. The PR does not edit this selector test or implementation. No waiver or failed-jobs rerun is requested or implied; a recurrence on the next natural run needs classification.

Separately, this run is red on one deterministic schedules-fixture failure caused by composition with the merged standby grammar; that is not this flake.

Flake ledger, 2026-10-23: api::mailbox::tests::the_pushed_value_is_what_the_poll_answers (fractal-api-mailbox) failed TRY 1 (0.137 s) and passed on retry in linux-tests shard 2, job 120282008789, on #3681 at 877de51. Reported by Intake A; Factory accepted it for that head (the PR changes only scripts/st-test, its test, doc and release note, no Rust). No waiver for any PR that touches Rust.

Flake ledger, 2026-10-23 (reported by Intake B, after the merge): fractal-api-client-v0 api::client_v0::tests::managed_codex_session_renders_its_exact_native_chat_not_only_status_and_timeline_send_identity failed TRY 1 and passed TRY 2 in linux-tests shard 2 on #3682 at 5d018d9 (artifact nextest-flaky-linux-tests-shard-2-1). #3682 had already merged when the report arrived (changes only scripts/st-test, its test, doc and note, no Rust); no disposition was recorded before the merge.

Flake ledger, 2026-10-23 (Factory): (1) fractal-api-client-v0 api::client_v0::tests::managed_codex_session_renders_its_exact_native_chat_not_only_status_and_timeline_send_identity: TRY 1 fail, TRY 2 pass, linux-tests-shard-2, PR #3682's own run (artifact nextest-flaky-linux-tests-shard-2-1). Accepted after the fact: #3682 changes only scripts/st-test; its merge group on main passed and it merged at 06:28Z. (2) fractal-api-mailbox api::mailbox::tests::the_pushed_value_is_what_the_poll_answers: TRY 1 fail, TRY 2 pass, shard 2, PR #3681 (job 120282008789); accepted, scripts-only PR. (3) action_coverage::cli_agent_and_shell_declarations_survive_restart: 'agent was created, but was not ready after 15s' in #3642's first merge group (run 39878344640); passed on the second group without changes. No waiver for any PR that touches Rust.

Flake ledger (Factory, 2026-10-23): fractal-reconcile reconcile::tests::one_cancellation_wake_converges_through_runtime_cleanup, TRY 1 fail (2.2 s), TRY 2 pass, shard 2, PR #3685 (job 120291507929). Accepted for the scripts/policy-only PR; no waiver for any PR touching Rust.

Rerun (failed jobs only) of run 39965662851, PR #3696 at 87de08a: the required isolation-vm job 120460191230 failed with no failed step. Annotation: "The self-hosted runner lost communication with the server." Runner loss, not a test failure. Rerun approved by the reviewer of record.

Intake B, #3702 at 6643b1d: linux-tests-shard-2 flaky 1 (smallclaims store::projection_stage_tests::each_point_inside_the_loan_is_charged_to_its_own_stage, TRY 1 FAIL 9.759s, TRY 2 PASS 8.132s). Factory accepted it as unrelated (the PR changes only scripts/ci-local); FINAL GO sent to the curator with that note.

Intake A rerun (failed jobs only) of run 39972753956, PR #3703 at ea766a8: required mail-redelivery-canaries job 120481254387 failed in ci-test-archive, the exact producer artifact transfer exceeded its 600 s deadline twice (409 MB of 1.8 GB, no tests started); linux-gate failed only as the aggregate. Hosted transfer stall, not the PR. First rerun at this head.

Accepted flake (Factory), seen on #3730 @02e196b24e linux-tests: fractal-delivery store::writer_bound_tests::a_catch_up_exchange_holds_the_writer_one_page_at_a_time, TRY 1 FAIL 25.9s, TRY 2 PASS 49.2s. Unrelated to that PR (test-only tests/*.rs helpers).

Intake A rerun (failed jobs only) of run 40026069295, PR #3731 at cf35bbf: linux-tests-shard-2, mail-redelivery-canaries and isolation-vm (and linux-gate as the aggregate) failed in the test-archive download, the exact producer artifact transfer exceeded its deadline twice, no tests started. Hosted transfer stall, not the PR (test-only change). First rerun at this head.

Known flake again (Factory-accepted), #3732 @3ffb6b025b linux-tests-shard-2: smallclaims projection_stage_tests::each_point_inside_the_loan_is_charged_to_its_own_stage, TRY 1 FAIL 7.3s, TRY 2 PASS 8.5s. Same as #3702.

Intake A (R3, test-only PR), #3831 at 194c500: linux-tests TRY 1 FAIL then FLAKY 2/3 for fractal-api-mailbox api::mailbox::tests::the_pushed_value_is_what_the_poll_answers (0.088 s, then 0.096 s). Recorded as unrelated: the PR changes only scripts/st-test-test and a release note, and it is the already-known SQLITE_LOCKED_SHAREDCACHE flake filed as #3758. Shard 2 zero flaky.

Intake A (R3, test-only PR), #3921 at 573e4fa: linux-tests TRY 1 FAIL (61.4 s, panicked at crates/fractal/tests/broken_gates.rs:233 'timed out waiting for the terminal exec diagnostic') then FLAKY 2/3 (2.4 s) for fractal::integration broken_gates::missions_and_doctor_surface_a_terminal_field_gate_behind_a_pending_gate. Recorded as unrelated: the PR changes only two fractal-store test files, a large-value inventory line and a release note, and the test passed on retry in 2.4 s after a wait timeout. Shard 2 zero flaky.

Flake on smalltalk#2551 at 014444b (linux-tests shard 1, 2026-10-26 23:17Z): fractal::integration ui_open::an_alert_a_seat_raises_for_itself_shows_above_the_input_and_is_dismissed_there, TRY 1 FAIL, FLAKY 2/3, passed on the third attempt. The PR does not touch ui_open or the attention path; the ui_open fixture fixes ec79d20, 77cbd4d and 067234f are already in that head, so the flake outlived them. Factory accepted it as unrelated at that head. No rerun used. (Intake A)

Flake on smalltalk#3022 at 9e0aa3c (linux-tests shard 1, 2026-10-27 03:12Z): fractal::integration revision_seats::isolated_revision_preserves_completed_builder_declarations_and_cleanup, TRY 1 FAIL, FLAKY 2/3, passed on the third attempt. The PR touches no revision or seats file. Factory accepted it as unrelated at that head. No rerun used. (Intake A)

Navigation Menu

Repository navigation

Hosted boot canaries fail at random on unrelated PRs (claude fresh seat, opencode idle minute) #2725

145 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Hosted boot canaries fail at random on unrelated PRs (claude fresh seat, opencode idle minute) #2725

Description

Activity

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

myobie commented on Oct 14, 2026

145 remaining items

myobie commented on Oct 22, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 23, 2026

myobie commented on Oct 25, 2026

myobie commented on Oct 26, 2026

myobie commented on Oct 26, 2026

myobie commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions