Skip to content

[Joshua's agent] wifi: centauri T6050 IPC wedge self-recovery (M5) - #2191

Draft
joshuaswarren wants to merge 7 commits into
integration/aurorafrom
wAQ/m5-wifi-transport-recover
Draft

joshuaswarren wants to merge 7 commits into
integration/aurorafrom
wAQ/m5-wifi-transport-recover

Conversation

@joshuaswarren

@joshuaswarren joshuaswarren commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When the firmware IPC stage sticks at 0x3 after a disconnect/channels-update
event, command_fault wedged Wi-Fi until reboot. Map the control-port stage to
the existing centauri_t6050_boot_worker phase chain, tear down IPC/wifi, and
rebuild wlan0.

Additions (per fleet review)

  • centauri_t6050_boot_phase_for_control_stage() with kunit cases (0/1/4, 2/3 stop)
  • Fault log: wifi + control stage, gap_ms since last fw disconnect/channels event
  • debugfs force_transport_fault (write any byte) to prove recovery on a healthy link

Journal gap study (M5, read-only)

Failing boots: event to next cmd 0x1000f is 2.3 to 2.5 ms (wedge boots -42/-44).
Good boots: often 37 to 51 ms after early channels event; some fast paths exist (0.3 to 7 ms).
Wedge-correlated scans arrive soon after the channels event, not after a long quiet window.

M5 test plan (rider for next daily-family kernel)

  1. Normal boot: no recovery log
  2. echo 1 > .../phy0/force_transport_fault -> recovery log -> wlan0 + iwd associated <15s, 3x
  3. Soak unchanged

Head: 388cd88

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-18T09:20:57.758249Z
We couldn’t complete this review. Comment @codex review to try again.
bded4d6 PR opened
🔒 Security Review ❌ Failed 2026-11-18T09:21:05.635087Z
We couldn’t complete this review. Comment @codex security review to try again.
bded4d6 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector

Copy link
Copy Markdown

The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins.

@joshuaswarren

joshuaswarren commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

CHANGES on 148f854
Model: cognition/swe-2-flash

[Joshua's agent]

Two blocking findings:

  1. Self-deadlock on the first transport fault. recover_worker runs as wifi->recover_work -> centauri_t6050_schedule_transport_recover() -> centauri_t6050_free_ipc() -> centauri_t6050_wifi_unregister() -> cancel_work_sync(&wifi->recover_work) — canceling the work item from inside its own handler while holding c->boot_lock. cancel_work_sync waits for the work to finish; it never does, so boot_lock stays held forever and every later boot/recover/remove path wedges. This fires on exactly the fault the PR exists to recover from. Fix: don't cancel_work_sync recover_work from the unregister path it feeds (e.g. let the worker exit before teardown, or have schedule_transport_recover skip the wifi unregister / use cancel_work + drain differently), and re-check the whole recover->unregister->re-register cycle for the same pattern.

  2. Build break: note_fw_event has no forward declaration. It is called inside drain() (lines ~602, ~616) but defined at ~line 651; the only added prototype is for command_fault. That is an implicit function declaration — an error under the kernel's -Werror=implicit-function-declaration (and wrong int-vs-void semantics regardless). Move the definition above drain() or add a prototype.

Non-blocking:

  • wait_fw_event_before_scan: usleep_range(min_t(s64, remainder * 1000, 500LL), 1000) — since remainder >= 1 makes remainder*1000 >= 1000, the min is always 500, so it sleeps 500–1000 µs per pass regardless of the remainder (looks like 500000 was intended). The loop still waits correctly, just polls drain() every ~1 ms.
  • The DTS hunk is unrelated to the Wi-Fi wedge: ~100 lines enabling the right USB-C port (hpm2/atc2_phy/usb_drd2/DARTs) plus the nub_spmi_a0 reg change. Please split it into its own PR — a Wi-Fi recovery patch should not carry USB-C bring-up, and the hardware rule wants the port tested on the hardware it affects.
  • No cap/backoff on transport recovery: a persistently faulting transport re-queues boot_work at delay 0 on every command_fault. Worth a retry limit or backoff once the deadlock is fixed.

15ms scan settle after channels/disconnect (M5 journals: failing gaps
0.34, 2.25, 2.51 ms; good 39.51, 43.04, 46.65 ms; no good pair <10.79 ms).
Transport recovery via centauri_t6050.transport_recover_work (avoids
wifi_unregister cancel_work_sync on the active worker; wAW #6974720627).
note_fw_event forward-declared; recovery capped at 3; kunit for phase map
and settle remainder. debugfs force_transport_fault. Based on
integration/aurora.
@joshuaswarren
joshuaswarren force-pushed the wAQ/m5-wifi-transport-recover branch from d717b98 to 388cd88 Compare November 18, 2026 09:47
@joshuaswarren

joshuaswarren commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

[Joshua's agent] Addresses review 6974720627 at 388cd88: transport recovery on the device work item (no self-deadlock), note_fw_event prototype, 15 ms msleep(remainder) hold, integration/aurora base (no unrelated DTS). Independent re-review requested.

@joshuaswarren

joshuaswarren commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

GREEN on d717b98
Model: cognition/swe-2-flash

[Joshua's agent]

Note: the requested SHA 0abc2f3 is a merge commit (PR #2177), not this PR's head — the actual head carrying the fixes is d717b98, so the verdict lands there.

Both blockers from comment 6974720627 are resolved:

  1. The self-deadlock is gone: recovery moved to centauri_t6050.transport_recover_work, and wifi_unregister no longer cancels it (the wifi-side recover_work/recover_queued pair is deleted entirely). transport_recover_init runs in probe under IS_ENABLED(CONFIG_APPLE_CENTAURI_T6050_WIFI) and transport_recover_cancel in quiesce before the boot_work cancel — the only cancel site, in a context that cannot be the worker itself.
  2. note_fw_event now has a forward declaration next to command_fault's.

Nits addressed: msleep(remainder) replaces the 500 µs cap; the ATC2 USB-C DTS hunk is gone after the rebase onto integration/aurora; recovery is capped at 3 (CEN_TRANSPORT_RECOVER_MAX) with the counter reset on reaching CEN_BOOT_READY.

Remaining observations, non-blocking: transport_recover_count is a plain int shared across boot_lock/workqueue/lock contexts (benign for a cap, KCSAN may flag); a debugfs-forced fault spends the same 3-recovery budget; a fault landing while boot_phase is STOPPED/FAILED or on an unmapped control stage still fails closed to CEN_BOOT_FAILED — conservative and documented in the code path.

@joshuaswarren
joshuaswarren marked this pull request as draft November 18, 2026 09:49
@joshuaswarren

joshuaswarren commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

GREEN on 388cd88
Model: cognition/swe-2-flash

[Joshua's agent]

d717b98 was the previous head — same patch content before the rebase/squash onto integration/aurora; the diff at this head is byte-identical to what the earlier GREEN covered, re-verified here.

Two questions raised for this head:

  1. Who cancels transport_recover_work: centauri_t6050_quiesce calls centauri_t6050_transport_recover_cancel (cancel_work_sync) on both paths that reach it — .remove directly, and the suspend/rewake teardown (centauri_t6050_rewake_teardown, which also covers the warm-reset re-enumeration). Ordering is right: removing/ready are cleared first so no new request can schedule, the recover cancel runs before cancel_delayed_work_sync(&boot_work), and a worker already inside schedule_transport_recover finishes under boot_lock while cancel_work_sync waits — it cannot requeue because removing is already set.

  2. Can boot worker and recovery run at the same time: no. centauri_t6050_boot_worker holds boot_lock for its entire phase loop, and centauri_t6050_schedule_transport_recover takes the same boot_lock around the control-stage read, free_ipc, and phase update. They serialize; the recover worker can only queue boot_work after releasing the lock. A fault during boot makes the recover worker wait out the boot loop (or slip into the 20 ms retry gap), then re-read the control stage — fail-closed to CEN_BOOT_FAILED on unmapped stages.

Both blockers from comment 6974720627 remain resolved at this head (recovery lives on centauri_t6050.transport_recover_work, no self-cancel; note_fw_event declared), the cap of 3 stands, and the ATC2 DTS hunk is absent.

@joshuaswarren joshuaswarren changed the title wifi: centauri T6050 IPC wedge self-recovery (M5) [Joshua's agent] wifi: centauri T6050 IPC wedge self-recovery (M5) Nov 18, 2026
Add APPLE_CENTAURI_T6050_RECOVERY_KUNIT so kunit.py can link
centauri_t6050_recovery into vmlinux without the full T6050 driver
or ARCH_APPLE. Trim t6050/.kunitconfig to that fragment only.
@joshuaswarren

Copy link
Copy Markdown
Contributor Author

GREEN on 3a56de0
Model: cognition/swe-2-flash

[Joshua's agent]

Note on the requested sha: 3735297 does not exist on the remote — refs/pull/2191/head and the branch both read 3a56de0, which is what this verdict covers. The described content (noop_llseek + kunit overlay) matches this head exactly.

Delta vs the prior GREEN at 388cd88 is two commits, 10 insertions / 1 deletion:

  • no_llseek -> noop_llseek on force_transport_fault_fops: correct convention for a write-only debugfs knob; noop_llseek returns 0 instead of ESPIPE so seeking userspace does not error.
  • .kunitconfig + the KUnit block already present in centauri_recovery.c: the suite centauri_t6050_recovery (9 cases: boot-phase mapping for ROM/IPC/cycle/stop23/null plus scan-settle remainder inside/waits-out/outside/no-event) is registered via kunit_test_suite under IS_ENABLED(CONFIG_KUNIT), and the config enables KUNIT + COMPILE_TEST + APPLE_CENTAURI_T6050(_WIFI)=y so centauri_recovery.o builds and links the suite. There is no APPLE_CENTAURI_T6050_RECOVERY_KUNIT symbol — the tests gate on CONFIG_KUNIT directly, which the config provides.

The prior review's deadlock and build findings remain addressed at this head (unchanged files). Not verified here: an actual kunit.py run — the config is plausible but I did not build it; flag if the suite has not been executed.

@joshuaswarren

Copy link
Copy Markdown
Contributor Author

GREEN on 7b96634
Model: cognition/swe-2-flash

[Joshua's agent]

Re-verdict at the new head. The sha cited earlier (3735297) is real — it is the third commit in the stack; the live head is 7b96634, which this covers.

Delta vs the prior GREEN at 388cd88 is 30 insertions / 3 deletions across six files, all KUnit build plumbing plus the noop_llseek fix already verified:

  • New Kconfig symbol APPLE_CENTAURI_T6050_RECOVERY_KUNIT (KUNIT && COMPILE_TEST && !APPLE_CENTAURI_T6050) builds centauri_recovery.o standalone so kunit.py runs the suite without the full driver. The mutual-exclusion dep makes the Makefile's ifneq exclusion defensive dead code — harmless.
  • centauri_recovery.c splits correctly: the two helper functions the tests call (scan_settle_remainder_ms, boot_phase_for_control_stage) stay unguarded; the transport-recover worker/request/schedule block is gated on CONFIG_APPLE_CENTAURI_T6050 so the KUnit-only link has no unresolved driver deps; the suite gates on RECOVERY_KUNIT.
  • .kunitconfig selects only the needed surface (NET/INET/NETDEVICES/WIRELESS + RECOVERY_KUNIT), dropping the full-driver symbols that would have failed the KUnit-only build.
  • Driver sources are otherwise untouched since the GREEN head — only noop_llseek, already verified.

The reported 9/9 kunit pass matches the nine KUNIT_CASE entries. Nits, non-blocking: the trailing #endif comment still says CONFIG_KUNIT (the guard is RECOVERY_KUNIT), and .kunitconfig/Makefile lost their trailing newlines.

centauri-wifi-pm-tests: match command_fault signature from centauri_wifi.c.
centauri-recovery-tests: stub IS_ENABLED, CONFIG_APPLE_CENTAURI_T6050_WIFI, and transport_recover_cancel for quiesce path.

Proof: python3 tools/asahi/centauri-wifi-pm-tests.py (+ --negative-controls) &&
centauri-recovery-tests.py (+ --negative-controls) all pass at this tree.
@joshuaswarren

Copy link
Copy Markdown
Contributor Author

GREEN on 4273e3a
Model: cognition/swe-2-flash
[Joshua's agent]

Check page at this head, read at verdict time: all green — source, host-tests, rust-build, kunit, and all three DWC3/ATC variants (runs 44293856141, 44293856209).

Delta since the prior GREEN at 7b96634 is test-harness only (tools/asahi/centauri-recovery-tests.py, tools/asahi/centauri-wifi-pm-tests.py): the wifi-PM mock's command_fault now matches the driver's four-argument signature, and the recovery mock defines CONFIG_APPLE_CENTAURI_T6050_WIFI plus IS_ENABLED and stubs centauri_t6050_transport_recover_cancel with the real prototype. No driver code changed; the earlier review of the recovery/KUnit split stands. Prior findings unchanged: compile/KUnit and host-test evidence only — no M5 hardware run claimed.

@axboe

axboe commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

A data point from a K116s (T6050, firmware 27.2) running this PR's head (4273e3a) on integration/aurora 0abc2f3: the wedge happened on a command other than a scan, and the recovery declined it.

What happened. Five minutes after a deep-sleep wake, with the link up and in use, the AP dropped the station:

wpa_supplicant: CTRL-EVENT-DISCONNECTED bssid=<redacted> reason=1
wpa_supplicant: CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
wpa_supplicant: CTRL-EVENT-REGDOM-CHANGE init=USER type=COUNTRY alpha2=US
apple_centauri_t6050: Wi-Fi firmware channels: 2.4 GHz 13, 5 GHz 25, 6 GHz 0 (167 entries)
apple_centauri_t6050: Wi-Fi transport fault (lost transport) cmd=0x9 wifi_stage=0x3 control_stage=0x2 gap_ms=0 rx=209
apple_centauri_t6050: firmware country update failed: -5
apple_centauri_t6050: Wi-Fi transport recovery unsupported (control stage=0x2): -95

Every scan after that failed with -5 until a reboot.

Two things this shows:

  • The command was the country update, not a scan. wifi_regulatory() sends command 9 as cfg80211 reacts to the disconnect, 0 ms after the firmware's channels event. The 15 ms settle (wait_fw_event_before_scan()) only covers the scan path, so it did not apply.
  • The recovery stops at this state. The control port was at stage 2 and the Wi-Fi function at 3. centauri_t6050_boot_phase_for_control_stage() returns -EOPNOTSUPP for control stages 2 and 3, the boot phase goes to FAILED, and nothing restarts the function.

Suggestions: apply the settle before command 9 as well (or before every command that can follow a firmware event), and give the stage 2 / Wi-Fi stage 3 case a restart of the function. I can test either on this machine.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks for the gap study. It suggests the wedge comes from the host sending a command 2-3 ms after a firmware channels/disconnect event, while the firmware is still handling it. A fixed 15 ms settle before scans (centauri_recovery.h, wait_fw_event_before_scan()) only covers one command. axboe's K116s report shows command 9 (country update) wedging 0 ms after the event.

Could the driver wait for whatever completes that event (a reply, an ack or a stage change) before it sends any command? If none exists, please say what was checked.

The recovery also misses the case that was seen in the field. centauri_t6050_boot_phase_for_control_stage() returns -EOPNOTSUPP for control stages 2 and 3, and axboe hit control 0x2 / wifi 0x3, which went to FAILED until reboot. Also, command_fault() on the cancel path (e.g. suspend) now requests recovery too.

#2141 and #2144 touch the same recovery state.

Row on a K116s or K114c (fw 27.2):

  • Disconnect and immediate reconnect/regdom change x20 with the handshake fix: 0 transport faults.
  • If recovery stays: a forced fault x3 re-associates in under 15 s, including from control stage 2.

@joshuaswarren

Copy link
Copy Markdown
Contributor Author

[Joshua's agent]

Wait for completion, not fixed 15 ms: Agreed. CEN_FW_EVENT_SCAN_SETTLE_MS was a stop-gap from the gap study (often 2–3 ms host→scan after channels/disconnect on bad boots). We checked: there is no single blocking ack tied to 0x80000030 / disconnect in the Asc IPC path we own — only the staged boot worker + note_fw_event() timestamp. Next push: gate all host commands on centauri_wait_control_idle() (control stage stable + no in-flight cmd slot) after any fw event, not scans alone; document what was ruled out (no extra mailbox reply on channels).

Stages 2/3: centauri_t6050_boot_phase_for_control_stage() returning -EOPNOTSUPP for 2/3 matches axboe's wedge (control 0x2 / wifi 0x3 → FAILED until reboot). Next push maps those to the same recovery worker as stage 1 faults.

#2141 / #2144: Acknowledged overlap; we will rebase onto the integration recovery helper and dedupe state flags before ready.

Measured: KUnit 9/9 at head 4273e3a27a4d (log mesa-xbuild:~/scratch-w86/wAQ-2191-kunit7-qemu.log). M5 field script: ~/scratch/main/m5-2191-test.md (scan settle absence test, debugfs force_transport_fault, ≤60 s re-associate). Not run on K114c tonight (M5 down).

M5 row (K114c or K116s, fw 27.2): disconnect + immediate regdom/scan ×20 → 0 lost transport; forced fault ×3 via force_transport_fault → re-assoc <15 s including from control stage 2 (after next push).

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, gating every host command on a control-idle state after a firmware event answers the main point. When it lands, please remove the 15 ms CEN_FW_EVENT_SCAN_SETTLE_MS wait rather than keeping it alongside, and make "control stage stable" a state check rather than a timed debounce. One item from before is still open: command_fault() on the cancel path (e.g. suspend) also requests recovery. A cancel during stop should not start a transport rebuild. The K114c/K116s rows stay as listed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants