Skip to content

accel/ane: h17: recover a hung engine and retry a failed boot - #1768

Draft
RyanTheTide wants to merge 1 commit into
integration/aurorafrom
fix/ane-h17-hung-recovery
Draft

RyanTheTide wants to merge 1 commit into
integration/aurorafrom
fix/ane-h17-hung-recovery

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Summary

On the Neo (T8140, H17 backend, ane_fw_cold.c) a request that never completes quarantines the session and the ANE stays unusable until reboot. A boot from the preload that fails, after a sleep or for the first request to a parked engine, quarantines the engine the same way. The usual cause of that failed boot is an ANE_SYS power-on that latched in the noirq resume, where the PMGR cannot run its PS reset recovery.

This brings H17 in line with the H16 (#1706) and T8112 (#1693) paths:

  • Hung request: the session's ops->hung() queues a recovery that runs the shared ane_t6021_restart_run() sequence (unplanned) with hung_stop and hung_settle_ms. It clears RUN in the wrapper, waits hung_settle_ms, powers the engine off without the firmware in macOS's order with the CPU and sub-block power states read back off, then boots it from the preload and replays every open program behind its handle. No reset is asserted. Requests wait in ane_h16_session_enter() meanwhile; system sleep flushes the recovery and blocks new ones.
  • Failed boot: retried once with ANE_SYS power cycled through apple_pmgr_ps_cycle() in process context, where a latched power-on is cleared. Quarantine only if that fails too. This is accel/ane: h17: retry a failed resume after power cycling ANE_SYS #1303's retry, moved to where the boot happens now (the parked engine boots on the next request, not at PM_POST_SUSPEND).
  • Quarantine across sleep: with hung_stop set, system sleep powers a quarantined engine off without the firmware and the resume boots it cold, so a failed recovery does not last until reboot. Programs closed while quarantined are released once the new firmware is up.
  • At runtime a power word that will not go off fails the recovery instead of being deferred to a deep sleep.

hung_stop=0 keeps the old behaviour for a hung session. debugfs ane-h17/<dev>/ gains recoveries, recover_failures and boot_retries.

Supersedes #1303.

Test

Neo hardware rows pending (forced hangs, s2r x3 with encoder rows, s2idle x3, failed-boot retry). checkpatch --strict clean.

🤖 Generated with Claude Code

On the Neo (T8140, H17 backend) a request that never completes
quarantines the session, and the engine then stays unusable until
reboot. A boot from the preload that fails, after a sleep or for the
first request to a parked engine, quarantines the engine the same way.
Typically that boot fails because ANE_SYS's power-on latched in the
noirq resume, where the PMGR cannot run its PS reset recovery, and genpd
carried on as if the domain were on.

Recover from both, as the H16 and T8112 paths already do:

- The session's ops->hung() queues a recovery with the shared restart
  sequence (ane_t6021_restart_run(), unplanned) and the hung_stop and
  hung_settle_ms parameters. It clears RUN in the wrapper, waits
  hung_settle_ms, powers the engine off without the firmware in macOS's
  order, with the CPU and sub-block power states read back off, then
  boots it again from the preload and replays every open program behind
  its handle. Nothing in the wrapper can prove the old firmware stopped,
  so the verified power-off is the proof. No reset is asserted. Requests
  wait in ane_h16_session_enter() while it runs; system sleep flushes
  it and blocks new ones.
- A failed boot from the preload is retried once with ANE_SYS power
  cycled through the PMGR (apple_pmgr_ps_cycle()) in process context,
  where a latched power-on is cleared. The engine is quarantined only if
  that fails too.
- With hung_stop set, a quarantined engine no longer stays down for the
  rest of the boot: system sleep powers it off without the firmware and
  the resume boots it cold. Programs closed while it was quarantined are
  released once the new firmware is up (ane_h16_session_recovered()).
- At runtime a power word that will not go off fails the recovery
  instead of being left for a deep sleep.

hung_stop=0 keeps the old behaviour: a hung session stays quarantined
until reboot.

debugfs ane-h17/<dev>/ gains recoveries, recover_failures and
boot_retries.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
m0d16l14n1 added a commit to m0d16l14n1/linux that referenced this pull request Nov 15, 2026
RyanTheTide: accel/ane: h17: recover a hung engine and retry a failed boot

With aurora-silicon#1863 (idle park) kept: the idle park leaves the engine alone while a
hung-request recovery is queued or running. aurora-silicon#1303 is not taken (aurora-silicon#1768 carries
its retry).
m0d16l14n1 added a commit to m0d16l14n1/linux that referenced this pull request Nov 15, 2026
…merge

The merge gave the shared closing brace to aurora-silicon#1768's last function.

Assisted-by: Claude Opus 5.5
Signed-off-by: Nick <nekreshen@yandex.ru>
@niteria

niteria commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Some Neo hardware data for the hung-request path, from a branch that does the runtime part of this independently: https://github.com/niteria/linux/commits/ane/h17-hung-recover (one commit, 28e9a31, on the Neo's integration base with #1863 idle park; no Signed-off-by yet).

Results on a J700 (Linux 7.3.0, Aura b9, booted with cma=256M so a 66 MB H17 program can load):

step result
known-good programs (an add and a 960x540 conv layer) pass
a program that hangs the engine (a wrong H17 conv chain from mil-hwx-compiler before its #45) EXECUTE times out (0x6e) → "restarting the engine" → "firmware recovered after a hung request in 144 ms (settle 100 ms)"
the known-good programs right after pass, no reboot
hung_stop=0, hang, then s2r (RTC wake) stays quarantined, as intended
hung_stop=1, s2r "quarantined engine recovered by system sleep", programs pass

recoveries 2, recover_failures 0. Before this, the same hang left every later H16_LOAD at -EIO until reboot, sleep included.

Differences from this PR's runtime path (in case any of them is useful here):

  • It never writes the wrapper register: no RUN clear. The engine is powered off under the hung firmware through the same power-down that sleep and idle park already use.
  • Before memory the old firmware could reach is reused, it requires that our power-down began while we held ANE_SYS, that genpd then reported ANE_SYS off with the PMGR actual-state bits off, that nothing has re-held it since, and that the CPU and sub-block power states read off. Without all of that, the session stays quarantined with its memory held, and the next sleep retries.
  • Idle park and the sleep path re-check for a quarantine after closing the session, since a failed gate release there quarantines it.
  • It has no failed-boot retry; this PR's apple_pmgr_ps_cycle() retry covers a case it doesn't.

For reference, macOS does the same thing in place: its ANE driver restarts the firmware after a request timeout, powering down without waiting for the firmware's POWER_DOWN acknowledgement, then cold-starting and reloading programs. That's from the driver's log strings in the 26.6.2 kernelcache, behaviour only.

Happy to run this PR's hardware rows on the same Neo with the same hang reproducer, or to fold anything from the branch in here, whichever you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants