Skip to content

T6000 J314s (M1 Pro) at EL2: sleep entry commits, then a watchdog reset; macOS's commit state and the resume mailbox on T600x? #2198

Open

Description

@CoreyH

I run a J314s on omacom's linux-aurora (m1n1-aurora, the kernel at EL2, spin-table CPUs), not under aura; Linux boots with the 13.5 firmware set, under system firmware 26.6.2. I've been working on the suspend-to-RAM entry from Linux on that machine. Notes: https://github.com/CoreyH/sleep-drain (RESEARCH.md).

Where it stands

  • A replay of macOS's entry takes every core down and commits, and the SoC restarts on a watchdog instead of sleeping. Moved from 30 s to 120 s, the restart follows the watchdog.
  • The reconfig engine's entry pair is read and decoded here (the second sequence is 246 ops; its first poll is one AON status bit, its power-downs come after). A marker in the second watchdog says the pair stops before its last twenty-odd ops.
  • I don't yet know whether the last core goes down.

This is the symptom #2075 had on the K116s before it was solved: the entry completes and a watchdog resets the SoC. There it was one power word iBoot leaves on, found by a diff against macOS's state at its commit.

Questions

  1. macOS's state at its commit on a T600x. Does anyone have the PMGR power words as macOS has them at its commit on an M1 Pro or Max (as the J813's ps-macos-s04.txt), or just the list of those that read on? This Mac's words under Linux I can read and post for the diff.
  2. The resume mailbox. soc: apple: sleep: per-SoC data and operations for the sleep sequence #1769 keeps the mailbox (/chosen/aurora,s2r-mailbox) in the shared part of sleep.c, so I take it to work the same way on every chip. Is there anything about T600x that would keep a loader from sending the boot core back through it?
  3. A completed entry on T600x. Has an M1 Pro or Max under aura ever come back through the boot core's reset vector after a completed entry? J314S is on the qualified list, but I couldn't tell whether that includes deep sleep.
  4. Data. Would T6000 data for sleep.c be welcome while the entry still resets here (the structure of the entry pair, its polls and gates, what was tried and ruled out), or only once an entry completes?

This work is done with an AI assistant (Claude). It keeps away from aura and m1n1 themselves; I've only read the kernel-side sleep drivers in your PRs (#1477, #1664), credited in my notes.

Activity

  1. RyanTheTide commented on Nov 19, 2026

    @RyanTheTide
    Contributor

    Answers from integration/aurora 0abc2f3 and aura main:

    1. macOS's commit state on T600x. We don't have it. No PMGR power-word dump at macOS's commit has been taken on an M1 Pro or Max, so there is nothing to diff against yet. The J813 and K116s lists are from other SoCs and won't carry over.
    2. The resume mailbox. Nothing in sleep.c is T600x-specific about /chosen/aurora,s2r-mailbox. The kernel uses whatever the loader publishes. aura publishes it on SoCs whose RVBAR is not locked (aura#217), and that includes T6000: on an M1 Pro under aura the node is published and the boot core parks. Under m1n1-aurora nothing publishes the node, so the kernel has no mailbox to use. A loader that wants to send the boot core back through it needs the same RVBAR setup aura#217 does.
    3. A completed entry on T600x. No. No M1 Pro or Max has come back through the reset vector after a completed entry. sleep.c has SoC operations only for T8140, T8132, T8142 and T6050 (apple,*-sleep match table). There is no T6000 entry, so on T6000 the J314s qualification covers s2idle only.
    4. Data. T6000 data is welcome now, here: the entry pair's structure, its polls and gates, and what was ruled out. It doesn't need to wait for a completed entry. A t6000 SoC entry in sleep.c needs one, since the sleep ops are only enabled where an entry has been shown to come back.
  2. CoreyH commented on Nov 19, 2026

    @CoreyH
    Author

    Thank you, that answers all four, and more plainly than I hoped.

    • Commit state: understood, there is none for T600x yet. I'll stop looking for one and say so in my notes.
    • Mailbox and RVBAR: this is the part I had wrong. My notes assumed iBoot leaves CPU0's RVBAR locked on this chip; I had never read it. I'll read the register and its lock bit on this J314s (under m1n1-aurora) and post what it shows.
    • Completed entry: noted that none exists on M1 Pro or Max, and that the J314S qualification is s2idle.
    • Data: I'll write it up in this issue, in one comment: the entry pair op by op (its polls, gates and handshakes), what each run ruled out, and how each was measured.

    Two results from yesterday that will be in it, since they bear on where the entry stops here:

    • The pair's second sequence never reaches its power-downs. A hardware witness: the keyboard backlight's PWM, whose power domain is the tenth word that sequence switches off, was set blinking right before the commit and blinked until the watchdog's restart 70 s later.
    • The AON status word that sequence polls first (0x2922c0044, bit 2 set, then bit 3 clear) was read after the commit with one core kept up, in two runs. Awake it reads 0x8. With the boot core up and both P complexes off, 0xb. With a P core up, and the boot core's and the E complex's power words reading off, 0x9. So bits 0 and 1 follow the two P complexes, and bit 2, the one polled, is still clear with the E complex's power word off while a P complex is up. What sets bit 2 I don't know; if that word is familiar from another SoC, I'd be glad of a pointer.

    This work is done with an AI assistant (Claude). It keeps away from aura and m1n1 themselves.

  3. CoreyH commented on Nov 19, 2026

    @CoreyH
    Author

    The T6000 data, as promised. Everything here is from one J314s (M1 Pro, 8 cores: 2 E, 3 + 3 P) on omacom's linux-aurora 7.2.11 at EL2 under m1n1-aurora, booted with the 13.5 firmware set under system firmware 26.6.2. The notebook with every run is https://github.com/CoreyH/sleep-drain (RESEARCH.md); section names are given below where a claim rests on one. I say "measured" for what a run recorded, "read" for what the SRAM's bytes say, and "inferred" for the rest.

    How it was measured

    • An out-of-tree module replays macOS's entry from Linux at the end of noirq, in a boot whose disk is read-only: the PMGR quiesce table (maralcbr's, from maralcbr/linux), then the cores' park and deep WFI as Naeem Malik's T8103 driver does it, then the commit word. Each run is one-way.
    • A run's record is a few bytes in the PMU's scratch (0x6004-0x601f), written by raw SPMI commands from the last core up, and read at the next boot. The PMU's RTC gives the time from the commit to the restart to within a second.
    • Two things that need no CPU give the rest: the watchdogs, and the keyboard backlight's PWM.

    The reconfig SRAM (read)

    Read once, 64 KiB, at the address PongoOS's table and the ADT's carve-outs give (the table-offset register's analogue 0x2925c0200 reads 0x1000, its lock 0x2925c0214 reads 0). The dump itself isn't public.

    • Two tables of four sequences each, in PongoOS's format, A0-A3 and B0-B3. They pair up: A_i writes phase codes to 0x2922c0110 and handshakes on 0x2925c4004 (the commit word + 4): bit 31 up, wait for bit 24, bit 31 down, wait for bit 24 to fall. B_i reads the phase codes in the same order and handshakes on 0x2922c010c: bit 1 up, wait for bit 0 to fall, and back. Pair 2 is the entry; 3, 0, 1 are the wake, in that order (inferred from their contents: B3 checks the 16 DCS words off and powers the fabric on, B0 resets SIO_CPU and the PMP and writes about 1,800 registers, A1 ends by bringing EACC up and writing 1 to 0x28e0d4008).
    • A2 opens, as every A sequence does, by arming watchdog 2: control 0, count 0, bite 30 s, control 4. It writes phase 4, raises its first handshake, then sets DCS and the fabric to p-state 2 and 0x28e20c400 to 0, each waited for; phases 5 and 7 follow. Its last op stops WD2.
    • B2 is 246 ops: 152 writes, 89 polls, 4 delays, End.
      • Ops 1-6, its opening: poll the AON word 0x2922c0044 for bit 2 set, a delay, poll it for bit 3 clear, then write 0x2922c0040's bit 1 to 0 and poll it clear. These three polls have no retry count.
      • Then 22 clock words written, the first handshake (ops 31 and 33), and from op 35 the power-downs: SIO's subtree, 43 words in dependency order (SIO_CPU first, FPWM0 tenth, SIO itself last), GPIO, five PCIe words, ANS2, the PMP, DWI. B2 writes 0 to 52 power words and polls 49 of them for bit 11 clear; six gate-status words for bit 30. Three coprocessors are raised and polled on the way (SIO_CPU, ANS2, the PMP: power word actual f, then bit 1 of the ASC's word).
      • Ops 177-187: bit 31 clear on eleven clock blocks. Ops 223-224: watchdog 0's and 1's counts zeroed. Ops 237-238: the PMU's register 0x0201 := 1 over SPMI, its ack polled at 241. The second and third handshakes around those.
    • Separate tables hold a down and an up sequence per soc cluster, the three CPU complexes among them. Every down sequence opens by raising 0x2922800b0 and then waits with no timeout; a CPU complex's waits for bit 6 of its soc-cluster word, writes 0x51 to 0x28e09a000 + 4n and waits for bit 5 of that to clear.
    • Stores to the SRAM from the AP are dropped (two runs), so the sequences can't be instrumented in place.

    Where the entry stops (measured)

    1. The commit starts A2 at once. With a core still up: phase 4 at 10 ms; at 1 s and at 25 s phase 4, 0x2925c4004's bit 31 up and bit 24 clear, and WD2 armed and counting from the commit. A2 sits in its first handshake.

    2. The restart is the engine's watchdog, not a sleep. It comes at WD2's 30 s. With WD2's bite rewritten to 120 s after the commit, it comes at 120 s.

    3. B2 never zeroes WD1's count (its ops 223-224). With WD1 armed before the commit so that an untouched count bites at 90 s, the restart comes at 90 s; preset for 70 s, at 70 s.

    4. B2 never reaches its power-downs. The keyboard backlight's PWM (fpwm0) was set blinking right before the commit. FPWM0's power word is the tenth B2 switches off. The light blinked for all 70 s.

    5. The cores do go down, as far as PMGR shows. Seven cores parked read off at their release, and both P complexes' power words with them. With a P core kept up as an observer after the commit, the boot core's power word reads off within 500 ms of its WFI and the E complex's within 500 ms more.

    6. 0x2922c0044, B2's first poll, with one core kept up after the commit:

      State CPU0, ECPM PCPM0 PCPM1 0x2922c0044 [3:0]
      awake on on on 1000
      boot core up, 1 s after the commit on off off 1011
      a P1 core up, after CPU0 and ECPM read off off off on 1001

      Bit 0 follows P complex 0 and bit 1 P complex 1. Bit 2, the one polled, is clear with the E complex's power word off while a P complex is up. It has never been read set. In all three, 0x2922c0040 reads 3 (bit 1 set, so B2's op 4 hasn't run). 0x2922c010c's bit 0 is up after the commit and clear awake.

    7. With every core in its WFI nothing can read the word, so whether bit 2 rises then isn't known. What is known for that state: the restart still comes at WD1's time, the light still blinks, an RTC alarm set 20 s after the commit fires (the PMU latches it) and nothing follows, and the last core's own timer, set 25 s out, doesn't bring it back to the next instruction.

    8. The PMU's 0x0201 reads 2 at every boot after a run, the value an ordinary boot shows.

    9. RVBAR, read on each core (cpu-impl-reg + 0x0, one 64-bit load made on the core itself, under m1n1-aurora):

      Core RVBAR
      cpu0 (E) 0x00100100059e4001
      cpu1 (E) 0x00111100059e4000
      cpu2-4 (P0) 0x01100100059e4000, 0x01121100059e4000, 0x01132100059e4000
      cpu5-7 (P1) 0x02100100059e4000, 0x02111100059e4000, 0x02122100059e4000

      All eight carry the same address, 0x100059e4000, and bit 0 is set on the boot core alone. I read that as the boot core's vector still locked, at the boot object's entry, on this boot chain. That isn't what answer 2 above says for T6000 under aura, so the two chains differ here, or I'm misreading bit 0.

    So B2 is held in its first six ops or in the first handshake, and A2 behind it. Which of the three polls or the handshake isn't shown.

    What was tried, with the same ending each time

    Each of these was made by the AP before the commit, read back where it could be, and left the restart where it was:

    • macOS's PMGR quiesce in its place in the table with its waits (only DISPLAY's and SIO_CPU's time out), then the DCPs stopped and the display domains off, after which every wait of macOS's is met;
    • the commit word written once and not three times; the SMC told (slpw, slpa, the AP idle notice acked);
    • SIO's firmware stopped; the PMP's CPU stopped after its idle pair; the 20 power words macOS has off that Linux leaves on;
    • the up-done flag acked on three soc-cluster words (ANE0, AFR, AVEMSR), all four cluster words reading 0 before the commit;
    • the fabric's p-state put back to 3, where macOS has it at its commit (Linux's suspend leaves it at 1);
    • B2's own steps made ahead of it: SIO's subtree off, the PCIe words off, GPIO's cycle, A2's three moves (DCS, fabric, 0x28e20c400), the CPU complexes' cluster words written last. With item 4 above these were never going to matter: B2 doesn't get that far.
    • The 22 clock words B2 writes before its power-downs don't take a write from the AP at all.

    Not known

    • What bit 2 of 0x2922c0044 is. The readings fit all three complexes off together, or the E complex in a sense its power word doesn't carry, or something that isn't the CPUs'.
    • Whether the last complex goes off when the last core takes its WFI.
    • Whether a boot core with bit 0 of its RVBAR set can be sent anywhere but that entry after a completed sleep.

    If the word at AON + 0x2c0044 or the cluster sequencer's raise at 0x2922800b0 means something to anyone from T8140 or T6050, that is the pointer I'm missing.

    Credits for what this stands on: maralcbr's entry tables, Naeem Malik's T8103 S2R drivers (malik-na/linux-aurora), the T8140, T6040 and T6041 sleep drivers in this repository for how to read the classes of wait, and PongoOS for the reconfig format. This work is done with an AI assistant (Claude); it keeps away from aura and m1n1 themselves.

  4. CoreyH commented on Nov 19, 2026

    @CoreyH
    Author

    A follow-up with what four more runs on the J314s (T6000) added today, since it changes the picture in the data above. Same labels: "measured" for what a run recorded, "read" for the SRAM's bytes, "inferred" for the rest.

    1. Each CPU complex's own down sequence runs to its end (measured). Each complex has a state word at 0x28e09a000 + 4n that its down sequence steps (read: 0x51 after its first untimed wait, then 0x53, 0x5b, 0x1b, and 0x1a as its last write; an up sequence ends on 0x40). Awake the three read 0x44 (E), 0x40, 0x40. With a P1 core kept up after the commit and the boot core in its last WFI, the E complex's reads 0x1a half a second after its power word reads off, the same as P complex 0's, and EACC's soc-cluster word reads 0. So the E complex goes down in the sequencer's sense too, not only by its power word.

    2. The gate word counts the complexes that are down (inferred from four readings). 0x2922c0044's low three bits: 000 awake; 011 with the boot core up and both P complexes down; with a P1 core up and CPU0's and ECPM's power words off, 001 right after ECPM read off (one run) and 011 half a second after that (another run). No assignment of one bit to one complex fits both 011s, so my table above ("bit 0 follows P complex 0, bit 1 P complex 1") was wrong. A count fits all four, and bit 2, B2's first poll, is then the third complex: all three down. The two samples with a P1 core up are from two runs, so no run has seen the word change.

    So with a core up the first poll is closed because the core is up, and the stall is in the state no core can read: every core in its last WFI, and B2 still short of its tenth SIO power word (the keyboard PWM keeps running until the watchdog's restart).

    3. ATC0's chain was on at the commit, and lowering it changes nothing (measured). The power-word list this recipe copies came from a trace made under a hypervisor on one USB-C port, so ATC0_COMMON and ATC0_USB_AON were on where the other ports' are at 0x300, and ATC0-2_USB stay on through Linux's suspend. With those six written 0x300 before the commit (each read actual 0 within 10 ms), the ending is the same to the second.

    What I'd like to ask, of anyone who has been through this stretch on T8140, T6040/41 or T6050: between "every core has taken its last WFI" and the engine's first SIO power-down, what did you find it waiting on? Here that stretch is B2's opening (the count word's bit 2 set, a delay, its bit 3 clear, then 0x2922c0040 written 0 and its bit 1 polled clear), B2's first handshake with A2, and A2's own three moves (DCS and the fabric to p-state 2, 0x28e20c400 to 0, each waited for with no timeout). In particular:

    • Is bit 3 of that word (set in every reading here, polled clear after the delay) something you know by name: a wake line still raised, a fabric or rail still up?
    • The M4 Pro's driver says a pending wake source that the commit's masks let through would hold the first gate (as a caution, not as a stall it saw). Here the AP wake mask leaves only the PMU's source open. Does that source need masking too, or is that how macOS has it?

    The next run here makes A2's three moves from the AP before the commit, with the PWM as the witness of whether B2 then reaches its power-downs. The notebook has each run (https://github.com/CoreyH/sleep-drain, RESEARCH.md, the sections dated 2026-11-19).

    As before: this work is done with an AI assistant (Claude), and it keeps away from aura and m1n1 themselves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions