Skip to content

smp: start all 30 of the M5 Ultra's CPUs, not 27, without a larger image - #262

Open
funkzy wants to merge 2 commits into
aurora-silicon:mainfrom
funkzy:t6050-ultra-cpus
Open

funkzy wants to merge 2 commits into
aurora-silicon:mainfrom
funkzy:t6050-ultra-cpus

Conversation

@funkzy

@funkzy funkzy commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What is missing

The M5 Ultra (Mac Studio, J775D) numbers its 30 CPUs 0 to 34 in the ADT: 6 * cluster + core, 18 per die, so 5, 11,
17, 23 and 29 are unused. smp_init() indexes every per-CPU table by cpu-id and stops at MAX_CPUS (32). On main the
loader logs "cpu-id 32 exceeds max CPU count 32: increase MAX_CPUS" for cpu-ids 32 to 34 and Linux gets 27 of the 30
CPUs.

What this adds

  • MAX_CPUS goes to 64, the width of the u64 CPU masks (smp_started_mask, hv_cpus_in_guest, s2r_twi_mask).
  • MAX_CPUS also sized the secondary stacks, one 64 KiB stack per cpu-id, so on its own it would add 2 MiB to every
    Stage 2's .bss and 2 MiB to a Stage 1's heap (dj's note in M5 Ultra multi-die groundwork: AIC IRQ numbers, CPU die, PMGR die stride, MIDR parts #178). The stacks get their own count instead,
    SECONDARY_STACKS (32, as now), and go to the CPUs present in cpu-id order: smp_init() records each CPU's slot
    in stack_slot[], in .data.smp_shared beside smp_reset_stacks, because a secondary reads it with the MMU off.
  • The stacks stay where they are: in .bss (svenpeter's e327b97, for PSCI), or Stage 1's heap (Ryan's f545f79).
  • A CPU past the 32nd present would get no stack and is skipped, as a cpu-id past MAX_CPUS still is.

Earlier raises took the same path for each larger Mac: 20 to 24 for the M2 Ultra (marcan's dabad98), 24 to 32 for
the M3 Ultra (jannau's 1a0e368).

Who runs the changed code

Every SoC: MAX_CPUS is compile-time.

  • Per-CPU tables double. In a RELEASE=1 CHAINLOADING=1 ESP_STAGE2=1 build the .bss objects grow by 12.6 KiB
    (hv_sprr_cpus 6.5 KiB, pcpu 2 KiB, hv_secondary_regs 1 KiB, the rest 3.1 KiB), which the alignment padding
    absorbs: .bss ends 0x1200 lower and _end is unchanged. .data.smp_shared grows by 6.6 KiB (kernel_park,
    spin_table_image, smp_reset_stacks, stack_slot) inside its 64 KiB padding, so _file_vital_end and
    _file_end are unchanged.
  • Stacks: same count, same place. Where the ADT's cpu-ids run 0 to n-1 with no gap, a CPU keeps the stack it
    had. Where they skip (as on J775D's first die, and J616s, which has no cpu9), a CPU after the gap takes a lower one
    of the same 32.
  • T8142 keeps its own heap block, indexed by cpu-id as before.
  • T8152: its twelve cpu-ids are 0 to 11, so its slots equal them, as its cpu_reset in start.S assumes.
  • Development builds only: the NV2 trace buffer (NV2_TRACE=1) grows from 2 to 4 MiB of heap and the SPTM-virt
    GL2 shadow (SPTM_VIRT=1) from 256 to 512 KiB.
  • cpu-id loops (start.S's reset-stack search, kboot's CPU pruning, hv, nv2, sptm_virt) run to 64 instead of 32.

Size

clang/lld 23.1.4, RELEASE=1 CHAINLOADING=1 ESP_STAGE2=1: aura.bin 3997696 bytes as on main; _file_vital_end
0x11b6c8 and _end 0x3d0000 unchanged; .bss 0x25bf30 to 0x25ad30. Code: +0x88 for smp_assign_stacks(), a few
instructions in the callers. A plain MAX_CPUS 64 would take .bss to 0x45ed30 and _end to 0x5d0000. CI's
esp-stage2-size configurations report main's vital sizes (1095728 and 744376 bytes). The exception is a development
build with SPTM_VIRT=1, whose .bss objects add 60 KiB (nv2_cpus 32 KiB, sptm_world 12.75 KiB) and cross a
64 KiB boundary, so _bss_end, _end and aura.bin grow by 64 KiB (CHAINLOADING=1 SPTM_VIRT=1: _bss_end
0x190000 to 0x1a0000, 1769472 to 1835008 bytes; with ESP_STAGE2=1, 4128768 to 4194304 bytes).

Validation

  • The build above, plus RELEASE=1 CHAINLOADING=1 (Stage 1, heap stacks), ESP_STAGE2=1 SPTM_VIRT=1 and CI's two
    esp-stage2-size configurations: all compile with main's warnings, none in smp.
  • The host pytest suite (tests/python, pytest 9.1.1): 707 passed, 56 skipped, as on main.
  • clang-format clean on the three files.
  • Rebased onto main at 3d517e6 (the T8142 sleep tracer, sptm_virt: T8142: macOS on all ten cores, and a trace of its sleep and wake #277, among the new commits) without a conflict; the diff is
    unchanged. There the build above has main's warnings, aura.bin 3997696 bytes and _file_vital_end 0x11b890, both
    as on main (the figures under Size are from the earlier base), and the host pytest suite as CI runs it gives
    750 passed, 56 skipped, as on main.

Hardware

  • J775D, main at eb66d6c plus this commit and the M5 Ultra GPU firmware-build PR, with the kernel uploaded over USB,
    booting integration/aurora at 7c71d6df6563 with the M5 Ultra linux pull request (and a compute-timestamp change
    that is no longer in this set): no "exceeds" line; the loader starts all 29 secondaries, cpu-ids 32 to 34
    included, and passes them to the device tree. Linux: "smp: Brought up 1 node, 30 CPUs", online 0-29, six
    clusters, a task run on each CPU. The GPU probes and passes its compute, chained-submit and 46-client checks, with
    no recovery, fault or timeout, and the kernel log has no error-level line.
  • J775D, main at eb66d6c plus this and the other aura pull requests below, booting integration/aurora at
    fc050503ffcd with only the linux pull requests below, one boot: "smp: Brought up 1 node, 30 CPUs", online 0-29;
    the GPU passes its compute, chained-submit and 46-client checks and a 628-case render CTS subset (571 pass, 57 not
    supported), with no recovery, fault or timeout.
  • J775D, main at 03d120d plus this and the other aura pull requests below (the same files as this branch), booting
    integration/aurora at 1f38ff8f4dc9 with the linux pull request below: "smp: Brought up 1 node, 30 CPUs", online
    0-29, on the 27.0 stub (the GPU checks and the CTS subset pass as above) and on the 27.2 stub (26B123) alike.
  • Not booted on any other Mac. dj, could you boot it once on a J714s? A boot on a T8152 (J873g) would check the one
    assembly path that reads a stack by cpu-id, and one on a T8140 would check its .bss limit.

Question

64 follows the u64 masks and #178's note. J775D needs 35; MAX_CPUS 36 (two dies of 18) would add 1.6 KiB of .bss
objects instead of 12.6 KiB. Which would you rather have?

Part of the M5 Ultra platform pack

Each pull request below stands on its own; together they bring the Mac Studio (M5 Ultra, J775D) up from public
branches to the desktop.

The M5 Ultra (T6050, J775D) numbers its 30 CPUs in the ADT 0 to 34:
6 * cluster + core, 18 per die, so cpu-ids 5, 11, 17, 23 and 29 are
unused. smp_init() indexes every per-CPU table by cpu-id and stops at
MAX_CPUS (32), so it logs "cpu-id 32 exceeds max CPU count 32" for
cpu-ids 32 to 34, and Linux gets 27 of the 30 CPUs.

MAX_CPUS goes to 64, the width of the u64 CPU masks (smp_started_mask,
hv_cpus_in_guest, s2r_twi_mask). It also sized the secondary stacks, one
64 KiB stack per cpu-id, so on its own it would add 2 MiB to every
Stage 2's .bss (and 2 MiB to a Stage 1's heap), as dj noted in aurora-silicon#178.
The stacks keep their own count instead, SECONDARY_STACKS (32, as now),
and go to the CPUs present in cpu-id order: smp_init() records each
CPU's slot in stack_slot[], next to smp_reset_stacks in
.data.smp_shared because a secondary reads it with the MMU off. They
stay in .bss (svenpeter's e327b97, for PSCI) or Stage 1's heap
(Ryan's f545f79). A CPU past the 32nd present would get no
stack and is skipped, as a cpu-id past MAX_CPUS still is.

Earlier raises took the same path for each larger Mac: 20 to 24 for the
M2 Ultra (dabad98), 24 to 32 for the M3 Ultra (1a0e368).

Every SoC runs this code, since MAX_CPUS is compile-time:
- The per-CPU tables double. In a RELEASE=1 CHAINLOADING=1 ESP_STAGE2=1
  build the .bss objects grow by 12.6 KiB (hv_sprr_cpus 6.5 KiB, pcpu
  2 KiB, hv_secondary_regs 1 KiB, the rest 3.1 KiB), which the
  alignment padding absorbs: .bss ends 0x1200 lower and _end is
  unchanged. .data.smp_shared grows by 6.6 KiB
  (kernel_park, spin_table_image, smp_reset_stacks, stack_slot) inside
  its 64 KiB padding, so _file_vital_end and _file_end are unchanged.
  The two configurations CI's esp-stage2-size job builds report the
  same vital size as main.
- The stacks: same count, same place. Where the ADT's cpu-ids run 0 to
  n-1 with no gap a CPU keeps the stack it had. Where they skip (as on
  J775D's first die, and J616s, which has no cpu9), a CPU after the gap
  takes a lower one of the same 32. T8142 keeps its own block, indexed
  by cpu-id as before. T8152's twelve cpu-ids are 0 to 11, so its
  slots equal them, as its cpu_reset in start.S assumes.
- Development builds only: the NV2 trace buffer (NV2_TRACE=1) grows
  from 2 to 4 MiB of heap, the SPTM-virt GL2 shadow (SPTM_VIRT=1)
  from 256 to 512 KiB, and an SPTM_VIRT=1 image by 64 KiB (its
  per-CPU tables in .bss). Default and release images keep their
  size.
- The cpu-id loops (start.S's reset-stack search, kboot's CPU pruning,
  hv, nv2, sptm_virt) run to 64 instead of 32.

Tested on J775D (M5 Ultra) only, with main at eb66d6c plus this
commit and the M5 Ultra GPU firmware-build commit, the kernel uploaded
over USB, booting integration/aurora at 7c71d6df6563: the loader starts
all 29 secondaries, cpu-ids 32 to 34 included, with no "exceeds" line;
Linux brings up 30 CPUs (online 0-29, six clusters) and runs a task on
each, probes the GPU and passes its compute, chained-submit and
46-client checks, with no error-level kernel message. Stage 1, ESP
Stage 2 and SPTM_VIRT builds compile with main's warnings, the host
pytest suite gives main's 707 passed and 56 skipped, and clang-format
is clean.

Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor

No-regression rows with this PR stacked on current main (aura-try Stage 2 trials, production kernels): J616s (T6040, cpu9 absent from the ADT) 14/14 CPUs online, s2idle 3/3; Neo (T8140, J700) 6/6 CPUs, image fits, s2idle 3/3; M2 8/8 and M1 Pro 10/10 CPUs, s2idle 3/3. 0 kernel errors and 0 DART faults on all four.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants