Skip to content

soc: apple: show the PMP request words and AGX vote state in the report status (stacked on #334) - #338

Closed
joshuaswarren wants to merge 6 commits into
omacom:integration/aurorafrom
joshuaswarren:w73/recorder-on-334
Closed

joshuaswarren wants to merge 6 commits into
omacom:integration/aurorafrom
joshuaswarren:w73/recorder-on-334

Conversation

@joshuaswarren

Copy link
Copy Markdown

[Joshua's agent] Stacked on #334: the first four commits are #334's; only the top commit (78ebe0e, soc: apple: show the fabric, DCS and PMP request words in debugfs (read only)) is new, and this PR is rebased onto integration/aurora when #334 merges. Review that one commit.

What it adds to pmp-report: the report status debugfs file now also exists for the T6001 report and, when the PMP is running, prints the request word (PS-REQ), the actual word (PS-ACK) and the AGX vote state (agx_vote: enabled voted users idle_ms failures, copied under the vote's own lock). Read only: two readq() at the offsets the driver already reads on every set, and the vote state copy. Nothing is written.

Why: a performance A/B on the 16-inch M1 Max (decode +25 to +29 percent with the AGX request set) needed a separate probe module to show which request bits and memory states each run had. With this file and the fabric and DCS words of the companion apple-pmgr-misc change (#336) the state sits next to every timing.

Checked: the same change applied to the 7.1.12 lab tree builds with its flags (object compiled) and a separate review of that patch found no write, no lifetime problem and the file created only for the T6001 report with the PMP ready. This commit applies cleanly on #334's head (9f2a319c). Not booted yet.

…n t600x

On the M1 Max (t6001) the PMP keeps memory at its idle floor for a busy
GPU until the AGX power request, bit 9 of the PMP report's PS-REQ word, is
set. Nothing sets it on Linux, so GPU reads top out at 241 GB/s. The same
chip reads 361 GB/s on macOS.

Report GPU activity to the PMP. The GPU driver calls
apple_pmp_agx_activity_begin() when a submission starts and
apple_pmp_agx_activity_end() when it finishes. pmp-report sets bit 9 when
the first operation starts and clears it 100 ms after the last one ends.
It also clears it at once when the GPU driver unbinds or the report goes
away, so the request is never left set past the GPU's power-domain
release. The register sequence is the one apple_pmp_report_set_state()
already uses for report entries: read 0xf80, write 0x107c0, poll 0x1000,
status 0x10, 50 ms acknowledge. A set that is not acknowledged is cleared
again and not retried for a second.

The vote is off by default. pmp_report.agx_vote=1 turns it on, and
pmp_report.agx_vote_idle_ms (default 100) sets the hold. With it off
nothing is written. It applies to apple,t6001 only; t6000 is untested.

The decisions (set, clear, hold, retry, wrap of the 32-bit millisecond
clock) are plain functions in pmp-agx-vote.h with no kernel dependency.

Setting the bit by hand with a test module on a 16-inch M1 Max moved a GPU
read microbenchmark from 241 to 368 GB/s. This patch has not yet been run
on hardware.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…one power request bit

The PS-REQ word of the PMP report is shared by every client of the PMP:
the report entries (display, ISP, video decode, the AFNC IOAs), the T6001
AGX vote (bit 9) and, out of tree, the ANE clock level of omarchy-ane
(bit 10). The report entries and the AGX vote read-modify-write the word
under rep->lock. A module outside this driver could only map the window
and read-modify-write it without that lock, so a concurrent write of
another bit could be lost.

apple_pmp_report_set_request(report_np, id, enable, no_ack) is the one
writer a client outside this file may use. It calls the same
apple_pmp_report_set_state() the entries use, so it takes the same lock
and does the same acknowledge poll (at most 50 ms). The caller owns the
bit it asks for.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…ment clearing

apple_pmp_report_set_request() lets a client outside pmp-report update one
bit of the PS-REQ word. On the T6001 bit 9 is the AGX request, which the
GPU vote owns: a second writer could clear it while the GPU works, or set
it and leave it set. Refuse it from the export with -EBUSY on a t600x
report. The vote itself writes through apple_pmp_report_set_state() and is
not affected.

Document what a caller must do after -ETIMEDOUT: the request was written
but not acknowledged and stays in the word, so the caller clears it with
@enable false and @no_ack true before giving up.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…st operation

A 9B model at 8192 tokens of context had one gap of up to 111.5 ms between
two decoded tokens in every process, once per run, with the GPU idle for a
good part of it. The 100 ms default hold dropped the request in that gap,
which costs one slow token while the PMP raises memory again. The longest
gap in any other configuration measured was 64 ms.

Raise the default to 250 ms, about 2.2 times the longest gap seen. The cost
is a few hundred milliseconds of the higher memory state after the last
job. pmp_report.agx_vote_idle_ms still overrides it at run time.

The default is one constant, AGX_VOTE_IDLE_MS_DEFAULT, used for both the
parameter and the initial state. The userspace test has a case that rides
through a 112 ms gap with the default and still drops the request after a
long idle.

Signed-off-by: Joshua Warren <816217+joshuawarren@users.noreply.github.com>
…ad only)

Two read-only debugfs views for sampling next to a timing: apple-pmgr-misc exposes the whole clock-generator word of the fabric and DCS system devices, and the PMP report status gains the request and actual words and, on the T6001, the AGX vote state. They answer which hardware state a measurement ran in without a separate investigation. Nothing is written and nothing changes with them absent.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
The status file was created only for reports with fast-die entries or a PMP this driver starts. It now exists for every report: it prints running, and only while the PMP runs the request and actual words (and on the T6001 the AGX vote state). This also keeps the probe free of a reference to the T6001 offsets, which the host test of the T6030 PMP graph extracts and compiles without them.

Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
@joshuaswarren

Copy link
Copy Markdown
Author

Note

🤖 GPT-6.1 Terra responding on behalf of Joshua Warren

This draft remains stacked on #334. Its current head does not include the latest #334 head. I will rebase the reporting change after #334 lands, then verify the resulting diff and request review.

@joshuaswarren

Copy link
Copy Markdown
Author

Closed: superseded upstream. The AGX vote landed on aurora-silicon/linux as #1290; the AGX-hold mechanism was replaced by scoped performance requests (996ef2a); the pmp-report status-file direction was overtaken when the raw DVFS dump was removed (9d5b025); the debugfs views for the fabric and DCS words live on in aurora-silicon/linux#2143. Kernel work continues on the aurora-silicon tree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant