Skip to content

kernel: Optional per-VM scene overlap on the S8003 - #157

Merged
henrikklee merged 10 commits into
OrchardLinux:mainfrom
brkosebay:a9-gpu-vm-overlap
Nov 17, 2026
Merged

henrikklee merged 10 commits into
OrchardLinux:mainfrom
brkosebay:a9-gpu-vm-overlap

Conversation

@brkosebay

Copy link
Copy Markdown
Contributor

Depends on #150. Only the last 2 commits are new in this PR.

Summary

First, signal completed-job fences before the S8003 cache-line touch/drain
work. This ordering change stands alone and is the first commit.

Then add powervr.apple_vm_overlap, default off. On S8003, geometry can depend
on its own VM's PM scenes instead of another client's fragment. Keep per-VM
fragment histories with reference-counted fences, and keep the existing
global ordering when the option is off. The submission lock remains.

This does not depend on either fix series or the other performance series.

Commits

  • Signal completed jobs before touching S8003 PM memory
  • Allow opt-in per-VM S8003 scene overlap

Hardware

On the iPad 5, with overlap on, the gap between one client's fragment job
and the next client's fell from 1.20 to 0.53 ms on composited glmark2, with
zero GPU faults. The score went from 511 to 520; that is unconfirmed (one
run each, and glmark2 drifts about 16% run to run here). glmark2's
direct-scanout run has only seven cross-client switches, so there is no
gain there. The option stays off by default.

Tested on Orchard main plus this series and our other pending patches (the J71t charging series and the rest of our iPad 5 development series), as one development kernel.

A10

Only the A9 was tested by us. The fence and per-VM history code is in the
shared Apple layer; overlap is enabled through the chip's vm_scenes hook.
It stays S8003-only because the hardware path differs: the S8003 firmware
gives each VM its own PM sets (up to two scenes), while the T8010 firmware
keeps one PM scene for all clients. Overlapping there would drop the
ordering that protects that shared scene, so the T8010 has no vm_scenes
hook and keeps global ordering whatever the parameter says. The first
commit moves only the S8003 event callback; the T8010 acknowledges events
where it did before.

A10 testers: please report rendering and any GPU fault or hang with
apple_vm_overlap 0 and 1. Both should behave the same on the T8010.

How to test

cat /sys/module/powervr/parameters/apple_vm_overlap            # N (off) by default
glmark2-es2-wayland --fullscreen
echo 1 | sudo tee /sys/module/powervr/parameters/apple_vm_overlap
glmark2-es2-wayland --fullscreen
  • A9: correct rendering and no new GPU fault or hang in sudo dmesg with
    overlap on. The gain is in cross-client switches under a compositor, so
    run another GPU client next to glmark2. Compare scores only in both
    orders (on then off as well as off then on), because one run is noise.
  • A10: identical behavior with 0 and 1 is the expected result.
  • Off switch: echo 0 | sudo tee /sys/module/powervr/parameters/apple_vm_overlap,
    or powervr.apple_vm_overlap=0 (the default) on the kernel command line.

Testing

  • A9 / apple-16k: PASS; image, modules and device trees built; no new warnings.
  • A10 / apple-4k: PASS; image, modules and device trees built; no new warnings.

The stock apple-4k CI config has CONFIG_DRM_POWERVR_APPLE disabled; that
checks the disabled-backend stubs. The apple-16k build enables the Apple
backend and compiles both the S8003 and T8010 implementations.

The branch's CI config is unchanged and the effective config after olddefconfig
matches it byte for byte. Compiler debug output is disabled only for the
local build to fit the host. Reapplying the patch series can emit the
existing PMGR blank-line-at-EOF application warning under either config.
No new compiler warnings are accepted.

…ial pages

On the iPad 5, vkCreateDevice() fails with VK_ERROR_INITIALIZATION_FAILED.
CREATE_HWRT_DATASET returns -EINVAL without a log line when Mesa creates
the dummy render target used during device creation.

The A9 allocates the parameter manager's pages itself. Its free list is
not grown by pvr_free_list_create(), so current_pages stays zero, but
hwrt_init_kernel_structure() still checks it against the minimum.

For chips with own_pm_pages, check initial_num_pages instead. This is
the limit used to size the S8003 parameter manager's render blocks.

Affects the Apple A9 (S8003). The A10 is unchanged.

Assisted-by: LLM
…rejected

After a rejected command stream on the iPad 5, the next exec in another
process oopses in free_bprm with a bad rss-counter. slub_debug reports
"Object already free" on kmalloc-512: the object was allocated and freed
in pvr_fw_cmd_init(), then freed again from pvr_job_put().

When stream parsing fails, pvr_job_process_stream() frees the command.
The caller then releases the job, whose release path frees it again.
Clear the command pointer and length after freeing the rejected command.

Upstream has the same code. Affects every PowerVR device.

Fixes: eaf01ee5ba28 ("drm/imagination: Implement job submission and scheduling")

Assisted-by: LLM
Every geometry and fragment job on the iPad 5 is refused with -EINVAL,
without a log line. The stream parser requires the A10's type-1 register
extension on every Apple chip, while the A9 supplies its registers in
extension type 2.

Put each chip's required extension type and mask beside its stream
definitions and obtain them through an Apple accessor. Keep the A10's
values and leave compute without an Apple must-have entry.

Affects the Apple A9 (S8003). The A10's validation values are unchanged.

Assisted-by: LLM
…ry job

Once rendering works on the iPad 5, every fragment job warns at
WARN_ON(job->paired_job) in pvr_job.c, thousands of times a minute.
Geometry jobs are never freed, and clients hang when destroying a
render target: glmark2 and vkcube run until killed.

Apple queues receive jobs linked by pvr_jobs_link_geom_frag(), but their
free path does not drop the fragment job's paired geometry reference.
That reference also keeps the geometry job's HWRT and context alive.

Follow pvr_queue_free_job(): drop the paired geometry reference and
clear paired_job before dropping the scheduler's reference.

Affects the shared Apple queue used by the A9 (S8003) and A10 (T8010).
The A10 has been build-tested only.

Assisted-by: LLM
A client's first render on the iPad 5 fails without a log line when
another client rendered last. Its geometry job returns -EBUSY, then
its fragment job is refused with "fragment job without a geometry job".

The call that takes a VM's PM sets is inside the dependency choice for
consecutive jobs of the same VM. A new VM can therefore reach submission
without a PM set when the previous fragment belonged to another VM.

Call vm_scenes for every geometry job before selecting its dependency.
Keep the existing prev_frag/last_frag choice and locking context.

Affects the Apple A9 (S8003). The A10 has no vm_scenes callback and is
unchanged.

Assisted-by: LLM
The iPad 5's rejected geometry and fragment streams return -EINVAL
without explaining which check failed. Applications fail to render,
and the kernel gives no values with which to diagnose the rejection.

Report malformed headers, invalid destination entries, short input,
unsupported extensions and missing must-have bits at the rejecting
check. Include the command type, entry or extension, and checked values.
Use one rate-limited warning per rejected stream; callers log short
reads so that propagated errors do not produce duplicate messages.

Keep successful parsing and error returns unchanged. Affects every
PowerVR device.

Assisted-by: LLM
Render failures on the iPad 5 can leave a fragment job without its
geometry job. S8003 submission errors otherwise return silently, so
the kernel does not say which engine or stage failed.

Add one rate-limited warning at each failing submission stage with the
engine, error, VM and reason. Separate the PM pool build and record
slot wait checks so that their errors can be distinguished.

Keep submission and recovery behaviour unchanged. A fragment without
its geometry scene is still refused.

Affects the Apple A9 (S8003). The A10 is unchanged.

Assisted-by: LLM
…e it loads

The iPad 5 loses Firefox's WebGL context under multiple GPU clients.
Firmware slot loads return -EIO, geometry jobs fail, and their fragment
jobs are refused. Single-client runs do not hit the failure.

VM map and unmap flushes take tag_lock but not submit_lock. They can
bump a slot's host tag while DC_SLOT loads its root. With two or more
bumps, firmware can copy an intermediate tag that matches neither the
loaded tag nor the current host tag, falsely failing the load check.

Mark each slot as loading under tag_lock while publishing its root and
tag. Defer flushes on those slots, verify the exact loaded tag, then
clear loading and apply any pending bump through the shared helper on
both success and failure. submit_lock keeps the VM's job behind the
pending bump, so firmware still invalidates stale translations before
using the slot. Report the slot, loaded, firmware and current tags on
a mismatch.

Affects: Apple A9 (S8003). The A10 is unchanged.

Assisted-by: LLM
…M memory

The S8003 event callback reads firmware and PM cache lines before completed
job fences are signalled. That host work delays the next scheduled job even
though completion stamps are already visible through coherent mappings.

Signal the fences and retire their runtime-PM references first, then run
the chip event callback. Keep the A10 event acknowledgement before its
retirement-watermark check, as before. No submission dependency changes.

Affects: Apple A9 (S8003). The A10 acknowledgement order is unchanged.

Assisted-by: LLM
Composited S8003 workloads serialize one client's geometry behind another
client's fragment, even though their PM sets are private to their VMs.

Keep each VM's last two fragment fences and add apple_vm_overlap, default
off. When enabled on a chip with per-VM scenes, geometry waits on its own
VM's previous fragment (or last fragment for one scene). Keep the global
scene dependencies with the option off and for chips without per-VM sets.
Reference and release the per-VM history with the existing VM lifetime.

Affects: Apple A9 (S8003). T8010 has no per-VM scene hook and keeps its
global dependencies regardless of the parameter.

Assisted-by: LLM
@henrikklee
henrikklee merged commit 4ae9409 into OrchardLinux:main Nov 17, 2026
11 checks passed
shaynelegault pushed a commit to shaynelegault/hardware that referenced this pull request Nov 19, 2026
OrchardLinux#151, OrchardLinux#152, OrchardLinux#156 and OrchardLinux#157 each numbered their patches after OrchardLinux#150 as if
they were the only follow-up. Number them in merge order: stability
(0034-0042), firmware interrupt and PM (0043-0049), eviction (0050-0057),
GPU time (0058), scene overlap (0059-0060).

Assisted-by: LLM
shaynelegault pushed a commit to shaynelegault/hardware that referenced this pull request Nov 19, 2026
OrchardLinux#151, OrchardLinux#152, OrchardLinux#153 and OrchardLinux#157 were each written on OrchardLinux#150 alone. Combined:

- gpu/0051: the BO memory cap charges a native BO at creation, also when
  its pages come later.
- gpu/0053: one record of the last two PM scenes, shared by the cache-line
  touch mask and the eviction veto.
- gpu/0054: the restore path and VM_MAP size their page tables from the
  device address, as gpu/0042 made VM_MAP do.
- gpu/0055: pools are prepared in the submit ioctl and sized for the
  firmware's feedback too; the job worker only switches to a prepared
  pool, between scenes, and never allocates.
- gpu/0058: completions are signalled before the S8003 rings are
  touched, in the poll worker of gpu/0041 and gpu/0043.

Two fixes on top: one submission order lock per device instead of one
for every PowerVR device (gpu/0059), and the job worker allocates as
before on chips that cannot evict (gpu/0060), so the A10 is unchanged.

Left out: GPU time in debugfs (OrchardLinux#156; belongs in per-client fdinfo
accounting) and the default-off scene overlap switch (OrchardLinux#157's second
patch). The s8003 display DART patch becomes devices/s8000/0017; main
took 0016 meanwhile.

Assisted-by: LLM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants