Skip to content

hk: VK_EXT_memory_priority and VK_EXT_pageable_device_local_memory - #243

Merged
DjDeveloperr merged 2 commits into
aurora-wipfrom
j716c/hk-memory-priority
Nov 13, 2026
Merged

DjDeveloperr merged 2 commits into
aurora-wipfrom
j716c/hk-memory-priority

Conversation

@erikwb

@erikwb erikwb commented Nov 12, 2026 •

Copy link
Copy Markdown

Problem

Under memory pressure the kernel may release the memory of an idle VM's
private BOs, and the VM's next submission then waits to get it back. An
application has no way to say which allocations should stay resident. The
kernel change in aurora-silicon/linux#1610 adds DRM_IOCTL_ASAHI_GEM_RESIDENCY
to ask that an object's memory be released last.

Change

  • include/drm-uapi/asahi_drm.h gains the GEM_RESIDENCY ioctl (the id
    after GEM_USERPTR), its struct and flag, and
    DRM_ASAHI_FEATURE_GEM_RESIDENCY (bit 35), as in drm/asahi: GEM_RESIDENCY, keep an object's memory resident linux#1610.
  • Memory priority 1.0 asks the kernel to release a memory object's BO after
    the VM's other BOs when it reclaims the VM's idle memory, and only if
    theirs was not enough (a hint: see drm/asahi: GEM_RESIDENCY, keep an object's memory resident linux#1610 for what
    reclaim reaches),
    whether given at allocation (VkMemoryPriorityAllocateInfoEXT) or later
    (vkSetDeviceMemoryPriorityEXT); any other priority returns it to the
    kernel's default policy, the only distinction the kernel makes. Freeing
    the memory object clears the request, since the BO may be reused from the
    BO cache. A refused request is logged and leaves the priority as it was.
  • Both extensions and their features are advertised only when GET_PARAMS
    reports DRM_ASAHI_FEATURE_GEM_RESIDENCY (and not over virtio).

Testing

Hardware: M5 Max MacBook Pro (J716c), kernel 7.3.0-ewb121+ (has
GEM_RESIDENCY and GEM_USERPTR), Mesa 26.3.0-devel (aurora-wip 1f629c9
plus this commit; Honeykrisp built alone). Builds with no new warnings.

Vulkan CTS with deqp-runner (2643 cases: dEQP-VK.memory.pageable_allocation.*,
dEQP-VK.memory.binding.priority{,_dynamic}.*, dEQP-VK.memory.allocation.basic.*,
dEQP-VK.api.info.get_physical_device_properties2.* and
vulkan1p2_limits_validation.*):

Pass Skip
Base 1404 1239
This branch 1718 925
This branch, feature bit masked out 1404 1239

The 314 new passes are all 202 pageable_allocation and all 112
binding.priority and binding.priority_dynamic cases. With the feature bit
masked out, the results equal the base case for case.

The 2443-case list of #242 (dEQP-VK.memory.external_memory_host.*,
memory.allocation.basic.*, memory.mapping.suballocation.full.*, the
external memory info cases, the properties and limits cases above and
dEQP-VK.api.device_init.create_device*) gives 1510 passes and 933 skips on
both the base and this branch, case for case.

The CTS priorities stay below 1.0, so a small Vulkan program checked the
residency request itself: a 64 MiB device-local allocation at priority 1.0
raises the kernel's user_kept (GPU gpu_memory in sysfs) from 0 to
67108864; setting the priority to 0.5, back to 1.0 and freeing the memory
give 0, 67108864 and 0, with four GEM_RESIDENCY calls that all succeed.
With the feature bit masked out the program sees memoryPriority and
pageableDeviceLocalMemory false and no GEM_RESIDENCY call is made.

The masked runs used an LD_PRELOAD shim that clears the bit from
DRM_IOCTL_ASAHI_GET_PARAMS, which is how the driver sees a kernel without
GEM_RESIDENCY.

Depends on

Not covered

  • Behaviour under real memory pressure (the kernel's release and restore)
    was not exercised from Mesa.

🤖 Generated with Claude Code

@RyanTheTide

Copy link
Copy Markdown

@erikwb the driver side reads correctly. This now conflicts in include/drm-uapi/asahi_drm.h with #242, which merged. Please rebase, keeping both sets of additions, and sync the header from aurora-silicon/linux#1610 once that merges so the ioctl id and feature bit 35 match what the kernel ships. Minor: EXT_pageable_device_local_memory sits out of order in the extension table.

Under memory pressure the kernel may release the memory of an idle VM's
private BOs, which its next submission then waits to get back. Memory
priority 1.0 now asks the kernel to release a memory object's BO after
the VM's other BOs when it reclaims the VM's idle memory, and only if
theirs was not enough (DRM_IOCTL_ASAHI_GEM_RESIDENCY, a hint), whether
given at allocation
(VkMemoryPriorityAllocateInfoEXT) or later (vkSetDeviceMemoryPriorityEXT);
any other priority returns it to the kernel's default policy. That is
the only distinction the kernel makes, so it is the whole mapping.
Freeing the memory object clears the request, since the BO may be
reused from the BO cache. A refused request is logged and leaves the
priority as it was (the call cannot fail).

Both extensions and their features are advertised only when GET_PARAMS
reports DRM_ASAHI_FEATURE_GEM_RESIDENCY (and not over virtio). The
include/drm-uapi copy gains the GEM_RESIDENCY ioctl, its struct and
flag, and feature bit 35.

Tested on the M5 Max with a kernel that has GEM_RESIDENCY:
dEQP-VK.memory.pageable_allocation.* 202 pass and
dEQP-VK.memory.binding.priority{,_dynamic}.* 112 pass (all skipped
without this change). A 64 MiB device-local allocation at priority 1.0
raises the kernel's user_kept count by 64 MiB; setting it to 0.5, back to
1.0 and freeing it give 0, 64 MiB and 0. With the feature bit masked out
of GET_PARAMS, as on a kernel without it, neither extension nor feature is
listed, no GEM_RESIDENCY call is made, and those cases are skipped as
before. dEQP-VK.memory.allocation.basic.* is unchanged.

Developed with Claude Code.
Signed-off-by: Erik Bourget <erik@ebourget.net>
@erikwb
erikwb force-pushed the j716c/hk-memory-priority branch from 565e4f7 to fc1d7d9 Compare November 13, 2026 03:07
@erikwb

erikwb commented Nov 13, 2026

Copy link
Copy Markdown
Author

Thanks. Rebased onto aurora-wip with #242 (fc1d7d9):

  • asahi_drm.h keeps both sets of additions. The GEM_RESIDENCY and GEM_USERPTR parts now match drm/asahi: GEM_RESIDENCY, keep an object's memory resident linux#1610's current head: DRM_ASAHI_GEM_RESIDENCY is the enumerator after GEM_USERPTR (no explicit offset), the feature bit is 35, and the struct and flag are identical. If #1610 changes before it merges I'll resync.
  • EXT_pageable_device_local_memory moved into order in the extension table.

Retested on the rebased base (the testing section has the new numbers): the priority CTS cases pass, the results with the feature bit masked equal the base case for case, and #242's external_memory_host list is unchanged. Heads-up: #259 adds feature bit 36 at the same place in the header, so whichever of the two lands second will need a trivial rebase.

mesa#264 added DRM_ASAHI_FEATURE_COMPUTE_VISIBLE_COMPLETION (bit 34) at the
place in include/drm-uapi/asahi_drm.h where this branch adds
DRM_ASAHI_FEATURE_GEM_RESIDENCY (bit 35). Both are kept, in bit order.
No other change.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr

Copy link
Copy Markdown

I pushed 6abda25d7909, a merge of aurora-wip (164341fa583) that resolves the include/drm-uapi/asahi_drm.h conflict with #264: its bit 34 comes first, then this PR's bit 35. Your commits and the PR's +/- lines are unchanged.

aurora-silicon/linux#1610 is merged (954a6a74). Its GEM_RESIDENCY ioctl, struct, flag and bit 35 match this header byte for byte, and an independent review of the Mesa side found no blocker.

Next steps, then the merge:

  • local CI (the asahi workflow);
  • the priority/pageable CTS on the M5 Pro, with a kernel that has GEM_RESIDENCY.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving and merging at 6abda25d7909.

  • Independent review (with linux#1610): no blocker.
    • The header hunks are byte-identical to the kernel's, which is merged as 954a6a74.
    • Priority 1.0 keeps an object resident; any other value returns it to the default.
    • Freeing a memory object clears its request before the BO goes back to the cache.
    • A refused request is logged, and the priority stays as it was.
    • Both extensions are advertised only with bit 35, and not over virtio.
  • Local CI on the M5 Pro (VK CTS lane slot m5-vkcts-mg243c): the arm is this head merged into aurora-wip 2e14956a88a (0d86debb055, tree 1397fb2b2196).
    • The asahi workflow builds clean.
    • meson test --suite asahi: 131 Ok, 0 Fail.
  • Hardware:
    • An earlier J714s run of this series, on a kernel with GEM_RESIDENCY, gave priority/pageable 426/426.
    • Your J716c rerun after the rebase passes the priority CTS cases.
    • The only code change since then is the conflict-only merge I pushed.

#259 adds bit 36 at the same place, so it needs one more conflict-only resolution after this lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants