Skip to content

mm: populate a subpage window's uffd-wp markers with the faulting page - #2225

Open
jgoyvaerts wants to merge 2 commits into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/mm-subpage-uffd-wp
Open

jgoyvaerts wants to merge 2 commits into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/mm-subpage-uffd-wp

Conversation

@jgoyvaerts

Copy link
Copy Markdown
Contributor

In an mm whose user page size is smaller than the native page (CONFIG_MM_SUBPAGE: 4K processes on a 16K kernel, such as everything running under FEX), a write fault on unpopulated anonymous memory fills the whole native window from one fresh native page. Each 4K page lands at its own offset, so the window is physically in VA order.

A uffd-wp marker (UFFD_FEATURE_WP_UNPOPULATED) is unpopulated memory too, but anon_subpage_missing_mask() counted only none PTEs as missing.

  • Effect: in a VMA write-protected before its first touch, every neighbour holds a marker, so each first write took a native page of its own for one 4K page. That is four times the memory, and no window was ever one native page in order.
  • Who it hurts: a device that maps native pages can then use none of it. One example is a GPU with 16K pages taking user memory through GEM_USERPTR. A write-watch built on PAGEMAP_SCAN sets up exactly this kind of memory; we hit it with one under FEX.

This change counts a uffd-wp marker in a uffd-wp VMA as missing, and gives the neighbour it stood for the uffd-wp bit, as the faulting page already gets from its own marker. The populated neighbours stay write-protected and unwritten for PAGEMAP_SCAN. Markers in a VMA that is no longer registered are left as they were.

The second commit adds a case to tools/testing/selftests/arm64/user4k/uffd-wp-init.c. The existing markers() case reads before it writes, so its first touch maps the zero page and never reaches this path. The new case covers a native-aligned window, write-protected with WP_ASYNC | WP_UNPOPULATED before any touch, then written once in its second 4K page:

  • it checks that all four 4K pages are present in one native page, in order;
  • it checks that the three neighbours carry the uffd-wp bit and the written page does not.

Tested on the J616s (M4 Pro, 16K kernel)

  • Probe in a 4K process: 8 MiB of anonymous memory, write-protected through userfaultfd with WP_ASYNC | WP_UNPOPULATED before its first touch, then written once per 4K page. Before, 0 of 512 16K blocks were one native page in order; after, 512 of 512. Memory registered after its first touch, or never registered, gave 512 of 512 either way. PAGEMAP_SCAN's written/unwritten report stayed exact, native and 4K.
  • The selftest: passes in both its 16K and 4K worker modes (all 29 cases):
    ok - uffd ABI 4K unpopulated WP first write: one native page, neighbours WP
    UFFD CONTRACT PASS
    
  • In use: this has run in our daily kernels since 11-16.

On this branch, mm/ builds with W=1 and no warnings, and checkpatch reports 0 errors and 0 warnings on both commits.

Not done

  • Read-before-write memory: memory that is read before it is written is unchanged. The read maps the zero page, and the copy on write lands in any free COW-pool slot.
  • Negative run: the new selftest case was not run on a kernel without the fix. Without it, page 0 stays a marker and the case's present-bit check fails.

🤖 Generated with Claude Code

In an mm whose user page size is smaller than the native page, a write
fault on unpopulated anonymous memory fills the whole native window
from one fresh native page, each 4K page at its own offset. That keeps
the window physically in VA order, and costs one native page for four
user pages.

A uffd-wp marker (UFFD_FEATURE_WP_UNPOPULATED) is unpopulated memory
too, but anon_subpage_missing_mask() counted only none PTEs as missing.
In a VMA write-protected before its first touch, every neighbour holds
a marker, so each first write took a native page of its own for one
4K page: four times the memory, and no window was ever one native page
in order. A device that maps native pages (a GPU with 16K pages taking
user memory through GEM_USERPTR) can then use none of it.

Count a uffd-wp marker in a uffd-wp VMA as missing, and give the
neighbour it stood for the uffd-wp bit, as the faulting page already
gets it from its own marker: the populated neighbours stay write
protected and unwritten for PAGEMAP_SCAN. Markers in a VMA that is no
longer registered are left as they were.

Measured with a probe in a 4K process on an M4 Pro (16K kernel): 8 MiB
of anonymous memory, write-protected through userfaultfd with
WP_ASYNC | WP_UNPOPULATED before its first touch, then written once per
4K page. Before, 0 of 512 16K blocks were one native page in order;
memory registered after its first touch, or never registered, gave
512 of 512.

Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
Add a uffd-wp-init case that write-protects a native-aligned window with
WP_ASYNC | WP_UNPOPULATED before any touch, then writes one byte to its
second user page. pagemap must then show every user page of the window
present in one native page in VA order, with the uffd-wp bit kept on the
neighbours and cleared on the written page. markers() never reached this
path: its window is not native-aligned, and its first touch is a read
that maps the zero page, so its write is a copy-on-write fault.

Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Assisted-by: LLM
@jgoyvaerts

Copy link
Copy Markdown
Contributor Author

Base control for the full arm64-user4k stall reported on #2114 (the same mm change on base 677be39, head 4c15fea), carried over here as noted when #2114 was closed. @DjDeveloperr asked for it there.

Setup: arm64_user4k.config + USERFAULTFD=y, booted with user4k=on arm64.user4k_test=1, KVM qemu with 1 vCPU, each run capped at 1500 s. Without arm64.user4k_test=1 the IOMMUFD tests skip, and both SHAs finish identically in about 1 s.

Results:

  • Bare base 677be39: stalls in user4k_iommufd_lifetime_test in 4 of 4 runs.
  • drm/asahi: GEM_USERPTR in 4K (user4k) processes #2114's head: stalls in the same test with the same signature. Tests 1–60 give identical results to base, with 0 failures.
  • The mm commit alone (695bc5e): behaves like base.
  • Which granule case hangs (4K or 64K) varies between runs of the same binary.

Signature: the test kthread loops forever on a level-2 write translation fault at TEST_VA in the first copy_to_user() after vm_mmap() (ESR 0x96000046, FAR 0x400000). After KUnit's per-test timeout, it then blocks in kthread_stop(), so user4k_iommufd_rollback_test never runs.

Why this change isn't involved:

So the stall is pre-existing and independent of #2225/#2226.

user4k_move_migration_race_test (arm64-user4k-nonlinear-swap) also fails on both SHAs, with moves 1 == initial 1. That failure is pre-existing too.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed as core mm. The change is right and contained:

  • It only reaches do_anonymous_subpage(). With no VM_UFFD_WP the mask and the PTEs written are identical to base: every non-target slot in current_mask is then pte_none, so the new marker test is false. Native-16K processes and non-uffd users see no change.
  • The PTL-held recompute (mm/memory.c:6944) and the per-neighbour marker test in the set loop (6972-6973) both read under the PTL, so they agree. A racing population only shrinks current_mask, and the dense-path abort is skipped for armed VMAs, so nothing is lost.
  • arm64 pte_mkuffd() write-protects, so a populated neighbour behaves like the marker it replaced: a WP fault in sync mode, a bit clear in async mode, and the marker comes back on a zap.

The stall control you posted settles #2114's question: base stalls 4/4, and the mm hunk here is byte-identical to 695bc5e. By construction this cannot reach a test that registers no userfaultfd.

Before merge, please post the negative run for the new case. Same uffd-wp-init binary, 4K worker mode, on a 16K kernel without 834e0e6 (0abc2f3 or a fleet kernel without the fix). I expect marker_window to fail at physical(a, ps) on page 0 with the earlier cases passing, then 29/29 on this head.

Two small things, neither blocking:

  1. Match the marker exactly (softleaf_to_marker(...) == PTE_MARKER_UFFD_WP) rather than "has the UFFD_WP bit". handle_pte_marker() ranks GUARD and POISONED above UFFD_WP, and this path would turn such a slot into a zero page. Nothing creates the combination today, so this is hardening.
  2. One sentence in the commit message: neighbours are now present (RSS, PAGE_IS_PRESENT), and a UFFDIO_COPY into one in a WP-only VMA returns -EEXIST where upstream would accept it. Both are already true for none neighbours on this path.

Optional for the test: after the first write, write a[0] and check that its bit 57 clears, pages 2 and 3 keep theirs, and physical() is unchanged.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, the change reads right as core mm: it only reaches do_anonymous_subpage(), and without VM_UFFD_WP the PTEs written are the same as before.

One request before merge: could you add the uffd-wp case as a test that fails on a kernel without this fix (a subpage window registered for write-protect, first touch on one subpage, then check that the other subpages still report the marker)? A selftest under tools/testing/selftests/mm/ or a small reproducer in the PR body is fine, as long as it shows the before/after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants