Skip to content

Hold 60 FPS on a Raspberry Pi 5: V3D defaults for the Vulkan backend, draw bounds for lights and shadows, frame pacing - #1154

Open
Babyhamsta wants to merge 3 commits into
OpenH1:masterfrom
Babyhamsta:pi5-perf
Open

Babyhamsta wants to merge 3 commits into
OpenH1:masterfrom
Babyhamsta:pi5-perf

Conversation

@Babyhamsta

Copy link
Copy Markdown

I wanted the game to hold 60 FPS on a Raspberry Pi 5 (8 GB). On master it sat at 30-40 in fights and had to run with HALO_VULKAN_DISCARDS=0 or it would crash in the GPU driver. This is what it took to get there. I play tested it on the Pi while working on it, and the crashes I hit along the way are fixed here.

Nothing here lowers image quality. Every renderer change is on by default only on Broadcom's V3D (vendor 0x14e4) and has its own switch, so other GPUs draw exactly as they do today. The one change every platform gets is the frame pacing fix in the first commit.

Problem

  • With discards on (the default) the game crashed now and then inside v3dv's occlusion query code (v3dv_cmd_buffer_emit_set_query_availability), or the kernel reset the GPU (v3d_clean_caches). v3dv ends every occlusion query with a compute job and a cache clean. So the Pi was stuck on HALO_VULKAN_DISCARDS=0.
  • v3dv gives no early depth test to draws recorded in a secondary command buffer, or to any shader that can discard. Hidden pixels got shaded in full.
  • V3D is a tiler, so a pass costs by the pixels it covers. A plasma bolt's light and an object's shadow are drawn over whole surfaces of the level, and the enhanced lights' shader pays for its loop on every pixel of the level even with no light near.
  • The driver waits more than it has to. A barrier that names the vertex stages holds the next frame's binning until the frame before is rendered, and the kernel makes a job wait for every earlier job that touches the same block of memory.
  • The render thread sat in a fence wait for each frame instead of playing the next frame's commands.
  • With frames paced to the display, a frame that took between 1.5 and 2 refreshes (game and renderer overlapping) was started too late every time, so the game dropped to 30 for as long as the fight lasted.

Fix

Three commits, each message has the details:

  1. Frame pacing (all platforms): a paced frame may take up to two refreshes, and a frame that overran widens the margin.
  2. Front end and game sources, all behind new backend capabilities that nothing sets in this commit: scissor a dynamic light's and a shadow's draws to what they can change, split the lightmap pass by the lights' reach, skip the lights' shader for models no light reaches, drop an alpha test that no texel of the texture can fail, clear the shadow atlas a cell at a time.
  3. Vulkan backend, on by default on V3D only: shader-counted lens flare visibility (no occlusion queries), passes recorded into the main command buffer, passes kept to the area they draw, narrower barriers, per-pass transfer waits, targets with their own memory, GPU completion told to the render thread.

Switches (all in port/linux/vulkan/README.md): HALO_VULKAN_VISIBILITY, HALO_VULKAN_FENCE_WAIT, HALO_VULKAN_SOFT_PASSES, HALO_VULKAN_CLEAR_ALONE, HALO_VULKAN_NARROW_BARRIERS, HALO_VULKAN_PASS_TRANSFERS, HALO_VULKAN_OWN_MEMORY, HALO_VULKAN_ALPHA_TEST_DROP, HALO_VULKAN_DRAW_BOUNDS. Two more are off unless asked for: HALO_VULKAN_FRAMES_AHEAD=1 and HALO_VULKAN_CUTOUT_PREPASS=1 (see Risk).

Verification

Raspberry Pi 5 8 GB, Raspberry Pi OS (trixie), stock Mesa 26.2.6, 640x480, no anti-aliasing. The scene is a recorded demo of the heaviest fight I found, in the pines on Halo (a30), 36 seconds.

Frame rate, no limit (worst second / mean of the 10 worst seconds / mean). The first row is this branch with every switch off and HALO_VULKAN_DISCARDS=0, which measures the same as master itself (52.3 / 59.2 / 70.7 classic). Each row adds to the one above.

classic lights enhanced lights
as master 52.5 / 59.5 / 71.0 35.5 / 39.0 / 47.3
+ discards, VISIBILITY=shader 54.5 / 61.8 / 75.9 36.5 / 39.5 / 49.1
+ FENCE_WAIT=0 56.0 / 64.2 / 78.6 37.3 / 40.6 / 50.5
+ SOFT_PASSES 56.8 / 65.7 / 81.4 38.0 / 41.4 / 52.0
+ CLEAR_ALONE 57.0 / 66.6 / 82.7 37.5 / 41.8 / 52.5
+ NARROW_BARRIERS, PASS_TRANSFERS, OWN_MEMORY 62.0 / 70.9 / 88.2 38.4 / 42.4 / 53.4
+ ALPHA_TEST_DROP 63.0 / 72.7 / 90.6 41.0 / 44.3 / 55.6
+ DRAW_BOUNDS (this branch's defaults) 65.9 / 77.0 / 95.9 45.8 / 56.6 / 75.4

Limited to 60, the way I play it (command below), same fight:

mean FPS seconds under 59 (of 36)
master, classic lights 37.9 36
this branch, classic lights 59.6 to 59.8 over three runs 1 to 4, lowest second 54.4
master, enhanced lights 29.0 36
this branch, enhanced lights 56.2 14

So with classic dynamic lights it holds 60 through that fight apart from a few seconds in the mid 50s. Enhanced lights are a lot closer than they were but not there yet.

Same pixels. The fight's frames, captured at a fixed frame time with lens flares off (their visibility is run to run noise), compared byte for byte:

  • master vs this branch, classic lights: 483 of 484 frames identical
  • master vs this branch, enhanced lights: 409 of 410 identical
  • this branch vs this branch with each switch flipped, one at a time: all identical (459 to 475 frames each)
  • the one frame that differs is the first one captured, a needle's trail right after the demo seeks. It differs between two runs of master as well (394 of 395).

master | this branch | difference x32, enhanced lights:

enhanced

Classic lights:

classic

No crash or GPU reset in any of these runs.

Tests: tools/test_vulkan_passes.py (new cases for the narrow barriers, a rectangle cleared alone, the transfers a pass waits for), tools/test_texture_alpha.py (new: the least alpha against the decoder's own alphas) and the frame pacing test pass on the Pi.

The frames and frame rates were measured at 8a21fee, the commit the demo was recorded on (a demo from there does not play back on current master, with or without this branch). The branch is rebased onto 321e0b4, where it builds on the Pi, the tests above pass and it starts.

Try it on a Pi 5

Display at 1080p60 (wlr-randr --output HDMI-A-2 --mode 1920x1080@60, a reboot can leave it at 4K30), then:

SDL_VIDEO_DRIVER=wayland SDL_VIDEO_WAYLAND_MODE_SCALING=aspect \
HALO_DISPLAY_MODE=fullscreen HALO_RESOLUTION=640x480 HALO_RESOLUTION_SCALING=original \
HALO_ANTI_ALIASING=off HALO_ANISOTROPIC_FILTERING=0 HALO_SHADOW_RESOLUTION=128 \
HALO_DYNAMIC_LIGHTS=classic \
HALO_NO_VSYNC=1 HALO_MAX_FPS=60 HALO_VULKAN_FRAMES_AHEAD=1 \
./halo

HALO_SHOW_FPS=1 shows the counter. No HALO_VULKAN_DISCARDS=0 needed any more.

Risk

  • Only the Linux arm64 build was built and run. I did not build Windows, Android, macOS, iOS or x86-64, and did not try VR or split screen. The renderer changes are off there unless switched on, but the code still has to compile.
  • The pacing change is shared, so every platform that paces by the display gets it. Only checked on the Pi at 60 Hz.
  • HALO_VULKAN_CUTOUT_PREPASS=1 (an alpha-tested model's depth first) is worth more on top (worst second 46.8 to 55.4 with enhanced lights) but is not exact: 214 of 483 frames identical, the rest differ in up to five pixels where a cut away texel lands at exactly the depth of the visible one. That is why it is off unless asked for.

prepass

  • The model test for "no light reaches it" uses the object's tag's bounding sphere with a margin. A model that draws well outside its own sphere could miss a light at its edge. I did not find one.
  • V3D is the only GPU these defaults were measured on. Other tilers might gain from the same switches but I have nothing to test them with.

…that ran long

With display.low_latency a frame starts its budget before its refresh: the
longest of the last 128 frames, and a margin. The budget left out any frame
that took more than a refresh and a half. On a Raspberry Pi 5 in a
firefight the game writes a frame in 9 ms while the renderer still draws
the one before, 17 ms: 26 ms from input to done, and yet a frame each
refresh. Those frames were given too little, started too late for their
refresh and were shown at the next one, so the game sat at 30 for as long
as the fight lasted. The limit is now the refreshes a frame can be started
before its own (FRAME_PACING_QUEUED + 1).

A frame that started on time, ran past the margin and missed its refresh
now widens the margin as well. Counting its cost alone gives the frames
after it exactly that much and no more: where the costs keep creeping up,
each new longest frame missed (about one in two hundred on the Pi).

This is the shared pacing, so it changes every platform that paces by the
display, not only the Pi. frame_pacing_test.c has the new limit's cases.
Checked on a Raspberry Pi 5 at 60 Hz; not run on other hardware.
…not fail, where a backend asks

A tile-based GPU pays for a pass by the pixels it shades. On a Raspberry
Pi 5 (Broadcom's V3D) in a firefight most of a frame went to pixels that
could not change: a plasma bolt's light and an object's shadow are drawn
over whole surfaces of the level, the enhanced lights' shader runs its
loop's set-up for every pixel of the level whether a light is near or not,
and V3D makes no early depth test for a shader that may discard, so every
alpha-tested draw shades its hidden pixels too.

This is the front end's and the game's half of what the Pi needed. Each
part is behind a capability of the backend's (xgpu_backend_capabilities),
and none is set here: the next commit sets them in the Vulkan backend, on
V3D alone.

- draw_bounds: a dynamic light's and a shadow's draws over the level say
  what part of the view they can change (port/linux/game/draw_bounds.c:
  the light's radius, the shadow's volume, and how far past them the
  textures' borders are still sampled), and the front end scissors them to
  it. With display.dynamic_lights enhanced, the lightmap pass's triangles
  are drawn in two parts, those in some light's reach by the lit shader
  and the rest by the shader without the lights' loop; a model no light
  reaches is drawn so too (halo_model_bounds, from the model's sphere at
  the tick and where the frame draws it).
- alpha_test_drop: a draw whose alpha is a texture's alone, no texel of
  which fails the test, is drawn without the test. A texture's least
  alpha is read from its texels (xgpu_texture_minimum_alpha: DXT1, DXT3,
  DXT5, A8R8G8B8; not known for the others, and for Custom Edition's
  channel orders) and kept while they are unchanged.
- clear_alone: each region's cell of the shadow atlas is cleared as its
  turn comes, the rest of the atlas left as it was.
- cutout_prepass: an alpha-tested model's depth first, then its colour
  where the depth is its own. This one is not exact, a pixel the test
  cuts away at exactly the depth of the one seen there is drawn too, so
  no backend asks for it by itself (HALO_VULKAN_CUTOUT_PREPASS=1).

The first three draw the same pixels; the frames compared are in the next
commit's message, which turns them on.

Checked: tools/test_texture_alpha.py (new: the least alpha against the
decoder's own alphas of the same texels, random and shaped blocks of each
format) passes on a Raspberry Pi 5. This commit alone builds there
(ninja linux_arm64). Not built for Windows, Android,
macOS or iOS.
…what they draw, waiting only for what they use

What a Raspberry Pi 5 needed from the Vulkan backend for 60 frames a
second. Each part has a switch (port/linux/vulkan/README.md) and is on by
default on V3D alone (vendor 0x14e4): elsewhere the renderer does as it
did.

- HALO_VULKAN_VISIBILITY=shader: the lens flares' visibility counted by
  the pixel shader. v3dv ends each occlusion query with a compute job and
  a kernel cache clean; with discards on, the game crashed there now and
  then (v3dv_cmd_buffer_emit_set_query_availability) or the kernel reset
  the GPU (v3d_clean_caches), which is why the Pi had to run
  HALO_VULKAN_DISCARDS=0.
- HALO_VULKAN_FENCE_WAIT=0: a thread tells the render thread when the GPU
  has done a frame, and it plays the next frame's commands meanwhile.
- HALO_VULKAN_SOFT_PASSES: a pass's draws kept as a list and recorded into
  the main command buffer. v3dv tests no draw of a secondary command
  buffer by its depth before shading it.
- HALO_VULKAN_CLEAR_ALONE: a pass kept to the part of its target it clears
  and draws.
- HALO_VULKAN_NARROW_BARRIERS, HALO_VULKAN_PASS_TRANSFERS,
  HALO_VULKAN_OWN_MEMORY: a barrier names the vertex stages only after a
  write they read, a pass waits only for the copies whose images it uses,
  and the targets have memory of their own. v3dv makes a frame's binning
  wait for the frame before only when a barrier names the vertex stages,
  and the kernel's driver makes a job wait for every earlier job that
  uses a block of memory it uses.
- HALO_VULKAN_ALPHA_TEST_DROP, HALO_VULKAN_DRAW_BOUNDS: the front end's
  capabilities of the commit before.
- HALO_VULKAN_FRAMES_AHEAD=1 (off by default everywhere): under FIFO, a
  frame started while the one before is still on the GPU.

Measured on a Raspberry Pi 5 (8 GB, Mesa 26.2.6, 640x480, no
anti-aliasing), a recorded fight among the pines of Halo (a30), 36
seconds, no frame limit; the least second, the mean of the ten worst and
the mean, in frames a second. The first line is the switches off with
HALO_VULKAN_DISCARDS=0, which measures as master does (classic: 52.3,
59.2, 70.7); each line adds to the one before.

                                      classic lights      enhanced lights
  as master                           52.5  59.5  71.0    35.5  39.0  47.3
  discards, VISIBILITY=shader         54.5  61.8  75.9    36.5  39.5  49.1
  FENCE_WAIT=0                        56.0  64.2  78.6    37.3  40.6  50.5
  SOFT_PASSES                         56.8  65.7  81.4    38.0  41.4  52.0
  CLEAR_ALONE                         57.0  66.6  82.7    37.5  41.8  52.5
  NARROW_BARRIERS, PASS_TRANSFERS,
  OWN_MEMORY                          62.0  70.9  88.2    38.4  42.4  53.4
  ALPHA_TEST_DROP                     63.0  72.7  90.6    41.0  44.3  55.6
  DRAW_BOUNDS (the defaults)          65.9  77.0  95.9    45.8  56.6  75.4

Limited to 60 under FIFO with HALO_VULKAN_FRAMES_AHEAD=1, classic lights,
the same fight: master 37.9 frames a second, every second under 59; this
branch 59.6 to 59.8, one to four of the 36 seconds under 59 (the least
54.4).

Checked: the fight's frames at a fixed frame time, lens flares off, are
byte for byte master's (8a21fee) with classic lights (483 of 484) and
with enhanced (409 of 410), and the branch's own with each switch the
other way (459 to 475 frames each). The one frame that differs is the
first one captured, a needle's trail just after the demo's seek, which
differs between two runs of master too (394 of 395). With
HALO_VULKAN_CUTOUT_PREPASS=1, 214 of 483 frames are identical and the
rest differ in up to five pixels: it stays off. No crash or GPU reset in
any of the runs. tools/test_vulkan_passes.py passes (new cases: the
narrow barriers, a rectangle cleared alone, the transfers a pass waits
for). Only the Linux arm64 build was built and run; not Windows, Android,
x86-64 or VR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant