Repository navigation
Hold 60 FPS on a Raspberry Pi 5: V3D defaults for the Vulkan backend, draw bounds for lights and shadows, frame pacing - #1154
Open
Babyhamsta wants to merge 3 commits into
Open
Babyhamsta wants to merge 3 commits into
Babyhamsta wants to merge 3 commits into
Conversation
…that ran long With display.low_latency a frame starts its budget before its refresh: the longest of the last 128 frames, and a margin. The budget left out any frame that took more than a refresh and a half. On a Raspberry Pi 5 in a firefight the game writes a frame in 9 ms while the renderer still draws the one before, 17 ms: 26 ms from input to done, and yet a frame each refresh. Those frames were given too little, started too late for their refresh and were shown at the next one, so the game sat at 30 for as long as the fight lasted. The limit is now the refreshes a frame can be started before its own (FRAME_PACING_QUEUED + 1). A frame that started on time, ran past the margin and missed its refresh now widens the margin as well. Counting its cost alone gives the frames after it exactly that much and no more: where the costs keep creeping up, each new longest frame missed (about one in two hundred on the Pi). This is the shared pacing, so it changes every platform that paces by the display, not only the Pi. frame_pacing_test.c has the new limit's cases. Checked on a Raspberry Pi 5 at 60 Hz; not run on other hardware.
…not fail, where a backend asks A tile-based GPU pays for a pass by the pixels it shades. On a Raspberry Pi 5 (Broadcom's V3D) in a firefight most of a frame went to pixels that could not change: a plasma bolt's light and an object's shadow are drawn over whole surfaces of the level, the enhanced lights' shader runs its loop's set-up for every pixel of the level whether a light is near or not, and V3D makes no early depth test for a shader that may discard, so every alpha-tested draw shades its hidden pixels too. This is the front end's and the game's half of what the Pi needed. Each part is behind a capability of the backend's (xgpu_backend_capabilities), and none is set here: the next commit sets them in the Vulkan backend, on V3D alone. - draw_bounds: a dynamic light's and a shadow's draws over the level say what part of the view they can change (port/linux/game/draw_bounds.c: the light's radius, the shadow's volume, and how far past them the textures' borders are still sampled), and the front end scissors them to it. With display.dynamic_lights enhanced, the lightmap pass's triangles are drawn in two parts, those in some light's reach by the lit shader and the rest by the shader without the lights' loop; a model no light reaches is drawn so too (halo_model_bounds, from the model's sphere at the tick and where the frame draws it). - alpha_test_drop: a draw whose alpha is a texture's alone, no texel of which fails the test, is drawn without the test. A texture's least alpha is read from its texels (xgpu_texture_minimum_alpha: DXT1, DXT3, DXT5, A8R8G8B8; not known for the others, and for Custom Edition's channel orders) and kept while they are unchanged. - clear_alone: each region's cell of the shadow atlas is cleared as its turn comes, the rest of the atlas left as it was. - cutout_prepass: an alpha-tested model's depth first, then its colour where the depth is its own. This one is not exact, a pixel the test cuts away at exactly the depth of the one seen there is drawn too, so no backend asks for it by itself (HALO_VULKAN_CUTOUT_PREPASS=1). The first three draw the same pixels; the frames compared are in the next commit's message, which turns them on. Checked: tools/test_texture_alpha.py (new: the least alpha against the decoder's own alphas of the same texels, random and shaped blocks of each format) passes on a Raspberry Pi 5. This commit alone builds there (ninja linux_arm64). Not built for Windows, Android, macOS or iOS.
…what they draw, waiting only for what they use
What a Raspberry Pi 5 needed from the Vulkan backend for 60 frames a
second. Each part has a switch (port/linux/vulkan/README.md) and is on by
default on V3D alone (vendor 0x14e4): elsewhere the renderer does as it
did.
- HALO_VULKAN_VISIBILITY=shader: the lens flares' visibility counted by
the pixel shader. v3dv ends each occlusion query with a compute job and
a kernel cache clean; with discards on, the game crashed there now and
then (v3dv_cmd_buffer_emit_set_query_availability) or the kernel reset
the GPU (v3d_clean_caches), which is why the Pi had to run
HALO_VULKAN_DISCARDS=0.
- HALO_VULKAN_FENCE_WAIT=0: a thread tells the render thread when the GPU
has done a frame, and it plays the next frame's commands meanwhile.
- HALO_VULKAN_SOFT_PASSES: a pass's draws kept as a list and recorded into
the main command buffer. v3dv tests no draw of a secondary command
buffer by its depth before shading it.
- HALO_VULKAN_CLEAR_ALONE: a pass kept to the part of its target it clears
and draws.
- HALO_VULKAN_NARROW_BARRIERS, HALO_VULKAN_PASS_TRANSFERS,
HALO_VULKAN_OWN_MEMORY: a barrier names the vertex stages only after a
write they read, a pass waits only for the copies whose images it uses,
and the targets have memory of their own. v3dv makes a frame's binning
wait for the frame before only when a barrier names the vertex stages,
and the kernel's driver makes a job wait for every earlier job that
uses a block of memory it uses.
- HALO_VULKAN_ALPHA_TEST_DROP, HALO_VULKAN_DRAW_BOUNDS: the front end's
capabilities of the commit before.
- HALO_VULKAN_FRAMES_AHEAD=1 (off by default everywhere): under FIFO, a
frame started while the one before is still on the GPU.
Measured on a Raspberry Pi 5 (8 GB, Mesa 26.2.6, 640x480, no
anti-aliasing), a recorded fight among the pines of Halo (a30), 36
seconds, no frame limit; the least second, the mean of the ten worst and
the mean, in frames a second. The first line is the switches off with
HALO_VULKAN_DISCARDS=0, which measures as master does (classic: 52.3,
59.2, 70.7); each line adds to the one before.
classic lights enhanced lights
as master 52.5 59.5 71.0 35.5 39.0 47.3
discards, VISIBILITY=shader 54.5 61.8 75.9 36.5 39.5 49.1
FENCE_WAIT=0 56.0 64.2 78.6 37.3 40.6 50.5
SOFT_PASSES 56.8 65.7 81.4 38.0 41.4 52.0
CLEAR_ALONE 57.0 66.6 82.7 37.5 41.8 52.5
NARROW_BARRIERS, PASS_TRANSFERS,
OWN_MEMORY 62.0 70.9 88.2 38.4 42.4 53.4
ALPHA_TEST_DROP 63.0 72.7 90.6 41.0 44.3 55.6
DRAW_BOUNDS (the defaults) 65.9 77.0 95.9 45.8 56.6 75.4
Limited to 60 under FIFO with HALO_VULKAN_FRAMES_AHEAD=1, classic lights,
the same fight: master 37.9 frames a second, every second under 59; this
branch 59.6 to 59.8, one to four of the 36 seconds under 59 (the least
54.4).
Checked: the fight's frames at a fixed frame time, lens flares off, are
byte for byte master's (8a21fee) with classic lights (483 of 484) and
with enhanced (409 of 410), and the branch's own with each switch the
other way (459 to 475 frames each). The one frame that differs is the
first one captured, a needle's trail just after the demo's seek, which
differs between two runs of master too (394 of 395). With
HALO_VULKAN_CUTOUT_PREPASS=1, 214 of 483 frames are identical and the
rest differ in up to five pixels: it stays off. No crash or GPU reset in
any of the runs. tools/test_vulkan_passes.py passes (new cases: the
narrow barriers, a rectangle cleared alone, the transfers a pass waits
for). Only the Linux arm64 build was built and run; not Windows, Android,
x86-64 or VR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I wanted the game to hold 60 FPS on a Raspberry Pi 5 (8 GB). On master it sat at 30-40 in fights and had to run with
HALO_VULKAN_DISCARDS=0or it would crash in the GPU driver. This is what it took to get there. I play tested it on the Pi while working on it, and the crashes I hit along the way are fixed here.Nothing here lowers image quality. Every renderer change is on by default only on Broadcom's V3D (vendor
0x14e4) and has its own switch, so other GPUs draw exactly as they do today. The one change every platform gets is the frame pacing fix in the first commit.Problem
v3dv_cmd_buffer_emit_set_query_availability), or the kernel reset the GPU (v3d_clean_caches). v3dv ends every occlusion query with a compute job and a cache clean. So the Pi was stuck onHALO_VULKAN_DISCARDS=0.Fix
Three commits, each message has the details:
Switches (all in
port/linux/vulkan/README.md):HALO_VULKAN_VISIBILITY,HALO_VULKAN_FENCE_WAIT,HALO_VULKAN_SOFT_PASSES,HALO_VULKAN_CLEAR_ALONE,HALO_VULKAN_NARROW_BARRIERS,HALO_VULKAN_PASS_TRANSFERS,HALO_VULKAN_OWN_MEMORY,HALO_VULKAN_ALPHA_TEST_DROP,HALO_VULKAN_DRAW_BOUNDS. Two more are off unless asked for:HALO_VULKAN_FRAMES_AHEAD=1andHALO_VULKAN_CUTOUT_PREPASS=1(see Risk).Verification
Raspberry Pi 5 8 GB, Raspberry Pi OS (trixie), stock Mesa 26.2.6, 640x480, no anti-aliasing. The scene is a recorded demo of the heaviest fight I found, in the pines on Halo (a30), 36 seconds.
Frame rate, no limit (worst second / mean of the 10 worst seconds / mean). The first row is this branch with every switch off and
HALO_VULKAN_DISCARDS=0, which measures the same as master itself (52.3 / 59.2 / 70.7 classic). Each row adds to the one above.VISIBILITY=shaderFENCE_WAIT=0SOFT_PASSESCLEAR_ALONENARROW_BARRIERS,PASS_TRANSFERS,OWN_MEMORYALPHA_TEST_DROPDRAW_BOUNDS(this branch's defaults)Limited to 60, the way I play it (command below), same fight:
So with classic dynamic lights it holds 60 through that fight apart from a few seconds in the mid 50s. Enhanced lights are a lot closer than they were but not there yet.
Same pixels. The fight's frames, captured at a fixed frame time with lens flares off (their visibility is run to run noise), compared byte for byte:
master | this branch | difference x32, enhanced lights:
Classic lights:
No crash or GPU reset in any of these runs.
Tests:
tools/test_vulkan_passes.py(new cases for the narrow barriers, a rectangle cleared alone, the transfers a pass waits for),tools/test_texture_alpha.py(new: the least alpha against the decoder's own alphas) and the frame pacing test pass on the Pi.The frames and frame rates were measured at 8a21fee, the commit the demo was recorded on (a demo from there does not play back on current master, with or without this branch). The branch is rebased onto 321e0b4, where it builds on the Pi, the tests above pass and it starts.
Try it on a Pi 5
Display at 1080p60 (
wlr-randr --output HDMI-A-2 --mode 1920x1080@60, a reboot can leave it at 4K30), then:HALO_SHOW_FPS=1shows the counter. NoHALO_VULKAN_DISCARDS=0needed any more.Risk
HALO_VULKAN_CUTOUT_PREPASS=1(an alpha-tested model's depth first) is worth more on top (worst second 46.8 to 55.4 with enhanced lights) but is not exact: 214 of 483 frames identical, the rest differ in up to five pixels where a cut away texel lands at exactly the depth of the visible one. That is why it is off unless asked for.