[peiikai]

For #602: the game's ambient occlusion is soft and flickers around objects at high resolutions.

What changed

Since #356 the AO stays at the Switch's size (480x270, blurred, plus a 240x135 level) at every render scale, so on a large frame it is the Switch's AO stretched up.

A new Video setting, Ambient occlusion: Default / High (default Default, the Switch's AO; ini ambient_occlusion = switch | high). High draws the AO at the size of the depth it is built from: 2x the render scale, clamped between the Switch's size and twice it. That is 960x540 at 1x and above, the Switch's size below 0.5x. The cost is 4x the AO pass at most, still a quarter of 1080p.

Scaling the targets alone is what broke #356. The AO shader takes its 4x4 sample-rotation index from uv / texel size with the game's texel size, and the blur's tap offsets are in uv of the Switch's size. Two shader patches in maxwell/translate.c (ao_grid_fix, checked against the original instruction words like light_tiles_fix) give both the real pixel grid:

• AO pass: the pixel index (pc 0x50 / 0x58) is multiplied by the target's scale (rscale[0]).
• AO blur: the shader is the 5-tap Gaussian the bloom also uses, so the renderer flags only the two blurs into the AO's 480-wide target (rscale[65].y = AO scale, 0 for every other draw). Their tap offsets around the centre tap are divided by it, which is the Switch's blur in pixels.

At the Switch's size both patches change nothing. The 240x135 level keeps the game's offsets.

Textures: the AO targets and their quarter-size depth-stencil are scalable again, with a scale of their own (tex_ao, tex_alloc_x/y) on both Metal (metal.m) and Vulkan (vulkan/textures.c). They reallocate when the setting changes. Dynamic resolution goes by the allocation scale, so its steps don't reallocate them.

Test switches (docs/debug-hooks.md): LARECOMP_AO_SCALE=<n> (fixed scale), LARECOMP_AO_NO_GRID=1 (A/B for the patches), LARECOMP_AO_SCALED=1 (the pre-#356 path, unchanged).

Not done: an edge-aware upsample from the AO to the frame. The game's composite multiplies the AO in with a plain bilinear fetch, and its own blur isn't depth-aware either. At 2x the bleed past edges is half as wide, and I didn't see halos in the comparisons below, so I left the game's composite alone.

Testing

Metal (Apple Silicon, macOS), hidden window, scratch saves. Field_11C (flat grass, fences, the house, trees) and Bow-wow's house (furniture), at render scales 0.25, 1 and 2:

• AO buffer (after the game's blur): 2x on the game's grid shows a crosshatch. High is clean and clearly sharper around fence posts, grass, the tree and furniture.
• 0.25x: the old scaled AO shows the A weird white haze/flicker when walking around #356 haze on the grass. Switch and High are identical there, as intended.
• 1x / 2x frames: no bands, dither or halos. A walking filmstrip (8 consecutive frames, camera scrolling) shows no crawling or flicker in either mode.
• The [Feature Request] Ambient Occlusion quality #602 reporter's setup: 2x (3840x2160), Bilinear, SMAA, Remove Blur on, 1920x1080 window, Toronbo Shores where their video was taken. Switch shows a jagged, speckled dark fringe along the wall bases, the sea urchins and the driftwood. High removes it and the contact shadow is smooth.
• Above 2x (LARECOMP_AO_SCALE=4, same setup): no visible difference from High. The mean pixel difference to High is a third of High's difference to Switch. The AO samples depth snapped to a 960x540 grid, so High already uses all the detail there is. That's why the setting stops at High.
• Every scene uses the same AO and blur programs (72f3481200 / 72f3231300), so the signature-checked patch applies everywhere.

Vulkan (Linux aarch64 VM, lavapipe): builds clean. The two patched shaders also pass glslc and spirv-val offline (plain and structured). In game, at the reporter's spot (render scale 1, Bilinear, SMAA, Remove Blur): Default shows the same speckled fringe at the wall bases as Metal, High is smooth, and there's no crosshatch, so the grid fix works on Vulkan too. No validation errors or device loss in the logs.

Comparison images: to be attached (AO buffer at 1x and 2x in the Field, the house; frames at 0.25x, 1x, 2x; the reporter's setup, Default / High / 4x; Vulkan Default / High).

🤖 Generated with Claude Code