Skip to content

Compile the software ray tracing pipelines after the editor opens, and let no frame wait for them - #15

Open
5hif7y wants to merge 5 commits into
mainfrom
rays-later
Open

5hif7y wants to merge 5 commits into
mainfrom
rays-later

Conversation

@5hif7y

@5hif7y 5hif7y commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Closes #7.

Rebuilt on main b33d9a2 after the second review. The comment below answers it point by point.

Changes

render: the editor opens before the software tracer is compiled

Renderer_Desc.tracer_later (the editor's, never a game's) starts the
renderer without the stages of rays traced by software: tracer_settle
asks for them on the existing pipeline thread with pipeline_make(early)
through pipeline_variants_ask, which now asks for the stages of a tracer
without constants too. Until they are in place the frame draws no world
and Stats.problem names the stage; a renderer without pipeline_thread
waits for it. A refused stage becomes Refused and is reported once, so
nothing waits on it. The pipelines of a stage of one tracer are kept as
variants, so the skin and TLAS-batch stages go through pipeline_of.

Captures (capture without ui, play.capture, recordings and the editor's
--capture) wait for every pipeline asked, early ones too, before their
settling frames.

render: captures wait for what they draw; overlays are variants

A capture that waited for every pipeline asked ahead of time compiled
all the variants of both kinds of shadows before its first frame: on a
cold driver cache it still waited after an hour. Instead a frame no
window shows (Frame_Desc.offscreen: capture, play.capture, recordings)
waits, as one of a renderer without pipeline_thread does, for each stage
it draws with, so its settling frames run with them and nothing else.
The editor's --capture waits for what its frames asked for.

The pipelines of surface overlays were kept apart from the variants, so
with tracer_later nothing asked for the software ones and the world was
never drawn. Each blend is now a Pipeline_Id whose tracers are variants.

render: a frame that waits waits for the program stages it draws

A frame no window shows promoted a program stage asked ahead of time but
drew its materials as their rows. It now waits for the queued job, as it
does for the stages of the renderer, without making it a second time.

editor: a scene thumbnail waits for frames drawn whole, not for the driver

A scene thumbnail is taken with app_capture_world, whose offscreen frames
wait for every stage they draw with. With tracer_later, the editor opened
on a scene whose stage of the tracer was still being made, and the
thumbnail of that scene waited for the driver on the thread of the UI:
with a stage that takes long to make, the window and the API stopped
answering until it was made.

renderer_frame_partial says whether the last frame drew without a stage
it asked for (a pass skipped while its pipeline is made, or no world
while a stage of the tracer is missing). The thumbnail of a scene waits
for a frame of the window that drew whole, and captures still wait for
what they draw.

render: no frame waits for the driver; readers draw again until whole

A frame no window shows no longer waits for the stages it draws with:
on the thread of the UI that froze the editor (a .particle preview, a
scene thumbnail) while the software tracer was made. What reads a frame
back draws again while renderer_frame_partial, which now also sees the
pipelines frames asked for and the program stages drawn as their rows,
says one is being made, within the wait it already had: app_capture_world,
recordings, the window of a capture with ui. A frame still partial then,
or one without a stage the device refused, fails the capture with the
problem the frames give, instead of an image without the world.

The stage pipelines of extensions are made on the pipeline thread too:
the first .particle on a cold driver cache no longer stops the window.
The scene tab shows the dot of what works while the viewport draws
without a stage being made.

Merge order

Independent of the other open PRs: it merges cleanly into main.

Not tested

🤖 Generated with Claude Code

@5hif7y 5hif7y self-assigned this Nov 9, 2026
@5hif7y

5hif7y commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Needs more testing as a user before merging.

The tests and the sessions listed in the description cover the engine side only. My lead and I will adapt it as needed, and tomorrow I will test it with a separate agent acting as a Mirai user.

@tomaszbk

Copy link
Copy Markdown
Member

Review: rework. This PR builds on code that main has since reshaped.

  • create_pipelines now takes a set of tracers, and tracer_settle compiles only the tracer the settings ask for.
  • So on main, a GPU with hardware ray tracing never compiles the software tracer.
  • The case left is a GPU without ray queries. Deferring that case should go through the existing pipeline thread (pipeline_make(..., early=true) in render/pipeline_thread.odin), not through a second thread pool.

Findings on the branch:

  1. Hang on a failed pipeline (host/player.odin:343, editor/editor.odin:537). After a pipeline fails, renderer_rays_pending never goes back to 0. A --frames N --capture run then repeats app_rays_wait and continue forever.
  2. Race: a frame can draw while a pipeline is missing (render/frame.odin:406 against render/ray_pipelines.odin:108-109).
    • The pending count tracks what the threads finished, not what has been installed.
    • If a compile finishes between the install step and the check, the frame draws the scene while a pipeline slot is still 0.
    • The result is a bind of an invalid pipeline and a wrong image.
  3. Captures are taken before the frame settles. With --capture, the run captures one frame after the rays are ready. Temporal history and GI have not converged yet, so the image does not match its reference.
  4. The pipeline-making algorithm exists twice. render/ray_pipelines.odin copies the cost sort and the thread pool of create_pipelines, next to Pipeline_Thread. It also does not check whether a thread failed to start (ray_pipelines.odin:89).
  5. A tuning number from one machine. The memory ratio of 2000 (ray_pipelines.odin:20-24) comes from a measurement on one driver. It now sets the compile thread count for every user (renderer.odin:1122).
  6. A black world in shipped games. The player sets rays_later=true (host/player.odin:263). On a GPU without ray queries, a game then runs its gameplay over a black world until the compile ends.
  7. Backward compatibility and timing figures.
    • tools/mirai/headless.odin:44 accepts an older editor build as ready. The project keeps no backward compatibility.
    • The editor message and the docs state timings ("minutes on a first start"). The docs should describe the behavior without measurements.

@5hif7y
5hif7y changed the base branch from session-cold-cache to main November 15, 2026 01:24
@5hif7y 5hif7y changed the title Compile the software ray tracing pipelines after the editor opens Compile the software ray tracing pipelines after the editor opens, in the editor only Nov 15, 2026
@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Rebuilt on main 8c5a2a6, editor only, as 65ba39b5, 689a48c4 and d51970a2, force-pushed to rays-later. It merges into main cleanly. Code without tests is +110/−41.

Approach.

  • Renderer_Desc.tracer_later is set by the editor only.
  • tracer_settle asks for the software tracer's stages on the existing Pipeline_Thread with pipeline_make(..., early=true), through pipeline_variants_ask.
  • There is no second pool and no cost sort.
  1. Hang on a failed pipeline. A refused stage becomes .Refused and is reported once, so nothing waits on it. Test: a_tracer_made_later_is_waited_for_by_a_capture_and_a_refused_stage_does_not_block.
  2. Race on a missing pipeline. Stages are installed by pipelines_install between frames. While a stage is missing, pipeline_of skips the pass and Stats.problem names it, so nothing binds a 0 pipeline.
  3. Captures before the frame settles.
    • A frame no window shows (Frame_Desc.offscreen: capture without ui, play.capture, recordings) waits for each stage it draws with. Then it runs the existing settling frames.
    • A queued job is promoted with pipeline_urgent, never compiled twice.
    • The editor's --capture waits for what its frames asked for.
    • Cold against warm: SSIM 0.9994.
  4. The algorithm twice. render/ray_pipelines.odin is gone.
  5. A tuning number from one machine. Gone with it: there is no memory ratio and no thread count. One finding about main here: on 4 cores and 16 GB without ray query, a cold main did not open its API in 30 min. Its private memory reached 7.2 GB at 1 min, 13.9 at 3 and 25.8 at 6, while it compiled the software tracer in parallel, and Windows paged. The underlying problem is memory per compile, not a thread count.
  6. A black world in shipped games. The player stays synchronous; tracer_later is editor only. A game never starts without its tracer.
  7. Compatibility and timings. tools/mirai/headless.odin is untouched. The editor problem line and the docs state no timings.

Also fixed: surface overlays were never queued, so with tracer_later the world was never drawn. Each blend is now a Pipeline_Id whose tracers are variants.

Measured on an RX 580, cold:

  • The editor opens in 10–13 s; main takes 620 s or more.
  • The first cold capture takes 990 s. Pipeline_Thread has 1 worker on 4 cores, unchanged.
  • A cold mirai capture exceeds its 600 s RPC limit by about 390 s. The limit was not changed.
  • render tests pass 280/282. The two failures fail the same way on main.

Not tested:

  • render check --quality against main: the cold base does not open on this machine (point 5).
  • A GPU with ray query (tracer_later does not apply there).
  • Linux and macOS.

@5hif7y
5hif7y requested a review from tomaszbk November 15, 2026 01:26
@5hif7y 5hif7y closed this Nov 15, 2026
@5hif7y 5hif7y reopened this Nov 15, 2026
@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Follow-up: a freeze found by a live run, fixed in f09a580.

What happened. I opened the editor of all six PRs merged onto main, directly on demo (not --copy), on the RX 580 machine.

  • The window and the API stopped answering.
  • A temporary log, since removed, showed app_capture_world for the scene thumbnail (512×512, 8 frames) starting and never ending, while the software tracer's Reflect stage compiled.
  • The thumbnail's frames are offscreen, so with this PR they wait for every stage they draw with. That wait ran on the UI thread.

The fix (+11/−2):

  • render.renderer_frame_partial tells whether the last frame drew without a stage it asked for: a pass skipped while its pipeline is made, or no world while a tracer stage is missing.
  • thumbnail_scene_capture waits for a window frame that drew whole before it captures. It keeps the request and retries between later frames.
  • Captures and recordings still wait for what they draw, as before.

Verified:

  • The same launch now answers engine.status in 0.4–0.9 s, with renderer_problem "the world is not drawn while the pipeline Reflect of rays traced by software is made".
  • The new test a_frame_drawn_without_a_stage_being_made_is_partial passes, and so does a_tracer_made_later_is_waited_for_by_a_capture_and_a_refused_stage_does_not_block.
  • render passes 281/283. The two failures fail the same way on main 8c5a2a6: render_far_from_the_centre_looks_as_at_it_and_a_jump_keeps_the_history and renderer_calls_without_memory_are_refused_and_keep_what_they_had.
  • The branch merges cleanly into main and with every other PR of this set.

A separate finding, not caused by this PR. On this machine (amdvlk, RX 580, 16 GB), one variant of the software tracer's Reflect stage does not finish compiling.

  • It reached 19 GB private memory after 15 min and never reached the pipeline store.
  • It is the likely cause of the cold main that never opened its API (25 GB, see the first comment).
  • bench compare cannot run here either: its headless frames wait for that stage.
  • With this PR the editor stays usable and names the stage, but the stage itself needs a look at the shader or the driver. I can open an issue for it if useful.

@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

The Reflect stage that never finished compiling on this machine (last comment) is the coverage of Terrain_Layers, which kept the whole program in every traversal because each of its texture samples writes feedback. Root cause and fix: #83 / #84. With #84, the main editor opened on demo answers at 268 s instead of never.

@tomaszbk

Copy link
Copy Markdown
Member

Review of the rebuilt branch: not ready yet, but close. The approach is now the right one, and most of the first review is solved. What is left is how frames wait.

Checked and correct

  • No race: pipelines are installed only from the main thread, the thread that draws.
  • No zero pipeline is bound: the world is not drawn until the required stages are in place.
  • No endless wait: a refused stage comes back as made with nothing in it.
  • Games, hardware ray tracing and Metal do not change.
  • The variants refactor is needed (stages need an asked/refused state) and it removes code.

What blocks the merge

  1. The editor can still freeze for minutes. renderer_frame sets r.frame_waits = !r.pipelines_async || frame.offscreen (render/frame.odin:396), and tracer_later_missing then calls renderer_pipelines_wait (render/renderer.odin:1411).
    • So every offscreen frame waits on the UI thread for all urgent jobs, also jobs that frame does not draw with, and also frames that draw no world (host/frame.odin:654).
    • Scenario: on a cold driver cache, open a .particle document, or show a browser cell of one that has no saved thumbnail (plugins/vfx/editor/preview.odin:92 goes through app_world_draw). The window blocks until the software tracer is compiled. It is the same kind of freeze as the thumbnail one you fixed in f09a580.
  2. The thumbnail fix is a special case and misses part of the wait (editor/thumbnail_subjects.odin:317). renderer_frame_partial does not see the pipelines of material programs that are still being made. The window frame counts as whole, and then the 8 thumbnail frames wait one by one for each program pipeline, on the UI thread.
  3. The viewport shows no world and nothing says why (render/frame.odin:406). The reason is only in renderer_problem, over the API. For minutes the user sees an empty viewport. The editor may show the dot of what is working here.
  4. A refused stage gives empty captures with no error (render/renderer.odin:1416). The error is set in one frame only. After that, capture returns an image with no world and ok=true.
  5. Two new doc lines are over 100 characters: shaders.md:88 and materials.md:402.

A simpler shape that fixes 1 and 2

  • Remove frame_waits from pipeline_of, program_graphics, program_compute and tracer_later_missing. No frame waits for the driver.
  • Let whoever reads a frame back (app_capture_world, recordings, the thumbnail) draw again while renderer_frame_partial or renderer_pipelines_pending is true. app_capture_world already does this for its last frame.
  • That is less code than the branch has now, and the UI thread never blocks.

For 4, keep the refusal as a state the capture can read, so a capture of a world that cannot be drawn fails with that message.

5hif7y and others added 4 commits November 15, 2026 10:36
Renderer_Desc.tracer_later (the editor's, never a game's) starts the
renderer without the stages of rays traced by software: tracer_settle
asks for them on the existing pipeline thread with pipeline_make(early)
through pipeline_variants_ask, which now asks for the stages of a tracer
without constants too. Until they are in place the frame draws no world
and Stats.problem names the stage; a renderer without pipeline_thread
waits for it. A refused stage becomes Refused and is reported once, so
nothing waits on it. The pipelines of a stage of one tracer are kept as
variants, so the skin and TLAS-batch stages go through pipeline_of.

Captures (capture without ui, play.capture, recordings and the editor's
--capture) wait for every pipeline asked, early ones too, before their
settling frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capture that waited for every pipeline asked ahead of time compiled
all the variants of both kinds of shadows before its first frame: on a
cold driver cache it still waited after an hour. Instead a frame no
window shows (Frame_Desc.offscreen: capture, play.capture, recordings)
waits, as one of a renderer without pipeline_thread does, for each stage
it draws with, so its settling frames run with them and nothing else.
The editor's --capture waits for what its frames asked for.

The pipelines of surface overlays were kept apart from the variants, so
with tracer_later nothing asked for the software ones and the world was
never drawn. Each blend is now a Pipeline_Id whose tracers are variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frame no window shows promoted a program stage asked ahead of time but
drew its materials as their rows. It now waits for the queued job, as it
does for the stages of the renderer, without making it a second time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…river

A scene thumbnail is taken with app_capture_world, whose offscreen frames
wait for every stage they draw with. With tracer_later, the editor opened
on a scene whose stage of the tracer was still being made, and the
thumbnail of that scene waited for the driver on the thread of the UI:
with a stage that takes long to make, the window and the API stopped
answering until it was made.

renderer_frame_partial says whether the last frame drew without a stage
it asked for (a pass skipped while its pipeline is made, or no world
while a stage of the tracer is missing). The thumbnail of a scene waits
for a frame of the window that drew whole, and captures still wait for
what they draw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frame no window shows no longer waits for the stages it draws with:
on the thread of the UI that froze the editor (a .particle preview, a
scene thumbnail) while the software tracer was made. What reads a frame
back draws again while renderer_frame_partial, which now also sees the
pipelines frames asked for and the program stages drawn as their rows,
says one is being made, within the wait it already had: app_capture_world,
recordings, the window of a capture with ui. A frame still partial then,
or one without a stage the device refused, fails the capture with the
problem the frames give, instead of an image without the world.

The stage pipelines of extensions are made on the pipeline thread too:
the first .particle on a cold driver cache no longer stops the window.
The scene tab shows the dot of what works while the viewport draws
without a stage being made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@5hif7y 5hif7y changed the title Compile the software ray tracing pipelines after the editor opens, in the editor only Compile the software ray tracing pipelines after the editor opens, and let no frame wait for them Nov 15, 2026
@5hif7y

5hif7y commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Reworked the way you proposed. 9b8b3c3 on top of the previous commits, rebased onto main b33d9a2, force-pushed to rays-later. Code without tests: +163/−87 against main, and +82/−84 against f09a580, so it ends 2 lines shorter.

  1. Freeze.
    • No frame of the windowed editor waits for the driver: frame_waits is gone from pipeline_of, program_graphics, program_compute and tracer_later_missing.
    • Renderers without a pipeline thread (headless sessions, tests) keep main's rule for variants.
    • The live run found a second cause of the .particle freeze. Plugin stage pipelines (render/extend.odin) were compiled on the UI thread the first time a pass asked for them, 9.4 s on a cold cache. They now go to the pipeline thread; every plugin already asks for them each frame and skips on 0.
    • The editor's own capture had a loop that waited for pending pipelines and blocked for minutes. It now redraws instead.
  2. Thumbnail and readers.
    • renderer_frame_partial also counts pipelines still being made, so materials drawn as their rows count.
    • app_capture_world, recordings and a capture with ui draw again while it is partial, within the wait they already had, then run their settling frames.
    • app_world_draw treats a partial draw as not drawn, so previews retry.
  3. Viewport. The scene tab shows the editor's pulsing dot while the frame is partial. renderer_problem still names the missing stage, with no timings.
  4. Refused stage. A capture still partial after its wait, or partial because a stage was refused, fails with that message instead of returning an empty image. The editor's --capture fails the same way.
  5. Docs. shaders.md:88 and materials.md:402 are now under 100 characters. materials.md:82 is also over, but it already was on main.

Tests. Each fails without its fix.

  • render: an offscreen frame does not wait and reports partial; a refused tracer stage leaves frames partial with nothing pending.
  • host: a capture fails with the reason while the tracer is being made, redraws until whole when given a wait, and fails with the refusal message when a stage is refused.
  • Results: render subset 5/5 with validation, host 2/2, editor 6/6, portals 1/1.

Live on the RX 580, windowed, opened directly on demo, with a cold tracer:

  • The API answered in about 19 s, then in 0.5–0.6 s per call at about 60 fps while the tracer compiled.
  • A .particle opened in 0.5 s, with no stall after it.
  • A capture with ui returned after its 10 s limit.
  • A world-only capture failed after the same limit with "the world is not drawn while the pipeline Direct_Filter of rays traced by software is made".

Not done:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A cold start blocks for ten minutes compiling software ray tracing pipelines

2 participants