Skip to content

A complete render stops waiting for a local PMTiles/MBTiles archive's tiles under CPU load: the archive work isn't counted in cache::in_flight() #1761

Closed

Description

@elpidioassinaturas

What happens

The tests in crates/engine/src/tests/archive_render_revision.rs (added with #1504) fail intermittently on main when the machine is busy:

  • archive_reload_refuses_spanning_exports_without_replacing_destination fails at line 103 (paused_map): the first render_placed of the old archive comes back with credits: [] instead of ["Old archive credits"].
  • a_reload_between_series_pages_refuses_the_engine_pdf_without_replacing_destination fails at line 331. The warm render's gaps say: some tiles were still on their way when the export stopped waiting, so parts of tile layers are empty: export again once the map has drawn them.

Each test passes when it runs alone. On an idle machine, the whole file passes 8 runs out of 8 with --test-threads=1. With every core busy, it failed on the first run. Several of our local cargo test runs on Windows hit it on main, with other builds running at the same time.

The same thing can happen to a user. An export (PNG, PDF, layout, series) of a map with a local PMTiles or MBTiles archive, made while the PC is busy, can come out with empty tile areas and the "export again" note, although nothing was slow to read.

Cause (measured)

I added a temporary print to complete in crates/render/src/lib.rs, after each wait_for_news_unless. It logs idle, generation(), drawn_at, cache::in_flight() and the elapsed time. I then ran the file with 16 busy loops on a 16-thread CPU. On the failing run:

DBG pass idle=1 gen=158 was=158 in_flight=0 t=4.2µs
DBG pass idle=2 gen=158 was=158 in_flight=0 t=29.4487ms
...
DBG pass idle=2 gen=168 was=168 in_flight=0 t=11.0003ms
DBG pass idle=3 gen=168 was=168 in_flight=0 t=24.7317ms

in_flight() is 0 on every pass, while r.tiles_pending is still true. So wait_for_news_unless returns at once, because its loop runs only while in_flight() > 0. Passes with no new generation follow each other a few milliseconds apart, and the IDLE_PASSES (6) budget from #1188 runs out within tens of milliseconds. Meanwhile the archive's tile is still being read or decoded.

In short, some of the work behind a local archive's tile happens outside IN_FLIGHT: the PMTiles header, directory and tile reads, or the MBTiles read, or the picture decode after it. Network fetches are counted (start, download_reaching, fetch_counted, fetch_local). The wait therefore treats "an archive tile is on its way" as "nothing is under way". On an idle machine that work ends before six passes have run, and on a busy one it doesn't.

Reproduce

On main (6240dc0):

export CARGO_INCREMENTAL=0 RUSTGIS_LANGUAGE=en
cargo test -p rustgis-engine --lib --no-run
for k in $(seq 1 $(nproc)); do ( while :; do :; done ) & done
cargo test -p rustgis-engine --lib tests::archive_render_revision
kill $(jobs -p)

This fails within a run or two. Without the busy loops, it passes.

Proposal

Count a local archive's work in IN_FLIGHT from the request to the answer: the archive reads and the decode that follows, as network fetches already are. Then wait_for_news waits for them, and IDLE_PASSES counts only passes where nothing is really on its way, which is what #1188 meant it to do.

Choices in this proposal:

  • Count, don't lengthen. Raising IDLE_PASSES or sleeping between idle passes would only move the threshold: a busier machine would still miss it, and every truly idle export would wait longer.
  • One counter. in_flight() already backs both cache::wait and wait_for_news. Counting the archive work there fixes every waiter at once.
  • Test. Add a test whose archive read is held by a barrier, as PauseSource holds a layer in these tests. render_placed must return the archive's tile, not the "still on their way" gap, however long the hold lasts within the minute budget. The current tests then stop depending on machine load.

Found while running main's engine tests on Windows 11 (16 threads).

Activity

  1. added 2 commits that reference this issue on Nov 11, 2026
  2. elpidioassinaturas commented on Nov 11, 2026

    @elpidioassinaturas
    ContributorAuthor

    It now fails on CI too, on ubuntu-latest: run 43009961762, job 130486964707. The PR is #1749, which doesn't touch tiles or rendering. The failure is a_reload_between_series_pages_refuses_the_engine_pdf_without_replacing_destination at archive_render_revision.rs:331, the same line as in the report above. So it's not only a Windows-under-load problem. #1765 counts the MBTiles metadata read in in_flight(), which is the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions