Skip to content

fix(media): run the filesystem index in the background - #111

Merged
ssavutu merged 1 commit into
mainfrom
patch/delta-media-serving
Jul 30, 2026
Merged

fix(media): run the filesystem index in the background#111
ssavutu merged 1 commit into
mainfrom
patch/delta-media-serving

Conversation

@ssavutu

@ssavutu ssavutu commented Jul 30, 2026

Copy link
Copy Markdown
Member

The media index could never complete over HTTP. Fixes the 504 seen on the first
production reindex attempt.

The bug

The index walks ~145k filesystem entries on the real corpus and takes minutes,
but it ran inside the request on r.Context(). The first proxy to give up
killed it:

POST /v1/media/index  status=500  duration_ms=60045

That is Nginx's default proxy_read_timeout of 60s — Cloudflare would have cut
in at ~100s regardless. The walk checked ctx.Err() and aborted cleanly, so the
6,485 rows it had inserted were committed and consistent, but no amount of
retrying could finish the job and the Media → Reindex button was unusable by
construction
on a corpus this size.

The fix

  • POST /v1/media/index starts the walk on a background context that outlives
    the request and returns 202 immediately.
  • GET /v1/media/index reports {running, started_at, finished_at, progress, error}.
  • A second POST while a run is in flight returns 409 instead of starting a
    duplicate.
  • Runs are capped at two hours, so a wedged filesystem can't leave the job stuck
    running and block every later attempt.
  • Progress counts entries walked, not files indexed. The corpus is mostly
    WordPress derivatives that are skipped before any stat, so an indexed-file
    counter appears frozen for long stretches. MediaIndexResponse gains Walked.
  • Frontend polls every 2s and shows live counts in the button. A 409 is treated
    as "join the run already in progress" rather than an error.

Unrelated bug this surfaced

refresh() was setSearch(current => current). React bails out of a same-value
setState, so the effect never re-fired and the list silently never reloaded
after an upload or index. It now bumps a token the load effect depends on.

Testing

  • go build, go vet, go test ./... clean; tsc -b, eslint, and the
    production build clean.
  • New unit tests for the job lifecycle: start/refuse-concurrent/progress/finish,
    failure recording, reset-on-restart, plus 409 and status handlers.
  • New integration test (CMS_TEST_DSN-gated, run against real MariaDB 11.8)
    asserting the POST returns 202 in under 2s and the walk completes after the
    request returned, with the derivative skipped and rows actually written by the
    background goroutine.
  • Database-level test that the progress callback fires and its final report
    agrees with the returned one, including on trees smaller than the callback
    interval.

Operational note

deploy/README.md updated. Once this is deployed, Media → Reindex works from the
UI; the partial index from the failed run is preserved and the rerun resumes,
skipping what is already there and keeping any alt text.

The index could never complete over HTTP. It walks ~145k filesystem entries on
the real corpus and takes minutes, but it ran inside the request on
r.Context(), so the first proxy to give up killed it: Nginx cuts an idle
upstream read at 60s (observed: status=500 duration_ms=60045, surfacing to the
client as Cloudflare's 504), and Cloudflare would cut in at ~100s regardless.
The walk aborted cleanly and committed what it had — 6485 of ~24k originals —
but no amount of retrying could finish it, and the Media -> Reindex button was
unusable by construction.

POST /v1/media/index now starts the walk on a background context that outlives
the request and returns 202 immediately; GET /v1/media/index reports
{running, started_at, finished_at, progress, error}. A second POST while a run
is in flight returns 409 rather than starting a duplicate. Runs are capped at
two hours so a wedged filesystem cannot leave the job permanently "running" and
block every later attempt.

Progress counts entries walked, not files indexed: the corpus is mostly
WordPress derivatives skipped before any stat, so an indexed-file counter
appears frozen for long stretches. MediaIndexResponse gains Walked for this.

The frontend polls every 2s and shows live counts in the button. A 409 is
treated as "join the run in progress" rather than an error, so clicking Reindex
while one is already going attaches to it.

Also fixes a real bug this surfaced: refresh() was setSearch(current => current),
which React bails out of, so the list never actually reloaded after an upload or
index. It now bumps a token the load effect depends on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ssavutu
ssavutu merged commit ae5bc6e into main Jul 30, 2026
5 checks passed
@ssavutu
ssavutu deleted the patch/delta-media-serving branch July 30, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant