Skip to content

fix(media): run the filesystem index in the background #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ssavutu merged 1 commit into from
Jul 30, 2026
Merged

fix(media): run the filesystem index in the background #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions deploy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,8 @@ the database. After the media rsync completes, populate it once from the CMS
(Media -> Reindex) or directly:

```bash
curl -X POST https://localhost/v1/media/index # admin session required
curl -X POST https://localhost/v1/media/index # admin session required; returns 202
curl https://localhost/v1/media/index # poll progress
```

It walks `MEDIA_ROOT/wp-content/uploads`, skips WordPress's generated `-WxH`
Expand All @@ -159,8 +160,17 @@ already-indexed files are skipped and any alt text set in the CMS is preserved
so re-run it after any later out-of-band rsync. Uploads through the CMS index
themselves and need no reindex.

Note this walks the whole tree, so on a large corpus over CephFS the first run
takes a while; run it once at cutover rather than on a schedule.
**The index runs in the background.** `POST` returns `202` immediately and `GET`
reports `{running, progress:{walked, scanned, added, skipped}, error}`; a second
`POST` while one is in flight returns `409`. This is not cosmetic: the real corpus
is ~145k filesystem entries and the walk takes minutes, while Nginx cuts an idle
upstream read at 60s and Cloudflare at ~100s. A synchronous version was cancelled
by those proxies every time and could never finish. A run is capped at two hours
so a wedged filesystem cannot leave the job stuck "running" forever.

Progress is reported per entry *walked* rather than per file indexed, because the
corpus is mostly derivatives that are skipped without a stat — counting only
indexed files would look frozen for long stretches.

### Disk

Expand Down
58 changes: 51 additions & 7 deletions frontend/src/pages/mediaView.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,20 @@ type MediaResponse = {
}

type IndexReport = {
walked?: number
scanned?: number
added?: number
skipped?: number
}

type IndexStatus = {
running: boolean
started_at?: string
finished_at?: string
progress?: IndexReport
error?: string
}

const PAGE_SIZE = 60

function formatBytes(bytes?: number) {
Expand Down Expand Up @@ -73,6 +82,8 @@ function MediaView() {
const [selected, setSelected] = useState<MediaItem | null>(null)
const [uploadStatus, setUploadStatus] = useState<string | null>(null)
const [isIndexing, setIsIndexing] = useState(false)
const [indexProgress, setIndexProgress] = useState<string | null>(null)
const [reloadToken, setReloadToken] = useState(0)

// Debounce so typing doesn't fire a request per keystroke; the filter itself
// is applied server-side because the library is far too large to filter here.
Expand Down Expand Up @@ -117,7 +128,7 @@ function MediaView() {

void load()
return () => controller.abort()
}, [fetchPage])
}, [fetchPage, reloadToken])

const loadMore = async () => {
setIsLoadingMore(true)
Expand All @@ -133,7 +144,10 @@ function MediaView() {
}
}

const refresh = () => setSearch((current) => current)
// Bumping a token is what actually re-runs the load effect. Re-setting `search`
// to its current value does not: React bails out of a same-value setState, so
// the effect never re-fires and the list silently stays stale.
const refresh = useCallback(() => setReloadToken((n) => n + 1), [])

const handleUpload = async (files: FileList | null) => {
if (!files || files.length === 0) return
Expand Down Expand Up @@ -223,22 +237,52 @@ function MediaView() {
}
}

// The index walks ~145k filesystem entries and takes minutes, so the server
// runs it in the background and we poll. A synchronous request could never
// finish: Nginx cuts an upstream read at 60s and Cloudflare at ~100s.
const pollIndexStatus = useCallback(async (): Promise<boolean> => {
const response = await apiFetch("/v1/media/index")
if (!response.ok) throw new Error(await errorMessage(response, `Status check failed (${response.status})`))
const status = (await response.json()) as IndexStatus

const p = status.progress
if (status.running) {
setIndexProgress(
`Indexing… ${String(p?.walked ?? 0)} files scanned, ${String(p?.added ?? 0)} added`,
)
return true
}

setIndexProgress(null)
if (status.error) {
setError(`Reindex failed: ${status.error}`)
} else if (status.finished_at) {
setNotice(`Indexed ${String(p?.scanned ?? 0)} assets — ${String(p?.added ?? 0)} new, ${String(p?.skipped ?? 0)} already known.`)
if ((p?.added ?? 0) > 0) refresh()
}
return false
}, [apiFetch, refresh])

const handleReindex = async () => {
setIsIndexing(true)
setError(null)
setNotice(null)

try {
const response = await apiFetch("/v1/media/index", { method: "POST" })
if (!response.ok) {
// 409 means one is already running — join its progress rather than error.
if (!response.ok && response.status !== 409) {
setError(await errorMessage(response, `Reindex failed (${response.status})`))
setIsIndexing(false)
return
}
const report = (await response.json()) as IndexReport
setNotice(`Indexed ${String(report.scanned ?? 0)} files — ${String(report.added ?? 0)} new.`)
if ((report.added ?? 0) > 0) refresh()

while (await pollIndexStatus()) {
await new Promise((resolve) => setTimeout(resolve, 2000))
}
} catch (err) {
setError(err instanceof Error ? err.message : "Unable to reindex media.")
setIndexProgress(null)
} finally {
setIsIndexing(false)
}
Expand Down Expand Up @@ -268,7 +312,7 @@ function MediaView() {
type="button"
>
<RefreshCw className={`w-4 h-4 ${isIndexing ? "animate-spin" : ""}`} aria-hidden="true" />
{isIndexing ? "Indexing..." : "Reindex"}
{indexProgress ?? (isIndexing ? "Starting…" : "Reindex")}
</button>
<input
accept="image/jpeg,image/png,image/gif,image/webp"
Expand Down
55 changes: 50 additions & 5 deletions server/docs/docs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1732,7 +1732,7 @@ const docTemplate = `{
}
},
"/v1/media/index": {
"post": {
"get": {
"security": [
{
"BearerAuth": []
Expand All @@ -1744,16 +1744,38 @@ const docTemplate = `{
"tags": [
"media"
],
"summary": "Reindex the media filesystem",
"summary": "Media reindex status",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/models.MediaIndexResponse"
"$ref": "#/definitions/models.MediaIndexStatusResponse"
}
}
}
},
"post": {
"security": [
{
"BearerAuth": []
}
],
"produces": [
"application/json"
],
"tags": [
"media"
],
"summary": "Start a media filesystem reindex",
"responses": {
"202": {
"description": "Accepted",
"schema": {
"$ref": "#/definitions/models.MediaIndexStatusResponse"
}
},
"500": {
"description": "Internal Server Error",
"409": {
"description": "Conflict",
"schema": {
"$ref": "#/definitions/models.ErrorResponse"
}
Expand Down Expand Up @@ -4497,6 +4519,29 @@ const docTemplate = `{
},
"skipped": {
"type": "integer"
},
"walked": {
"type": "integer"
}
}
},
"models.MediaIndexStatusResponse": {
"type": "object",
"properties": {
"error": {
"type": "string"
},
"finished_at": {
"type": "string"
},
"progress": {
"$ref": "#/definitions/models.MediaIndexResponse"
},
"running": {
"type": "boolean"
},
"started_at": {
"type": "string"
}
}
},
Expand Down
55 changes: 50 additions & 5 deletions server/docs/swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1729,7 +1729,7 @@
}
},
"/v1/media/index": {
"post": {
"get": {
"security": [
{
"BearerAuth": []
Expand All @@ -1741,16 +1741,38 @@
"tags": [
"media"
],
"summary": "Reindex the media filesystem",
"summary": "Media reindex status",
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/models.MediaIndexResponse"
"$ref": "#/definitions/models.MediaIndexStatusResponse"
}
}
}
},
"post": {
"security": [
{
"BearerAuth": []
}
],
"produces": [
"application/json"
],
"tags": [
"media"
],
"summary": "Start a media filesystem reindex",
"responses": {
"202": {
"description": "Accepted",
"schema": {
"$ref": "#/definitions/models.MediaIndexStatusResponse"
}
},
"500": {
"description": "Internal Server Error",
"409": {
"description": "Conflict",
"schema": {
"$ref": "#/definitions/models.ErrorResponse"
}
Expand Down Expand Up @@ -4494,6 +4516,29 @@
},
"skipped": {
"type": "integer"
},
"walked": {
"type": "integer"
}
}
},
"models.MediaIndexStatusResponse": {
"type": "object",
"properties": {
"error": {
"type": "string"
},
"finished_at": {
"type": "string"
},
"progress": {
"$ref": "#/definitions/models.MediaIndexResponse"
},
"running": {
"type": "boolean"
},
"started_at": {
"type": "string"
}
}
},
Expand Down
38 changes: 33 additions & 5 deletions server/docs/swagger.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,21 @@ definitions:
type: integer
skipped:
type: integer
walked:
type: integer
type: object
models.MediaIndexStatusResponse:
properties:
error:
type: string
finished_at:
type: string
progress:
$ref: '#/definitions/models.MediaIndexResponse'
running:
type: boolean
started_at:
type: string
type: object
models.MediaOverview:
properties:
Expand Down Expand Up @@ -2143,16 +2158,29 @@ paths:
tags:
- media
/v1/media/index:
post:
get:
produces:
- application/json
responses:
"200":
description: OK
schema:
$ref: '#/definitions/models.MediaIndexResponse'
"500":
description: Internal Server Error
$ref: '#/definitions/models.MediaIndexStatusResponse'
security:
- BearerAuth: []
summary: Media reindex status
tags:
- media
post:
produces:
- application/json
responses:
"202":
description: Accepted
schema:
$ref: '#/definitions/models.MediaIndexStatusResponse'
"409":
description: Conflict
schema:
$ref: '#/definitions/models.ErrorResponse'
"501":
Expand All @@ -2161,7 +2189,7 @@ paths:
$ref: '#/definitions/models.ErrorResponse'
security:
- BearerAuth: []
summary: Reindex the media filesystem
summary: Start a media filesystem reindex
tags:
- media
/v1/poll:
Expand Down
Loading
Loading