fix(index): resume interrupted full indexing - #120
Conversation
A full index wiped the database at the start and wrote its completion marker only at the very end, so any crash, kill or power loss left no marker -- and the next launch began another full index, which immediately erased every batch the previous run had finished. On a large project one crash cost hours; repeated crashes meant the index could never complete at all. Schema v3 adds deep_indexed_hash and deep_index_attempts to assets, plus an in_progress marker in meta. Migration is additive and leaves existing rows untouched; a v0.21.3 build reading a v3 database is inert rather than broken. Force-reindex still wipes, which is its documented purpose. Two records with deliberately DIFFERENT durability requirements, which is the whole mechanism and is easy to get backwards: The deep-index checkpoint commits in the SAME transaction as the data it vouches for, so a rollback can never leave a checkpoint pointing at rows that are not there. The attempt counter commits in its OWN transaction, opened and committed BEFORE the batch work transaction. This is what makes the poison pill work. A crash mid-asset must leave evidence behind, and a statement inside an open transaction is not durable -- SQLite runs a DELETE journal here, so the crash would roll the marker back along with the work and the counter would still read zero on resume. The asset that killed the editor would then be re-queued forever: a crash-every-launch loop, strictly worse than the bug being fixed. The frame-budget commit does not cover it either, since it only fires after at least one asset has been processed and so never protects the FIRST asset of a batch -- exactly where a resumed queue puts the poison asset. After repeated interrupted attempts an asset is skipped, logged, and surfaced through project get_stats rather than retried indefinitely. The completion gate tolerates per-asset failures. Making last_full_index conditional on a zero error count would reproduce this very issue from the opposite direction: deep-index errors are routine on real projects (assets referencing classes from disabled plugins, redirector stubs) and the codebase already carries an SEH guard because some animation assets crash on load. One such asset would mean the marker is never written and every launch runs a full index. Only structural failures fail the gate now. bDeferFirstTimeIndex is honoured on the resume path. It is the escape hatch for the GC worker-context crash class, so bypassing it when resuming would strip it at the exact moment a user needs it. Also routes a non-force monolith_reindex to the resume path. Without last_full_index an interrupted run cannot pass the incremental check, so it fell through to StartFullIndex and reset -- and since that action is how an agent typically kicks the index after a crash, it silently destroyed the progress this change exists to preserve. Reported and diagnosed by @Hvizeu (#117), whose PR #120 established the lifecycle design this follows.
|
Shipped in v0.22.0. I land contributor fixes as my own commits rather than merging the branch — I keep the shipped history single-author for release integrity, and credit you in the release notes instead. No reflection on the patch. Your diagnosis was exactly right and I verified every link: I did land it smaller, and there were three things I changed rather than took: The poison pill. This is the important one. Your version records completions but never attempts, so a hard crash leaves no trace and the resume re-queues the asset that killed the editor — a crash-every-launch loop, which is worse than the bug. It is made sharper by the Subtlety worth recording, because I got it wrong first and a review caught it: the attempt marker has to commit in its own transaction before the batch work. Written inside the batch transaction it is not durable — SQLite runs a DELETE journal here, so the crash rolls the marker back along with the work and the counter still reads zero on resume. The pill can never fire in the one case it exists for. The completion gate. The animation indexer. Per-asset transactions and Also not taken: post-pass resume checkpoints (they self-invalidate on the common interruption case), the One note on the checklist: the three automation tests exercise |
A full index wiped the database at the start and wrote its completion marker only at the very end, so any crash, kill or power loss left no marker -- and the next launch began another full index, which immediately erased every batch the previous run had finished. On a large project one crash cost hours; repeated crashes meant the index could never complete at all. Schema v3 adds deep_indexed_hash and deep_index_attempts to assets, plus an in_progress marker in meta. Migration is additive and leaves existing rows untouched; a v0.21.3 build reading a v3 database is inert rather than broken. Force-reindex still wipes, which is its documented purpose. Two records with deliberately DIFFERENT durability requirements, which is the whole mechanism and is easy to get backwards: The deep-index checkpoint commits in the SAME transaction as the data it vouches for, so a rollback can never leave a checkpoint pointing at rows that are not there. The attempt counter commits in its OWN transaction, opened and committed BEFORE the batch work transaction. This is what makes the poison pill work. A crash mid-asset must leave evidence behind, and a statement inside an open transaction is not durable -- SQLite runs a DELETE journal here, so the crash would roll the marker back along with the work and the counter would still read zero on resume. The asset that killed the editor would then be re-queued forever: a crash-every-launch loop, strictly worse than the bug being fixed. The frame-budget commit does not cover it either, since it only fires after at least one asset has been processed and so never protects the FIRST asset of a batch -- exactly where a resumed queue puts the poison asset. After repeated interrupted attempts an asset is skipped, logged, and surfaced through project get_stats rather than retried indefinitely. The completion gate tolerates per-asset failures. Making last_full_index conditional on a zero error count would reproduce this very issue from the opposite direction: deep-index errors are routine on real projects (assets referencing classes from disabled plugins, redirector stubs) and the codebase already carries an SEH guard because some animation assets crash on load. One such asset would mean the marker is never written and every launch runs a full index. Only structural failures fail the gate now. bDeferFirstTimeIndex is honoured on the resume path. It is the escape hatch for the GC worker-context crash class, so bypassing it when resuming would strip it at the exact moment a user needs it. Also routes a non-force monolith_reindex to the resume path. Without last_full_index an interrupted run cannot pass the incremental check, so it fell through to StartFullIndex and reset -- and since that action is how an agent typically kicks the index after a crash, it silently destroyed the progress this change exists to preserve. Reported and diagnosed by @Hvizeu (tumourlove#117), whose PR tumourlove#120 established the lifecycle design this follows.
2 participants
What and why
Full-project indexing currently starts over after an editor shutdown or crash because an interrupted run never writes
last_full_index. On large projects, repeated restarts can prevent the initial index from ever completing.This change adds a schema-v3 recovery lifecycle that:
last_full_indexChecklist
ENGINE_MINOR_VERSIONgate, with a working 5.7 path (N/A: no newer engine API added)Docs/specs/SPEC_MonolithIndex.md)CHANGELOG.mdupdated under## [Unreleased]Testing notes
git diff --check upstream/masterpasses.Monolith.Index.Recovery.CheckpointTransaction,Monolith.Index.Recovery.Lifecycle, andMonolith.Index.Recovery.PostPassCleanuppass under UE 5.7: 3 succeeded, 0 failed.Editor Win64 Developmentcompleted the final 49-action build successfully, including the changed Monolith sources, tests, dependent plugin modules, and DLL links.Editor Win64 Developmentcompleted the final 23-action build successfully, including the changed Monolith sources, tests, dependent plugin modules, and DLL links.