[aryasaatvik]

Why

Production traces showed the reindex scan phase dominating wall-time: ~30 min for an all-skip pass over ~5,400 tools, ~3 tools/s. The cause is read amplification — scan calls tools.manifest() (a full tool_schema_manifest table read) once per partition page, then filters to its partition and slices SCAN_PAGE rows. With PARTITION_COUNT=12 and SCAN_PAGE=25 that's ~216 scan messages, each re-reading all ~5,400 rows cross-region (SIN→NRT): O(N²/pageLimit) ≈ 1.17M row-transfers to scan 5,400 tools. partition is computed in-app (hash(path) % count), so it can't be pushed down to SQL without a migration.

What

• @executor-js/sdk — createExecutor exposes its cache KeyValueStore on the instance as executor.cache (Cloudflare KV in prod, in-memory fallback otherwise). Plugins reach durable cache storage without re-threading a binding; wrap with KeyValueStore.toSchemaStore like the internal tools.list / schema-view caches.
• @executor-js/plugin-semantic-search — create reads the manifest once, partitions it (preserving each entry's full-list ordinal), and writes one snapshot per partition to executor.cache (index-manifest/{runId}/{partition}). scan reads only its partition's snapshot — KV-only, no D1 fallback on the hot path. A miss fails the message so the queue retries; the host delays the initial scan fan-out once to let the write propagate (separate host PR). Snapshots are deleted on run completion.

Result

• D1 manifest reads: ~1.17M → ~5,400 (a single pass at create).
• Per-scan-message memory: full manifest (~5 MB) → one partition (~400 KB).
• Scan wall-time: ~30 min → seconds–minutes (and the host delay PR keeps the fast path certain).

Safety / notes

• KV-only by design: a snapshot miss surfaces loudly (queue retry / DLQ) rather than silently degrading to the slow D1 path.
• Snapshots are partition-sharded so no single KV value approaches the 25 MB cap (scales to ~300k tools).
• KV has no native TTL through the abstraction, so cleanup is explicit on completion; a stalled / terminally-failed run leaves its (small) snapshot until overwritten — a known, bounded leak.
• Behavior preserved: the scan's skip-vs-changed classification, ordinal ordering, pagination, and maxTools cap are unchanged (the snapshot is built from the same listToolManifests call).

Gate

tsgo 43/43 · oxlint --deny-warnings 0/0 · oxfmt · vitest sdk 350/350, plugin-semantic-search 78/78 (new: manifest-read-once-per-create, KV-miss-fails-no-fallback, snapshot-cleared-on-complete).

Stacked deploy: needs the host PR that delays the initial scan fan-out for KV propagation.

[greptile-apps]

Greptile Summary

This PR eliminates the O(N²/pageLimit) D1 read amplification in the semantic-search reindex scan phase by snapshotting the full tool manifest once at run creation, partitioning it into per-partition KV slices, and having each scan message read only its slice from KV. The createExecutor instance now exposes executor.cache so plugins can reach the same durable KV store without re-threading a binding.

• packages/core/sdk/src/executor.ts: Adds readonly cache: KeyValueStore.KeyValueStore to the Executor type and wires it from the already-existing internal cacheStore.
• packages/plugins/semantic-search/src/sdk/manifest-snapshot.ts: Introduces schema types (ManifestSnapshotEntry, ManifestSnapshotItem) and the key-derivation helper for per-partition KV snapshots.
• packages/plugins/semantic-search/src/sdk/tool-search-index.ts: create writes the snapshot before the D1 run record; scan reads only its partition's KV slice (hard fail on miss, no D1 fallback); fail (now typed as IndexStores instead of IndexCollections) and complete both call deleteManifestSnapshot at terminal state. Four new tests verify the manifest-read-once, KV-miss-fails, and cleanup-on-complete/fail behaviours.

Confidence Score: 5/5

Safe to merge; the optimisation is well-scoped and the previously flagged issues (terminal-fail cleanup and inaccurate reconcile comment) are both addressed in this revision.

The snapshot write-before-D1 ordering, the best-effort delete, and the KV-only hard-fail-on-miss are all deliberate, acknowledged trade-offs. Test coverage is thorough: four new test cases exercise exactly the invariants the PR claims (manifest read once, KV miss fails, snapshot cleared on complete, snapshot cleared on terminal fail). The only remaining nit is a dead maxTools field on ScanInput, which is a misleading interface artefact and not a runtime defect.

No files require special attention. tool-search-index.ts is the heaviest change but the test suite covers all new branches.

Important Files Changed

Filename	Overview
packages/core/sdk/src/executor.ts	Exposes the already-existing internal cacheStore as executor.cache — a two-line addition with no behavioural change to existing functionality.
packages/plugins/semantic-search/src/sdk/manifest-snapshot.ts	New file; defines the Effect Schema types and key-derivation helper for per-partition KV snapshots. Logic is straightforward and well-documented.
packages/plugins/semantic-search/src/sdk/tool-search-index.ts	Core logic change: create writes the KV snapshot before the D1 run record; scan reads only the partition slice; fail type widened to IndexStores; deleteManifestSnapshot called at both terminal states. ScanInput.maxTools is now a dead field — the constraint is baked into the snapshot at create time.
packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts	Adds an in-memory cache mock and four targeted snapshot tests covering manifest-read-once, KV-miss hard-fail, snapshot-cleared-on-complete, and snapshot-cleared-on-terminal-fail.
.changeset/semantic-search-manifest-snapshot.md	Changeset marking both affected packages as minor version bumps with an accurate description of the optimisation.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Host
    participant Cr as create()
    participant KV as executor.cache
    participant D1
    participant Sc as scan()

    Host->>Cr: runId, partitionCount, maxTools
    Cr->>D1: listToolManifests (once)
    D1-->>Cr: manifests[0..N]
    Cr->>KV: write partition snapshots (one key per partition)
    KV-->>Cr: ok
    Cr->>D1: runs.put(runId, running)
    D1-->>Cr: ok
    Cr-->>Host: total, partitionCount

    loop queue fan-out per partition
        Host->>Sc: runId, partition, limit
        Sc->>KV: readManifestSnapshot(runId, partition)
        KV-->>Sc: partition slice
        Sc->>D1: jobs.count(runId, partition)
        D1-->>Sc: existing
        Sc->>D1: putJobs(selected page)
        Sc-->>Host: hasMore
    end

    Host->>D1: complete / fail (terminal)
    D1->>KV: deleteManifestSnapshot (best-effort)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Host
    participant Cr as create()
    participant KV as executor.cache
    participant D1
    participant Sc as scan()

    Host->>Cr: runId, partitionCount, maxTools
    Cr->>D1: listToolManifests (once)
    D1-->>Cr: manifests[0..N]
    Cr->>KV: write partition snapshots (one key per partition)
    KV-->>Cr: ok
    Cr->>D1: runs.put(runId, running)
    D1-->>Cr: ok
    Cr-->>Host: total, partitionCount

    loop queue fan-out per partition
        Host->>Sc: runId, partition, limit
        Sc->>KV: readManifestSnapshot(runId, partition)
        KV-->>Sc: partition slice
        Sc->>D1: jobs.count(runId, partition)
        D1-->>Sc: existing
        Sc->>D1: putJobs(selected page)
        Sc-->>Host: hasMore
    end

    Host->>D1: complete / fail (terminal)
    D1->>KV: deleteManifestSnapshot (best-effort)

Loading

_{Reviews (2): Last reviewed commit: "fix(semantic-search): correct snapshot r..." | Re-trigger Greptile}