Skip to content

feat(core): index failed tool results for retrieval - #15

Open
Cyb3rN8 wants to merge 1 commit into
tommy0103:mainfrom
Cyb3rN8:feat/index-failed-tool-results
Open

feat(core): index failed tool results for retrieval#15
Cyb3rN8 wants to merge 1 commit into
tommy0103:mainfrom
Cyb3rN8:feat/index-failed-tool-results

Conversation

@Cyb3rN8

@Cyb3rN8 Cyb3rN8 commented Jul 30, 2026

Copy link
Copy Markdown

search() covers messages.text and nothing else, so a past failure is findable only when the assistant happened to quote the error — and it usually doesn't, because the model fixes an error rather than restating it. That makes "how did I fix this last time" miss precisely the cases it gets asked about most.

Measurement

Against a real index — 380k messages, 127k tool results, 2.5 GB of transcripts — I sampled 400 error identifiers (ENOENT, TypeError, error TS2339, …) taken from is_error=1 rows and asked whether messages_fts could locate the session they occurred in:

share
found in the right session via messages_fts 33.7%
found only in other sessions (wrong hit) 29.0%
absent from messages.text entirely 37.3%

InputValidationError (96 rows) and JSONDecodeError (52 rows) are typical: zero hits in messages_fts, because they are runtime class names that only ever appeared in tool output.

For contrast, the same probe over all tool output — not just failures — is far less alarming (48.8% locatable), and over file names it is 74%. Failures are the sharp edge, which is why this PR indexes only them.

What this does

tool_errors_fts indexes is_error rows only — 0.6% of tool_results by size (1.3 MB here), about 6 MB of inverted index on a 950 MB database.

Successful tool output stays unindexed on purpose. It is ~99% of the volume (Read 98.9 MB, Bash 90.7 MB on my index), and full-text indexing it would add roughly 1.1 GB. It doesn't earn that: once retrieval has produced a session_id, a LIKE scoped to it is ~1 ms via idx_tr_session, and even an unscoped scan of all 209 MB is 1.7–3.2 s — fine as a deliberate last resort.

Refreshed wholesale in finalize, not maintained by triggers. tool_results is written with INSERT OR REPLACE, and REPLACE fires DELETE triggers only when recursive_triggers is on (off by default) — a trigger pair would leave superseded error text searchable forever, and re-indexing the same tool call would silently accumulate duplicates. A full refresh of a few thousand rows costs ~0.5 s warm (4.7 s on a cold page cache), against a 113 s full build. Both index paths do it: packages/core/src/indexer.ts via rebuildToolErrorsFts(), and the app's rebuildFts().

Every write path stays correct without bookkeeping, since the refresh is unconditional: force build (DELETE FROM tool_results then re-index), incremental build, and per-session retraction all converge.

Verification

tests/tool-errors-fts.test.mjs, 7 tests — including the two that motivate the no-trigger decision: re-indexing the same tool_use_id leaves no stale text searchable, and a result that stops being an error drops out. Also idempotence, is_error filtering, session pivot, and the force-build path.

Full suite / lint / typecheck / build:core / build:cli: no regression — failing tests on my machine are identical with and without this change (22 files, all better-sqlite3, which cannot build on Node 26). I could not exercise the app's rebuildFts() at runtime for that reason, only typecheck it — that call site deserves a second look.

Notes

🤖 Generated with Claude Code

search() covers messages.text only, so a past failure is findable just when the
assistant happened to quote the error. Measured against a 380k-message index,
~37% of failed tool calls leave no trace in messages.text at all -- the model
fixes the error rather than restating it -- which makes "how did I fix this last
time" miss exactly the cases it is asked about most.

tool_errors_fts indexes is_error rows only: 0.6% of tool_results by size, ~6MB
of inverted index on a 950MB database. Successful tool output stays unindexed --
it is 99% of the volume, and a LIKE scoped to a session_id is already ~1ms once
retrieval has produced one.

Refreshed wholesale in finalize rather than maintained by triggers: tool_results
is written with INSERT OR REPLACE, and REPLACE fires DELETE triggers only when
recursive_triggers is on, so a trigger pair would leave superseded error text
searchable forever. A full refresh of a few thousand rows costs ~0.5s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cyb3rN8 pushed a commit to Cyb3rN8/obelisk that referenced this pull request Aug 19, 2026
…es up with the search guard

Companion to the two upstream branches merged above, covering the local-only
surface (tommy0103#15's tool_errors_fts and the trigram-pinned skill doc):

- persist writes tool_results with ON CONFLICT DO UPDATE instead of INSERT OR
  REPLACE: the rowid stays stable and the UPDATE trigger fires, which is what
  makes trigger maintenance of tool_errors_fts sound (REPLACE fired no DELETE
  trigger — the reason this table was refreshed wholesale every finalize).
- schema.sql grows rowid-aligned ai/ad/au triggers on tool_results (is_error
  rows only); hash pin updated in provider-schema-stability.test.mjs.
- rebuildToolErrorsFts is now rowid-aligned and runs once per index under the
  __tool_errors_fts_synced__ marker (same one-time-heal pattern as
  __messages_fts_synced__), removing the remaining ~5.5 s fixed cost of every
  incremental build. The force path relies on the triggers (its DELETE FROM
  tool_results cascades through AD), fixing the pre-existing gap where a force
  rebuild left tool_errors_fts stale until the next incremental finalize.
- SKILL.md: the short-term guidance now describes the automatic guard
  (degraded: short-token-post-filter / like-scan) and the hit-centered snippet
  field instead of prescribing manual sql() LIKE fallbacks.
- tests/tool-errors-triggers.local.test.mjs pins every persist write shape
  against the triggers and the refresh's rowid alignment.

Incremental build fixed cost measured on this machine's 1.8 GB index:
~30.2 s (messages_fts rebuild) + ~5.5 s (tool_errors_fts refresh) → both now
one-time heals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant