Skip to content

fix(memory): filtered search actually filters; migrate AgentDBClient to agentdb 3.7.2 (2.3.0) - #18

Merged
ruvnet merged 2 commits into
mainfrom
weaver/dspy-agentdb37
Nov 9, 2026
Merged

ruvnet merged 2 commits into
mainfrom
weaver/dspy-agentdb37

Conversation

@ruvnet

@ruvnet ruvnet commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Bug

AgentDBClient.search(q, { filter }) passed the filter to a backend that never read it (src/memory/agentdb/client.ts:270 → the in-memory Map store at :172). Filtered searches came back unfiltered. Checked against the old client: filter: { kind: 'x' } returned all 4 items, and a filter matching nothing also returned 4. The RaBitQ path dropped the filter too.

Change

  • agentdb ^1.6.1 → ^3.7.2, moved to optionalDependencies and loaded lazily with a native import(). agentdb 3.x is ESM-only, and tsc's CommonJS output would turn import() into require().
  • When agentdb loads, vectors go to an agentdb RVF store (agentdb/backends createBackend('rvf')):
    • async searchAsync
    • native name-based filters { op, field, value } (the shape FilterBuilder emits)
    • isolated in-memory stores
    • cleanup() calls closeAsync(), which releases the lock and removes the temp dir
    • vectorDimension maps to dimension
    • a durable .rvf file when an explicit storage.path is given. The default config never writes to disk.
  • Fallback: an in-memory store with the same filter semantics, applied before top-k. It is used on Node < 20.9 or when agentdb is not installed.
  • toFilterExpr normalises the predicate DSL ($eq $ne $lt $le $gt $ge $in $range) and raw expressions. Unsupported shapes throw instead of quietly widening the search. FilterBuilder silently skips them.
  • EmbeddingService now uses the 3.x config shape. The old { dimensions } config throws on 3.x. It loads on the first embed(), and agentdb's silent mock-embedding fallback counts as unavailable.
  • Deterministic errors (metadata type conflict, size limits, dimension mismatch) are no longer retried.
  • New: storage.backend (auto | agentdb | memory), backendKind, isPersistent, and exported filter helpers.

Tests (193 pass, previously 180)

  • tests/memory/agentdb-real.spec.ts runs a child process (ts-node) against real agentdb 3.7.2, because jest's vm can't import() ESM. It checks:
    • filtered search returns only matching items (DSL, expr, range, filter before top-k)
    • in-memory isolation and temp-dir removal
    • persistence across close and reopen
    • Skipped on Node < 20.9, since agentdb requires >=20.9.0.
  • tests/memory/agentdb-filter.spec.ts covers the fallback path, a regression test for the ignored filter, the RaBitQ path, and rejection of unsupported filters.

npm audit --omit=dev

  • Before: 24 (2 critical, 10 high, 12 moderate)
  • After: 13 (1 critical, 2 high, 10 moderate)
  • Removed: agentdb, @xenova/transformers, sharp and @modelcontextprotocol/sdk highs from the agentdb 1.x chain, plus stale transitive lock entries (hono, express, body-parser, proxy-addr, fast-uri, ip-address).
  • Remaining: these are not from agentdb. tar/gl/node-gyp come via js-pytorch; zod is a direct ^3 dependency and fixing it needs zod 4.

Notes

  • Minor bump to 2.3.0: new options, and isNative now means "agentdb vector backend active".
  • agentdb keeps one type per metadata field.
  • Tiers, sign codes and includeVectors are tracked in process, so they are not restored after a reopen. This is documented in the README and CHANGELOG.
  • The README gains a "Works with agentdb" section.

🤖 Generated with claude-flow

…entdb 3.7.2

AgentDBClient.search({ filter }) handed the filter to a backend that never
read it, so filtered searches returned unfiltered results (a filter matching
nothing returned everything). The RaBitQ path ignored it as well.

- agentdb ^1.6.1 -> ^3.7.2, now an optional dependency loaded lazily with a
  native import() (agentdb 3.x is ESM-only; tsc's CJS output would turn
  import() into require()).
- With agentdb loaded, vectors live in an agentdb RVF store
  (agentdb/backends createBackend('rvf')): async search, native name-based
  filters ({ op, field, value }), isolated in-memory stores, closeAsync() on
  cleanup, durable single-file store when an explicit storage.path is given.
- Fallback in-memory store applies the same filter semantics before top-k.
- Filters are normalised by toFilterExpr; unsupported shapes throw.
- EmbeddingService uses the 3.x config and is loaded on first embed();
  agentdb's mock-embedding fallback counts as unavailable.
- Deterministic backend errors are not retried.

Tests: real-import child-process suite against agentdb 3.7.2 (filtered
search, isolation, persistence across reopen), fallback filter suite, and a
regression test for the ignored filter. Release 2.3.0.

Co-Authored-By: claude-flow <ruv@ruv.net>
… is absent (Node 18)

npm skips optional agentdb 3.x on Node 18 (engines >=20.9), and jest.mock of
a missing module fails to resolve.

Co-Authored-By: claude-flow <ruv@ruv.net>
@ruvnet
ruvnet merged commit fe83ea3 into main Nov 9, 2026
4 checks passed
@ruvnet
ruvnet deleted the weaver/dspy-agentdb37 branch November 9, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant