feat(search): let a caller or a tenant exclude fan-out children (09/18 c-03) by arkash20 · Pull Request #1708 · caura-ai/caura

arkash20 · 2026-09-23T12:32:13Z

Implements the decision taken on pm-0918-c-03: option (a) — a request flag defaulting to today's behaviour, plus a per-tenant default. The evidence behind the choice is in #1703 (docs-only, merges independently).

A70's fan-out writes a short single-claim child per extracted fact, and those children stay retrievable alongside their parent. On the store that prompted this they were ~28% of returned rows — a caller asking for 50 memories got 36 memories and 14 fragments, worth 79.8% vs 82.2% on that harness.

Filtering them client-side does not work, and that is the design constraint: dropping rows after the response distorts top_k. Ask for 50, drop 14, get 36, then over-fetch and guess. So the filter runs server-side and before the trim, where the existing 2× SEARCH_OVERFETCH_FACTOR absorbs it — at ~28%, 100 candidates become ~72 survivors and all 50 slots still fill.

Three layers, request beats tenant beats global

layer	where	default
`include_derived` on the request	`POST /search`, `POST /recall`	unset (tri-state)
`search.include_derived`	per tenant, `PUT /settings`	unset
`INCLUDE_DERIVED_DEFAULT`	`constants.py`	`True` — unchanged

The global default deliberately does not move. /search is in the frozen broker subset and a default flip is a semantic change oasdiff cannot see, so it needs a BREAKING CHANGE: trailer and a second store's evidence. A tenant that wants that behaviour today sets search.include_derived=false and gets it for its own store.

The predicate is a conjunction, and both halves are load-bearing

parent_memory_id IS NOT NULL  AND  source = 'atomic_fact_fanout'

parent_memory_id alone also catches auto-chunk children, which are not redundant — the parent holds the whole document but carries one embedding over all of it, so those children are the only vectors that can match a specific passage. Excluding them would surface days later as "long documents stopped being findable" with nothing pointing at this filter. And source alone is forgeable: it is deliberately absent from PLATFORM_ONLY_KEYS because ingest writes it too. The argument sits in a comment at the predicate, not only in the doc.

Applied in PostFilterResults and in the legacy search path from one shared helper in search_trim.py, so the _USE_PIPELINE_SEARCH rollback lever cannot silently revert it — two ranking features already shipped pipeline-only for exactly that reason.

Not repeating the c-04 defect

search.include_derived is registered in DEFAULT_SETTINGS, in _LEAF_TYPES as bool, and on the resolver. On pm-0918-c-04 the switch existed only as a ResolvedConfig property: _check_keys rejected every write, PUT /settings answered 422, and the resolver served the default anyway — so it read as shipped.

test_the_tenant_default_survives_a_real_settings_put goes through the HTTP route, not a constructed ResolvedConfig, because constructing it directly is what hid that. Verified by removing the DEFAULT_SETTINGS entry and re-running, which fails with the exact c-04 error:

AssertionError: PUT failed — the knob is unsettable:
{"detail":"Unknown settings key(s): ['search.include_derived']"}

The bool leaf type earns its place separately: "false" is a truthy string, so without it a tenant that switched derived rows off would resolve to ON while the dashboard rendered their "off" back to them.

Verification

20 new tests, all passing. Precedence is tested at all three layers including the both-supplied case in both directions, and the global default is pinned with a test whose failure message says to use a breaking-change trailer rather than edit the assertion.
Full suite: 93 failures before and after, 7,754 → 7,774 passing (+20). The 93 are pre-existing local failures — baselined by running the identical suite on unmodified origin/main.
ruff check + ruff format --check clean. mypy reports zero errors in the changed files (the 142 in the transitive graph are pre-existing and identical at baseline).
Broker baseline regenerated in its own commit; --check passes and the pinned oasdiff image reports "No breaking changes to report, but the specs are different", exit 0.

MCP: a limitation, and a silent drop that is now its own row

This PR does not add include_derived to the MCP tools — that pulls in the 40+ location tool-surface checklist, and the tenant setting covers MCP callers. /recall was wired, and the line between the two is worth stating so it does not read as inconsistent: RecallRequest subclasses SearchRequest, so a caller can send the field to /recall whether or not the route reads it — accepting and ignoring it is silent divergence, the class of ax-0917-m-19 / m-16 / oss-0922-m-03. MCP does not expose the field at all, which is a limitation rather than a divergence.

That last claim was checked rather than assumed, and it did not survive. Probing mcp.call_tool("caura_recall", ...) through the real dispatch:

argument sent	what happened
`include_derived: false`	`search_memories` received `include_derived=<ABSENT>`
`bogus_param_xyz: 2`	no error, no warning
`limit: 2`	no error; response reports `effective_top_k: 5`

Every one returns a normal success envelope with isError absent. The argument is dropped during the SDK's argument binding, before the handler. So MCP is a silent-drop surface too — and limit silently returning the default 5 is ax-0917-h-05's original defect, still unfixed on the surface agents actually use.

That is broader than this row (it is the argument-binding path, so all 12 tools) and is filed as oss-0923-h-01 rather than widening this PR. The field description now says so at the point of use: REST only, use the tenant setting from MCP, and an unknown MCP argument is dropped silently rather than reported.

For Arkady — flipping PersonaMem's tenant

Not run here: this is a live-data action on a real tenant. Idempotent, replace the placeholder:

curl -sS -X PUT "https://memclaw.dev/api/v1/settings?tenant_id=<PERSONAMEM_TENANT_ID>" \
  -H "X-API-Key: $CAURA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"search": {"include_derived": false}}'

Read it back with curl -sS "https://memclaw.dev/api/v1/settings?tenant_id=<PERSONAMEM_TENANT_ID>" -H "X-API-Key: $CAURA_API_KEY" | jq .search.include_derived — expect false. Settings are cached per tenant, so allow for the cache TTL before re-measuring.

🤖 Generated with Claude Code

github-actions · 2026-09-23T12:32:23Z