Frontmatter values are not indexed as searchable text (canonical phrase miss)

[Squintsisastud]

Summary

Values stored in YAML frontmatter of an ingested markdown page are not promoted into the searchable chunk text, so they don't hit on tools/call search. The page is otherwise indexed and discoverable via body content.

This matters because a common pattern is to put a unique "canonical search phrase" in frontmatter as metadata, then expect the agent to find the page by querying that phrase. The page is found if the phrase also appears in the body, but not if it only lives in frontmatter.

Repro

Vault file:

---
type: dr-payables-snapshot
date: 2026-05-19
entity: Forever Crystals, S.R.L.
canonical_search_phrase: dr-payables-usd-2026-05-19-quechibot-canonical
---

# DR Cuentas por Pagar (USD) — 2026-05-19

Entidad: **Forever Crystals, S.R.L.**
...body content with vendor names like ANTARES, LANDMARK REALTY, etc...

After a full sync (launchctl kickstart gui/$(id -u)/com.openclaw.gbrain.daily-sync), the page is indexed and the page count for type: dr-payables-snapshot increments correctly.

Expected

Searching for dr-payables-usd-2026-05-19-quechibot-canonical via MCP tools/call search returns this page.

Actual

• search "dr-payables-usd-2026-05-19-quechibot-canonical" → []
• search "Landmark Realty Corp" (body term) → returns the page (page_id 1834, slug finance/dominican-republic/payables/2026-05-19-dr-payables-usd, type dr-payables-snapshot)

So the page is in the graph, but tokens that exist only in YAML are unreachable.

Versions

• gbrain 0.31.10 (pglite engine)
• macOS 25.3.0 arm64

Why this matters

Multi-agent setups (in our case OpenClaw + Hermes) use frontmatter-tagged canonical phrases as a clean way for the issuing agent to write a receipt, and for the receiving agent (or a human) to verify the document is indexed and accessible. Today the only safe workaround is to duplicate the phrase into the markdown body, which works but defeats the purpose of structured metadata.

Suggested fix options

1. Include scalar frontmatter values in the searchable text for each page chunk (probably the cleanest). The values are short and the index cost is negligible. Could be gated by a config flag (e.g. search.index_frontmatter_values: true) so existing behavior is preserved if anyone depends on it.
2. Promote a known set of "search-relevant" keys (e.g. aliases, tags, canonical_search_phrase, title) into the searchable chunk by default, and let users extend the allowlist.
3. At minimum, document the current behavior so authors know to put canonical phrases in the body, not only frontmatter.

Happy to PR option 2 if it sounds right; let me know which direction you prefer.

Related

Not a blocker — we've already worked around it by adding **Canonical GBrain search phrase:** \`` to the body of every receipt-bearing file. Filing because (a) the YAML-only failure mode is surprising and (b) frontmatter-as-metadata is a natural pattern for agent receipts.

[Squintsisastud]

Update — body placement isn't enough either

Followed my own workaround and added the phrase to the markdown body as **Canonical GBrain search phrase:** \``. Confirmed via grep that it's in the body. Re-synced. Still misses.

Digging deeper, the issue isn't just frontmatter scoping — it's that hyphen-concatenated identifier-style phrases like:

dr-payables-usd-2026-05-19-quechibot-canonical
caja-bento-scoped-keys-2026-05-20-quechibot-canonical

don't match as literal strings, even when present in the body. The same page is reachable by natural-language queries like "scoped api keys replit" or "caja bento scoped keys 2026".

Looks like the tokenizer is splitting on hyphens (so each token becomes dr, payables, usd, 2026, etc.) but then BM25/vector match against the concatenated identifier as a whole fails because no chunk surface preserves the full identifier as one token, AND the individual tokens are too common to rank the right page first.

Refined ask

Two things would help:

1. Index frontmatter scalar values (original ask).
2. Preserve identifier-style phrases as searchable tokens. Either:
   • Add an alternate tokenization pass that keeps hyphenated identifiers intact (e.g. anything matching [a-z0-9]+(-[a-z0-9]+){2,} as a literal token in addition to the split tokens), or
   • Index a normalized phrase-id form derived from anchors/code spans in the body (e.g. inline \code`` spans become indexed literally).

The second approach (treating inline \code`` spans as literal-search tokens) feels like the cleanest behavioral fix because it gives authors an explicit way to opt a phrase into literal search just by wrapping it in backticks.

Workaround for now

Receipts ship two phrases:

• A literal identifier in backticks for structured cross-reference.
• A natural-language phrase ("Caja Bento finance agent access handoff 2026-05-20") that the existing chunker handles fine for human verification.