P0: Cross-layer retrieval returns UI noise and drives agents into broad fallback

[mohanagy]

Problem

A real activation test against openstatusHQ/openstatus with @lubab/madar@0.31.3 used this natural codebase question:

Trace how a failed monitor check becomes an incident, triggers notifications, and affects the public status-page status in this repository. Cite the exact files and symbols involved, identify any inconsistent status-computation paths, and clearly state any remaining uncertainty. This is read-only; do not change files.

Automatic activation worked and Madar was used before raw repository context. Retrieval quality did not:

• Claude called Madar roughly ten times;
• most packs repeatedly surfaced unrelated status-page UI rendering components;
• Madar eventually surfaced apps/workflows/src/checker/alerting.ts, but did not assemble the end-to-end backend flow;
• Claude then used direct reads, grep, and shell discovery to finish the answer;
• the useful final answer was attributable primarily to raw source inspection, not the graph pack.

The missing flow crosses boundaries that a call graph alone does not naturally connect:

• Go checker failure handling;
• a hard-coded HTTP/Cloud Tasks route boundary;
• TypeScript workflow ingestion;
• writes and reads through incidentTable, monitor.status, and statusReport;
• provider notification dispatch;
• tRPC, JSON-feed, and Connect-RPC status-page read models.

This is a real-world follow-up to #552 and #555. Their machinery is present, but their intended agent outcome is not achieved for this cross-layer query.

Expected behavior

For a multi-obligation runtime-flow question, Madar should internally decompose the request, retrieve evidence for each obligation, expand across structural/data/integration boundaries, merge the cumulative evidence, and return one actionable answerability contract. The agent should not need to invent ten follow-up queries.

Proposed retrieval stages

1. Detect the question's obligations: failure source, transport, incident persistence, notification dispatch, public read model, and inconsistencies.
2. Retrieve diverse anchors per obligation rather than taking one global top-k dominated by UI vocabulary.
3. Expand through callers/callees/imports plus route literals, shared persistence symbols, and writer/reader relationships.
4. Merge, deduplicate, and rescore cumulative evidence after one or two bounded recovery attempts.
5. Return exact missing obligations and verification targets, or terminal insufficient with a scoped fallback allowance.

Acceptance criteria

• Add a reproducible OpenStatus packed-artifact receipt for the exact prompt, without committing the external repository contents.
• Add a small committed regression fixture that represents a cross-language or cross-service flow through a route boundary and shared persistence state.
• Retrieval identifies direct evidence for all five answer obligations: failure/transport, incident state, notifications, public status computation, and inconsistencies.
• The real OpenStatus pack includes relevant evidence from the checker, workflow, schema/service, and status-page layers.
• At least 70% of selected files in the receipt are human-judged relevant, with no more than two unrelated UI files.
• One initial retrieval plus at most two bounded internal recovery attempts produces the cumulative result.
• A normal Claude or Codex session needs no more than two attributable Madar tool calls for the question.
• verify_targets permits only its listed targets; broad raw search is allowed only for terminal insufficient with broad_search_fallback: allowed.
• The successful agent run uses no repo-wide grep/glob, shell discovery, or Explore subagent.
• Retrieval reasons expose which obligations, communities, and fallback stages changed the result.
• Expected OpenStatus filenames/symbols exist only in evaluation data; none are added to production retrieval rules.
• Claude and Codex each pass three consecutive documented trials before a release claim is made.
• Final answers receive human semantic review and are no worse than the direct-search baseline.

Success metrics

• first useful cumulative pack within 10 seconds on a warm graph;
• no more than two agent-visible Madar calls;
• complete obligation coverage;
• selected-file precision >= 70%;
• zero unauthorized broad fallback;
• fewer total repository tool calls than the direct-search baseline.

Relevant code

• src/runtime/retrieve.ts
• src/runtime/retrieve/conceptual-fallback.ts
• src/runtime/retrieve/slicing.ts
• src/runtime/context-pack.ts
• src/runtime/mcp-response-evidence.ts
• src/contracts/retrieval-plan.ts
• tests/unit/retrieve-conceptual-fallback.test.ts
• tests/unit/pack-quality-fixtures.test.ts
• docs/benchmarks/suite/

Related

• Build a multi-repo benchmark suite for adoption proof #469
• [P1] Replace scalar pack confidence with cumulative evidence recovery #552
• [P1] Add conceptual-query retrieval fallbacks and self-hosted regressions #555

[mohanagy]

Implementation is now in draft PR #566 with a checked-in share-safe OpenStatus acceptance receipt.

The exact prompt returns 8 relevant symbols across 7 unique files, no unrelated UI files, 1,666/1,800 serialized tokens, confidence 0.89, answer_from_pack, and broad_search_fallback: not_needed. All repository tests pass when run in bounded forked batches (2,729 passed, 1 skipped).

This issue remains open intentionally. Before a release/public claim, we still need three consecutive Claude trials, three consecutive Codex trials, and human semantic comparison against the direct-search answer.

[mohanagy]

v0.31.4 strict SPI rerun — still PARTIAL

A fresh OpenStatus design-partner run was repeated after a confirmed SPI build:

{ "use_spi": true, "spi_mode": true, "nodes": 15653 }

The agent made one Madar retrieval call and was forbidden from shell/search or opening paths not returned by the pack.

What improved

The pack directly covered notification delivery/deduplication and the public JSON-feed route:

• apps/workflows/src/checker/alerting.ts
• apps/status-page/src/app/api/status/[[...path]]/route.ts
• apps/status-page/src/content/status-json.ts

It correctly supported the feed-level mismatch: current.json uses pageIndicator(page.status), while incidents.json serializes only page.statusReports.

What remains missing

1. Separate indexing blocker: SPI marked 77 paths unsupported_spi_language, including:

   • apps/checker/handlers/checker.go
   • apps/checker/checker/update.go

   That prevents the checker, Tinybird persistence, and Go-to-Cloud-Tasks stages from being retrieved at all. Tracked separately in P1: Default generation needs capability-aware auto mode for mixed-language repositories #570.

2. Retrieval/ranking blocker inside indexed TypeScript: all of these were indexed by SPI but omitted from the one-call pack:

   • apps/workflows/src/checker/index.ts — public /updateStatus incident state machine
   • packages/api/src/router/statusPage.ts — statusPage.get aggregation
   • apps/status-page/src/content/status-vocab.ts
   • apps/status-page/src/app/(status-page)/[domain]/[locale]/(public)/client.tsx

   The pack instead included an unrelated dashboard monitor-update form and a changelog PNG.

Acceptance result

The exact strict prompt remains PARTIAL. SPI improves the JSON/feed side, but #565 is not ready to close: the agent still cannot produce the complete cross-layer trace from one bounded pack without raw broad fallback.

[mohanagy]

v0.32.0 auto-mode strict rerun — still PARTIAL

Capability-aware auto extraction improved the result: the one-call pack now includes apps/workflows/src/checker/index.ts, so incident creation/update is directly evidenced alongside notifications and JSON feeds.

The pack still omitted the Go checker/Tinybird/Cloud Tasks path and the statusPage.get/status-vocabulary/public-HTML path. The agent correctly returned PARTIAL because the outage-indicator conclusion still depended on unread aggregation code.

A deliberately narrow follow-up is tracked in #574: reuse the existing obligation/recovery machinery for one bounded phase-complete evidence pass, with no new mode, planner, confidence state, or repository-specific rule.

[mohanagy]

Core Reset scope amendment

Governance PR #597 merged as b65319caa55a5a838a7739e52b0e5fb586e2845d and activates the generic evidence-path implementation in #596.

For this issue's OpenStatus case:

• TypeScript/JavaScript incident mutation, notification delivery, public HTML, and JSON-feed phases remain required diagnostic evidence;
• Go checker detection and Tinybird persistence are explicitly unsupported boundaries for the current Core Reset scope;
• Madar must not claim complete mixed-language coverage;
• the supported-scope retrieval result and the later recorded Claude/Codex/human trials remain open acceptance work.

This issue stays open. #596 replaces the retrieval implementation; it does not close the real agent-usage and adoption failure recorded here.

[mohanagy]

Core Reset disposition — closed as not planned

The original v0.31.x failure remains valid historical evidence, but its complete acceptance contract is no longer part of the supported Core Reset scope.

• The replacement product supports TypeScript/JavaScript only; the OpenStatus Go checker and Tinybird phases are explicit unsupported boundaries.
• PR refactor(core-reset): replace retrieval stack with deterministic evidence paths #600 replaced the old retrieval architecture and passed the blocking JavaScript/TypeScript held-out gates.
• OpenStatus remains an honest failing, non-blocking mixed-language diagnostic; this closure does not claim that its full cross-language trace passed.
• The production paths named in this issue were retired and do not exist on core-reset.
• This issue does not block the Core Reset beta.

If polyglot or mixed-language indexing returns to scope later, it should start from a new issue and the current architecture rather than reopening the retired v0.31.x contract.