[Feature]: Add a table-driven natural-language graph relevance corpus

[Baziar]

Problem statement

Graph search already handles common conversational noise and several engineering
intents. What is still missing is one readable regression corpus proving that a
small set of realistic questions keeps returning the expected evidence near the
top as providers and ranking rules evolve.

Proposed solution

Current foundation

• deterministic local search with bounded output;
• natural-language stopword filtering and engineering-term normalization;
• rare-term, exact-label, alias, language, and architecture-intent weighting;
• project-subject qualifiers that stop generic architecture terms from
  outranking an exact project or runtime subject;
• deterministic score tie-breaking and broad-query diversification;
• focused tests for authentication wording, generic service noise, long
  architecture queries, language facets, and operational surfaces.

Remaining work

Create one compact table-driven fixture with at least five user questions. Keep
the expected entity IDs and maximum accepted rank next to each query so a ranking
change produces an understandable failure.

The initial questions should cover:

• authentication ownership;
• API/service ownership;
• dependency impact;
• CI/release workflow location;
• health endpoint location.

Acceptance criteria

• At least five realistic queries and expected entity IDs share one fixture.
• Every primary expected entity appears within the top three results.
• Conversational filler cannot outrank the specific engineering terms.
• Duplicate logical labels/aliases cannot consume the bounded result budget.
• Two identical runs return the same entity order and scores.
• Returned expected entities retain proof references where the fixture provides proof.
• The suite is deterministic, offline, and passes in Linux/macOS/Windows CI.
• Public graph/search contracts remain backward compatible.

Completion evidence

Publish the query table in the test failure output and document the focused local
command. If every criterion is already proven elsewhere on current main, close
#14 with links to those exact tests instead of adding duplicate fixtures.

Suggested starting points:

• packages/cli/src/workspace-knowledge-graph-query.ts
• packages/cli/src/__tests__/workspace-knowledge-graph.test.ts
• packages/cli/scripts/real-world-qualification.mjs

Alternatives considered

The following alternatives or adjacent concerns are intentionally outside this issue:

• embeddings, remote semantic search, or model-based ranking;
• general provider semantic conformance, which is tracked separately;
• claiming relevance beyond the published fixture/query set.

Expected impact

Contributors get a small objective relevance target, while developers, IDEs,
MCP clients, and agents receive more repeatable bounded evidence.

[Baziar]

This issue is ready for contribution.

If you would like to work on it, please leave a comment before starting so we can confirm the scope and help you find the relevant graph contracts and tests.

Questions and implementation proposals are welcome.

[Baziar]

The issue scope has been updated after reviewing the current implementation.

Workspai now covers stopword filtering, engineering-term normalization,
deterministic ranking, bounded retrieval, and several focused relevance cases.

The remaining work is narrowed to a table-driven corpus with five realistic
queries, top-three expectations, duplicate-budget protection, and repeated-run
determinism.

The issue remains open, but its estimated effort has been reduced from Medium
to Small.

[Tammannakhan]

I would like to work on the CI and release-workflow query family. I plan to add table-driven tests covering a direct question, a conversational paraphrase, a filler-word variation, and a concise variation. I will verify the top-three result bound, negative distractor behavior, deterministic order and scores, and retained proof references. My test environment is Windows 11.

[Baziar]

Thanks. Your scope is confirmed. Here is the exact first task:

Create a branch for issue #14.
Open packages/cli/src/tests/workspace-knowledge-graph.test.ts.
Add one table-driven test for the existing .github/workflows/ci.yml fixture. Use its pipeline entity as the expected result and add one clearly unrelated workflow entity as the negative distractor.
Put these four forms of the same intent in the table: a direct question, a conversational paraphrase, a version with filler words, and a concise version. Each row must contain the query, the expected pipeline entity ID, and maxRank: 3.
For every row, call the existing searchKnowledgeGraph function and check that the expected pipeline appears in positions 1-3 and still has its proof reference.
Run every query twice and check that both runs return the same ordered entity IDs and scores. Also check that the unrelated workflow does not replace the expected pipeline in the top three.
Do not change workspace-knowledge-graph-query.ts in this first step. If any test fails, post the query, returned IDs, ranks, and scores on issue #14 before proposing a ranking change.
From the repository root, install and run only the focused test:

corepack npm ci

corepack npm --workspace workspai test -- src/tests/workspace-knowledge-graph.test.ts

If the test passes, open a small pull request linked to issue #14. Include the Windows 11 test result and mention any check you could not run. This task is only the CI/release-workflow query family, not the complete issue.