Skip to content

fix: stop recommendation engine reading Intent Routing rows as agents - #27

Open
divya0795 wants to merge 1 commit into
9thLevelSoftware:mainfrom
divya0795:fix/recommendation-engine-catalog-rows
Open

divya0795 wants to merge 1 commit into
9thLevelSoftware:mainfrom
divya0795:fix/recommendation-engine-catalog-rows

Conversation

@divya0795

Copy link
Copy Markdown

Summary

parseCatalog() in scripts/recommendation-engine.js takes any markdown table row whose first cell is kebab-case. It doesn't limit itself to the Agent Catalog section and doesn't check the File column. So the five rows of Section 2: Intent Routing in skills/agent-registry/CATALOG.md (harden, document, skip-frontend, skip-backend, security-only) get parsed as agents. For those rows the "Primary Agents" column is read as specialty and "Secondary Agents" as taskTypes, which means they score against prompts like real agents do.

Repro (on main @ 30662e3)

$ node -e 'console.log(require("./scripts/recommendation-engine.js").parseCatalog().length)'
54            # 49 agents/*.md files + 5 intent rows

$ node scripts/recommendation-engine.js "harden api security audit"
  ... engineering-security-engineer, testing-api-tester, harden, project-management-project-shepherd

$ node -e 'const r=require("./scripts/recommendation-engine.js");console.log(r.recommendAgents({prompt:"write auth"}).recommendations.map(x=>x.id))'
[ 'document', 'project-management-project-shepherd' ]     # top pick is not an agent

None of harden, document or security-only has an agents/<id>.md. Anything that loads the recommended agent's file gets a missing path.

Fix

Only accept rows whose File column is agents/<id>.md (backticks stripped). Every Agent Catalog row has that column, including the Custom Division row format that agent-creator writes. No intent row has it.

if (!/^agents\/[a-z0-9-]+\.md$/.test(cols[1].replace(/`/g, ''))) continue;

Verification (Node 20.20.2, same as CI)

  • New test in tests/recommendation-engine.test.js: parseCatalog() ids must equal the agents/*.md set, and harden must not be recommended. It fails on main and passes with the fix. parseCatalog().length is now 49.
  • All existing recommendation fixtures pass unchanged.
  • npm run validate: PASS 316 / FAIL 0. npm run release:check passed. Checksums diff is clean (scripts/ isn't in the published set). npm test: 1391 / 1391 pass.

I ran 20 prompts (the fixture prompts plus 10 extra) through the engine before and after the change. Three things change: the phantom ids are gone, real agents take some of the freed slots, and in two prompts a real agent drops out. For security audit of auth, the default output goes from 4 entries to 2 (engineering-security-engineer, project-management-project-shepherd), and write docs for the backend loses engineering-senior-developer. Both drops come from existing behavior, not this change. When the shortlist is shorter than topN, coordinator injection overwrites the last real candidate instead of appending, and the refill loop is capped at shortlist.length. You can see the same thing on main with review auth, where testing-qa-verification-specialist is dropped. Before this change the phantom harden row happened to pad the shortlist and hid it in these two cases. I left it out to keep this PR to one fix, and can send it separately if that's useful.

🤖 Generated with Claude Code

parseCatalog accepted any table row with a kebab-case first cell, so the
Intent Routing table in CATALOG.md (harden, document, skip-frontend,
skip-backend, security-only) was parsed as five extra agents and could be
recommended. Only accept rows whose File column is agents/<id>.md.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant