Test official MCPs and improve MCP library

[bradflaugher]

This is a big and generic ticket, but I want you to test the built-in mcps that are found in the settings, /connectors, menu and try and and make sure they work and improve the catalog. So if there are important ones we are missing, let's add them and let's make sure that the ones that are there generally work.

---

Implementation plan

This issue is broad on purpose — without a plan it becomes "click around forever." Break it into inventory → automated smoke → manual OAuth pack → catalog fixes with a written status table as the deliverable.

Background

• Built-in directory: internal/clientconfig/builtin_remote_catalog.yaml (~275 entries), trust model in docs/MCP-CATALOG.md.
• Listings are not connections; connect still goes through per-user remote MCP / OAuth.
• Existing testing notes: docs/MCP-TESTING.md.

Phase 1 — Inventory (1–2 days)

Export a matrix of every built-in entry:

| name | auth | provenance | category | featured | can CI hit? | last verified |

Mark can CI hit? as: yes (open or api_key with fixture) / oauth-manual / tenant / dead-suspect.

Phase 2 — Automated smoke (CI-safe)

1. For open entries: optional live tools/list behind a nightly flag.
2. For api_key fixtures: table-driven test for 3–5 representative servers.
3. Fail CI on catalog shape regressions (missing setup_hint, bad provenance, featured community, etc.).
4. Add scripts/mcp-catalog-lint.sh that HEAD/GETs docs_url / detects obvious 404s (rate-limit aware).

Phase 3 — Manual OAuth pack (core product path)

Bar for "official MCPs work":

1. GitHub
2. One Google surface (Drive or Gmail)
3. Slack or Notion
4. Stripe (api_key path)

Per connector: Add from directory → OAuth/key → tools/list non-empty → one real tool call in chat → survive restart/refresh → disconnect cleanly.

File results in this issue (or docs/MCP-CATALOG-STATUS.md).

Phase 4 — Improve the library

From Phase 1–3 findings only: hide/remove dead endpoints; fill missing setup_hint / setup_url / docs_url; add high-demand missing hosted endpoints only with verified URL + auth + setup_hint; keep Featured shelf 8–20, no community featured; do not list stdio-only packages in the remote directory.

Phase 5 — UX polish (if bugs found)

Fix only regressions found while testing. Coordinate with #988 (test single-account first).

Out of scope

• Building missing stdio connectors into the bundle.
• Guaranteeing 275 live OAuth handshakes in CI.
• Continuous production health-checks of third-party endpoints.

Acceptance

• Status table committed or posted (pass / fail / skip + date).
• OAuth pack above green on a real deploy.
• Dead/broken built-ins hidden or fixed.
• At least one catalog improvement PR with clear before/after entry counts.
• docs/MCP-CATALOG.md still accurate after changes.

[bradflaugher]

Status check against current main (2026-08-14): still open. The directory is real; the verification program is not done.

Already in the repo:

• Built-in catalog of ~275 hosted entries with provenance/auth/setup_hint shape tests (docs/MCP-CATALOG.md, TestBuiltinRemoteCatalog).
• fleet mcp test [--all] [--deep] for bundled servers (handshake + optional auth-status/probe). Docs say per-user hosted connectors are not covered.
• Add-time handshake for api_key/open remote connections.

Not done vs this issue’s acceptance:

• No committed status table (docs/MCP-CATALOG-STATUS.md or equivalent) for official OAuth pack (GitHub, Google, Slack/Notion, Stripe).
• No scripts/mcp-catalog-lint.sh / nightly live tools/list.
• No Playwright/e2e that actually completes official OAuth.
• Catalog improvements from a real test pass have not been filed as a before/after PR against this issue.

Keep this as the umbrella; #1006 is the narrower OAuth-pack slice.

[KristianYe]

Status against this plan after the #1006 OAuth pack (2026-09-16)

#1006 (the manual OAuth pack) is closed; its results and the fleet fixes it produced are on main. What that covers here, phase by phase.

Phase 3 — manual OAuth pack: done, with two vendor-side blocks

Twelve hosted connectors were run against real vendor accounts (add from the directory → consent → tools list → a real tool call from chat and from a scheduled task → refresh → sign-out, plus seats and sharing). Full table: docs/MCP-CATALOG-STATUS.md; operator guidance: docs/HOSTED-CONNECTORS-RUNBOOK.md.

• PASS: GitHub, Notion, Linear, Slack (needs the app's Model Context Protocol toggle and an HTTPS callback), Azure DevOps (tenant-native Entra account), Stripe, Grafana Cloud, Uptime Robot; Plaid passed on the 09-14 build.
• Blocked by the vendor: Google Drive — the Workspace MCP servers are a Developer Preview; a personal Gmail account connects and refreshes but every tool call answers The caller does not have permission.
• Bar item 4, Stripe over the api-key path: not done; Stripe was verified through OAuth (self-registered client) only.

Fleet fixes the pack produced (all merged)

#1405, #1446, #1449, #1454, #1464, #1465, #1467, #1471, #1472, #1481, #1482, #1483, #1485, #1488 — client-secret guard, prompt roster, refused-mount marking, Google offline access, Slack's resource indicator, Entra's templated issuer, RFC 8414 path-inserted metadata (13 vendors), servers without protected-resource metadata, proxied authorization-server documents (5 vendors), required arguments in deferred mode, and the catalog corrections in #1501 (supersedes #1495/#1496).

Phase 1 — inventory: partial

All 231 official OAuth / tenant / open entries were probed with fleet's own mcpoauth.Discover and the add-time guards (appendix of the status page): 177 discover cleanly, 31 tenant entries cannot be probed without a customer value, 12 open entries answer initialize, 7 still fail. Not yet in the matrix: the 49 official api-key entries and the 8 community/third-party ones; the can CI hit?, category and featured columns.

Phase 2 — automated smoke: mostly not started

Catalog shape regressions are covered by TestBuiltinRemoteCatalog (provenance, placeholders, client_secret: required pins). Not done: the nightly live tools/list for open entries, the table-driven api-key fixture test, and scripts/mcp-catalog-lint.sh for docs_url 404s (the probe checked links once by hand: two were dead, one is fixed).

Phase 4 — library improvements: field fixes only

#1472 flagged 20 manual-client entries as secret-required; #1501 corrected seven entries (URL, auth type, docs link, manual-client flags). Entry count is unchanged at 288 (20 featured, the shelf's maximum); nothing was added or removed. Left as they are by decision: Square and Smartlead (SSE-only endpoints fleet's transport cannot speak), the Plaid/Intercom fallback (their MCP path answers 401 under every sub-path). Open findings recorded on the status page: F8 (broker masks vendor 4xx argument errors), F10 (connect failures announced as "needs re-authorization"), F12 (Bugsnag's plain-http metadata pointer), F13 (saved connections keep their pre-correction URL until re-added).

Acceptance checklist

• Status table committed (docs on main) and posted (Test oauth flow with official MCPs #1006)
• [~] OAuth pack green on a real deploy — green on a local rig against real vendors; not repeated on a production deploy; Google blocked by the vendor
• [~] Dead/broken built-ins hidden or fixed — seven fixed; Square/Smartlead/Plaid/Intercom left by decision; Bugsnag and four 403-from-our-network vendors unresolved
• Catalog improvement PR with before/after entry counts — corrections only, count unchanged
• [~] docs/MCP-CATALOG.md still accurate — spot-checked for the changed fields and entries; not re-read end to end

What's left, in order of value

1. scripts/mcp-catalog-lint.sh (docs/setup URL 404s) and the nightly open-entry smoke — CI-safe, named in the plan.
2. Inventory rows for the 57 api-key / community entries and the missing columns.
3. Stripe over the api-key path.
4. F8 / F10 / F12 / F13 if wanted.
5. Any hosted endpoints worth adding, and whether to hide the two SSE-only entries.

[KristianYe]

Phase 1 — inventory: done (PR #1515)

The matrix the plan asks for is in #1515: the appendix of docs/MCP-CATALOG-STATUS.md becomes one row per built-in entry — all 288 — with name | auth | provenance | category | featured | can CI hit? | last verified, plus the probe verdict and notes carried over from the 2026-09-14 audit so nothing from #1501 is lost.

How the 57 missing rows were filled (2026-09-16). The 09-14 discovery probe could only reach the 231 official OAuth/tenant/open entries. The rest — 51 api_key, Composio/Make/Smithery/Zapier, two self-hosted templates — got three checks each: an MCP initialize with no key; the same with an obviously invalid key attached exactly the way fleet attaches a real one (Authorization: Bearer, the entry's api_key_header, or its api_key_query); and fleet's own add-time validation (initialize + tools/list over the SSRF-safe client) replayed with that key. Third-party OAuth entries went through mcpoauth.Discover; every docs_url was fetched.

can CI hit? — one value added to the plan's four, because "yes (api_key with fixture)" is true of nothing today:

class	entries	meaning
yes	12	open endpoint, answers initialize with no credentials
key-fixture	53	needs a vendor key as a CI secret (all api_key entries + 3 open entries with the key in the URL); no fixture exists yet
oauth-manual	182	browser consent cannot run in CI
tenant	40	{placeholder} only a customer can fill
dead-suspect	1	Synter Ads — its URL serves an HTML page

Last verified: live 12 · probe 2026-09-14 187 · probe 2026-09-16 55 · not probeable 34 (31 tenant hostnames, 2 self-hosted templates, Render's DNS on the audit box). Auth: oauth 182 · tenant 40 · api_key 51 · open 15. Provenance: official 280 · third_party 5 · community 3. Featured 20.

New findings (rows F14, F15, C10, C11, V4 in the audit table):

• F14 — 25 of the 51 api_key vendors accept an invalid key through fleet's add-time check; they verify the key only at the first tool call, so a mistyped key is saved with a "connected, N tools" confirmation and fails in the first turn. The other 26 reject at the handshake as documented. docs/MCP-CATALOG.md gets a one-sentence caveat in Inventory every built-in remote MCP entry for #986 Phase 1 #1515; the fix (one cheap read-only call after tools/list where a vendor documents one, or honest card copy) is Phase 2 material.
• F15 — 19 api_key entries also publish OAuth protected-resource metadata (Braintrust, Brevo, Buffer, Censys, Coda, fal.ai, Fireflies, Instantly, Kong Konnect, Mollie, Paddle, PagerDuty, Parallel, Raygun, Razorpay, Tavily, Upsun, Vultr, Whop) — candidates for one-click auth: oauth after a live add each.
• C10 — Composio's documented URL 307-redirects to …/{SERVER_ID}/mcp?user_id=…; fleet refuses redirects. Verify with a real server id before touching the catalog.
• C11 — Synter Ads (community, hidden by default) is the only dead-suspect entry.
• V4 — Coda and Leonardo.Ai docs_url answer 403 / 500 to a plain GET (bot walls, most likely); ZoomInfo's 404 was already known.

Also confirmed: all 51 api_key endpoints are alive and speak MCP; Zapier and Make discover cleanly (Make: no none auth method, no scopes — #1488's confidential retry covers it); no api_key entry lacks a setup_hint or setup_url.

Acceptance so far

• Status table committed or posted — docs/MCP-CATALOG-STATUS.md (Board tags and tag filter, seven catalog corrections, and the #1006 OAuth pack record (supersedes #1495, #1496, #1499) #1501) + the inventory (Inventory every built-in remote MCP entry for #986 Phase 1 #1515)
• OAuth pack green on a real deploy — 8 PASS, Google vendor-blocked, Stripe api-key path not exercised
• Dead/broken built-ins hidden or fixed — 1 dead-suspect (C11) + C10 to verify
• At least one catalog improvement PR with before/after counts — Board tags and tag filter, seven catalog corrections, and the #1006 OAuth pack record (supersedes #1495, #1496, #1499) #1501 (288 → 288, field fixes)
• docs/MCP-CATALOG.md still accurate — F14 caveat in Inventory every built-in remote MCP entry for #986 Phase 1 #1515

Next (Phase 2, in the plan's order): scripts/mcp-catalog-lint.sh (docs_url check + catalog shape) → nightly tools/list smoke for the 12 yes entries → the F14 add-time gap → api_key fixture test for 3–5 vendors once keys exist.