Discovery Engine — Phase 1: detector core (contract + registry + emitters) + sitemap/robots

[badchars]

Part of the Discovery Engine epic #125.

Scope — foundation + first (safe) detectors

Deterministic detector framework, so every later detector plugs in without touching the core.

• Detector contract (discovery/detector.ts): the Detector interface (name, kind, applies, detect) + DiscoveryContext (page/browser handle, base URL, scope matcher) + DiscoveryResult (pages[], endpoints[], confidence).
• Registry (discovery/registry.ts): ordered list; run applies()-passing detectors; collect results.
• Emitters (discovery/emit.ts): route pages → BFS crawl queue (dedup/scope via existing enqueue), endpoints → ingest (reuse the worker's ingest POST).
• Detectors (simple, browser-fetch, auth+proxy-aware):
  • sitemap — /sitemap.xml, /sitemap_index.xml (+ nested) → pages.
  • robots — /robots.txt Allow/Disallow paths → pages (in-scope only).

Constraints

• Runs inside the hackbrowser worker (uses the authenticated, proxy-configured browser context for fetches).
• Additive: seeds the existing BFS queue; no change to the crawl loop's behavior.
• Each detector independently unit-testable.

Commits (granular)

Core contract → registry → emitters (pages) → emitters (endpoints) → sitemap detector → nested-sitemap handling → robots detector → scope/dedup wiring → tests. One behavior per commit.