Summary
On a hash-routed SPA whose navigation is button/router-driven (no <a href> links), the crawl discovers only a handful of routes and completes early. Observed on a modern SPA (Angular-style hash routing, /#/..., button navigation): ~4 pages explored despite the app having many routes/functions.
What works (not the problem)
- Hash routing is handled:
normalizeUrl (state.ts) keeps #/route / #!/route / fragment view-state as distinct pages, so hash routes are not collapsed.
- Element scanning is generic: scanner.ts collects buttons, ARIA roles, framework click-attributes, web-component buttons,
cursor:pointer elements, etc. — not a hardcoded element list. The planner can click any framework's interactive elements.
- Auth (SSO) and crawl lifetime are fine.
Root cause
collectDOMLinks (agent.ts) — the bulk link-harvest that seeds the BFS queue — reads only a[href]:
const hrefs = await page.$$eval("a[href]", ...)On a button/router-nav SPA this yields ~0 links, so route discovery falls entirely on the planner clicking nav elements per page. The planner explores per-page with a step cap and its own prioritization; it does not systematically click every nav/menu item, so most routes are never enqueued.
Proposed improvement (generic, not per-app)
The element scanner is already generic; the gap is specifically in link/route harvesting. Extend it with generic SPA-nav signals:
- Harvest declarative nav targets — in addition to
a[href], query elements carrying a destination attribute and resolve/enqueue it (relative to base + hash):
[routerLink], [ng-reflect-router-link] (Angular)[data-href], [data-route], [data-url], [data-to]href on non-anchor elements, [role=link]
- Harvest nav-landmark controls — collect interactive elements inside
nav, [role=navigation], [role=menu]/[role=menuitem], sidebars, and mark them as high-priority nav targets for the planner. This covers imperative nav (a <button> that calls router.navigate() with no declarative target) — which can only be discovered by clicking.
- Leverage the existing url-change-on-click signal — the crawl already detects
page.url() !== urlBefore after a click; use it to treat any element that changes the URL as a route source, and feed newly seen routes back into the queue.
Declarative nav can be harvested statically (1); imperative nav requires clicking (2 + 3). Both are generic — they encode "how SPAs navigate", not any specific app.
Impact
Modern enterprise apps are overwhelmingly button/router-nav SPAs, so this directly determines crawl depth (and therefore how many endpoints the proxy-agents get to test) on real targets.
Summary
On a hash-routed SPA whose navigation is button/router-driven (no
<a href>links), the crawl discovers only a handful of routes and completes early. Observed on a modern SPA (Angular-style hash routing,/#/..., button navigation): ~4 pages explored despite the app having many routes/functions.What works (not the problem)
normalizeUrl(state.ts) keeps#/route/#!/route/ fragment view-state as distinct pages, so hash routes are not collapsed.cursor:pointerelements, etc. — not a hardcoded element list. The planner can click any framework's interactive elements.Root cause
collectDOMLinks(agent.ts) — the bulk link-harvest that seeds the BFS queue — reads onlya[href]:On a button/router-nav SPA this yields ~0 links, so route discovery falls entirely on the planner clicking nav elements per page. The planner explores per-page with a step cap and its own prioritization; it does not systematically click every nav/menu item, so most routes are never enqueued.
Proposed improvement (generic, not per-app)
The element scanner is already generic; the gap is specifically in link/route harvesting. Extend it with generic SPA-nav signals:
a[href], query elements carrying a destination attribute and resolve/enqueue it (relative to base + hash):[routerLink],[ng-reflect-router-link](Angular)[data-href],[data-route],[data-url],[data-to]hrefon non-anchor elements,[role=link]nav,[role=navigation],[role=menu]/[role=menuitem], sidebars, and mark them as high-priority nav targets for the planner. This covers imperative nav (a<button>that callsrouter.navigate()with no declarative target) — which can only be discovered by clicking.page.url() !== urlBeforeafter a click; use it to treat any element that changes the URL as a route source, and feed newly seen routes back into the queue.Declarative nav can be harvested statically (1); imperative nav requires clicking (2 + 3). Both are generic — they encode "how SPAs navigate", not any specific app.
Impact
Modern enterprise apps are overwhelmingly button/router-nav SPAs, so this directly determines crawl depth (and therefore how many endpoints the proxy-agents get to test) on real targets.