Repository navigation
chore(sync): rebase dev onto upstream main - #53
Closed
aryasaatvik wants to merge 145 commits into
Closed
aryasaatvik wants to merge 145 commits into
aryasaatvik wants to merge 145 commits into
Conversation
…sefulSoftwareCo#1007) Two flashes visible in the onboarding flow: 1. create-org flickered 'Create your organization' → 'Loading' → back. The header was gated on the pending-invitations atom being initial OR waiting, and that atom is keyed on auth — so the create-org submit (which invalidates the auth keys) put it into 'waiting' and flipped the whole page to 'Loading'. Almost no new user has a pending invitation, so we now show NO loading placeholder at all: the create form renders immediately and stays put, and invitations (when they exist) just appear above it once the fetch resolves. Removed the InvitationsSkeleton entirely. 2. The connect-MCP step flashed the app sidebar. 'Continue to app' / 'Skip' navigated to the BARE /{-$orgSlug}, which mounts the shell at / and then OrgSlugGate fires a SECOND navigation to canonicalize / → /<slug>. That double hop is the window where the shell paints over the still-mounted onboarding page. Now it navigates straight to /<slug> (the page already has the org slug), so the shell mounts once. Both pre-existing; the new auth-routing flow test (UsefulSoftwareCo#1006) made them visible.
UsefulSoftwareCo#1006) The org-slug column was nullable and the runtime self-healed it: every resolveOrganization (and the SSR gate) lazily minted a slug for any row that lacked one. That left a permanent nullable state and a heal-on-read cost. Move slug generation to the single moment a row is inserted — upsertOrganization mints the slug at INSERT (ON CONFLICT DO NOTHING absorbs the id/slug races, retrying with a fresh candidate) and keeps the existing slug on conflict. The three insert sites (createOrganization, acceptInvitation, the WorkOS-fallback upsert in resolveOrganization) all get a slug for free; resolveOrganization and the SSR gate's organizationDisplay no longer heal. The schema column is now NOT NULL (migration 0004), backfilled once for legacy rows. Deploy order matters and is documented in the backfill script: backfill → deploy this code → apply 0004. The old code inserts a NULL slug then heals, so NOT NULL cannot land while it still serves. Also adds e2e/cloud/auth-routing-flow.test.ts: one browser session walking the whole arc — signed out → /login → hosted-AuthKit sign-in → org-less onboarding → create org → canonical /<slug> dashboard — through the real dev-server SSR gate. The hops are covered piecewise elsewhere; this is the narrative regression guard and the only test pinning the bare-/ → /<slug>/ dashboard canonicalization a fresh login lands on.
…2e + framework dwells (UsefulSoftwareCo#1002) * Self-host: serve the org-scoped MCP path the Connect card prints The console Connect-an-agent card prints `<origin>/<organizationId>/mcp` — a convention the multi-tenant cloud worker routes by stripping the org segment at the edge. Self-host only served the bare `/mcp`, so a connecting MCP client completed OAuth and then 404'd on reconnect. Self-host is single-tenant (the session pins the one org), so the segment carries no routing meaning: both front-ends (the prod Bun server in serve.ts and the vite dev middleware) now strip a single leading segment from `/<seg>/mcp` and the matching OAuth discovery path, mirroring cloud's edge rewrite but accepting any segment. Coverage gap this slipped through: every MCP e2e built `${baseUrl}/mcp` itself and connected to that, so none exercised the URL the card actually hands the user. New scenario selfhost/mcp-connect-card-url scrapes the URL out of the rendered card and asserts the server serves it (was 404, now 401) — it fails on the pre-fix build. Also adds selfhost/mcp-oauth-consent, which records the MCP OAuth browser flow and documents that it currently auto-consents (no approval screen). Shared pure helper (stripMcpOrgSegment) with unit tests; verified green on both the selfhost (vite) and selfhost-docker (production image) targets. * Self-host: require an MCP OAuth approval screen Connecting MCP clients were auto-consented — a signed-in user's client got a token with no prompt. Better Auth's MCP authorize only shows consent when the request carries prompt=consent (clients don't send it) AND a consentPage is configured. So: configure oidcConfig.consentPage -> the SPA route /mcp-consent, and force the screen by injecting prompt=consent on every /api/auth/mcp/authorize in the shared Better Auth handler (force-mcp-consent, unit-tested). The handler also looks the registered client_name up from the DCR row and appends it to the consent redirect so the screen reads "Connect <name>?" rather than an opaque id (self-declared — a label, not a trust signal). The /mcp-consent page (rebuilt from the earlier test/selfhost-e2e-org-setup work) shows the client, scopes, an explicit note that the grant is limited to the MCP server (not a web-app login, no other API calls), and Allow/Deny posting to /api/auth/oauth2/consent. Also fixes a dev-only routing bug the new route exposed: the vite middleware matched MCP paths with rawUrl.startsWith("/mcp"), which swallowed /mcp-consent (and its source module) and misrouted them to the API handler. Now matches on the exact pathname. The mcp-oauth-consent e2e asserts the approval screen appears with the client name, Allow grants a code, and gates the connection. Green on selfhost (vite) and selfhost-docker (prod image); unit + lint + typecheck clean. * e2e: real OpenCode binary through self-host MCP OAuth consent, spliced Drives the actual opencode binary through the full MCP OAuth handshake against self-host (its own DCR/PKCE/scopes/token store), with the browser hop driven by a real Playwright session that lands on the forced /mcp-consent approval screen and clicks Allow — the consent redirect goes to OpenCode's own localhost callback, so OpenCode receives the code and finishes the grant for real (asserts a stored access token + "connected"). The agent PTY (terminal.cast) and the browser (session.mp4) run concurrently via Effect.all and mark focus terminal→browser→terminal, so scripts/film.ts cuts them into one film.mp4: the agent asks → we approve in the browser → the agent is connected. Green on selfhost (vite) and selfhost-docker (prod image); skips where the opencode binary isn't installed. This is the real-client + spliced-recording coverage that previously lived only on the unmerged test/selfhost-e2e-org-setup branch (record-flow-oauth.ts). * e2e: framework-managed human dwells, gated by E2E_FILM A scenario author shouldn't hand-code waitForTimeout to make a recording readable — pacing is the recording's concern, not the scenario's. A dwell is a property of a focus transition: tabbing between tools, a developer lingers a beat to take in where they landed. So the framework owns it: - timeline.ts gains beat()/enterFocus() + isFilming(). beat() is a no-op unless the run is being filmed (E2E_FILM=1, also implied by the desk's E2E_DESK). - The browser step() enters focus (lingering on the outgoing window) and beats at the end so each named step's result holds on screen. - The Cli surface holds the terminal's final frame at end of drive. So fast/CI runs pay nothing, and a filmed run paces itself into a watchable 'moving between apps' splice. The OpenCode consent scenario drops all its hand-rolled dwells and just names two browser steps + enterFocus at the terminal transitions — verified: 2.9s film unfilmed, 7.2s paced under E2E_FILM=1, green both ways. * docs(e2e): add the OpenCode self-host MCP OAuth consent recording The paced film (E2E_FILM=1) of selfhost/mcp-opencode-consent — the real opencode binary asking to connect, the browser approval screen, and connected — embedded in the PR so reviewers see the flow, not just read about it. * Self-host: drive mcp-oauth end-to-end test through the consent screen The forced MCP approval screen makes /authorize redirect to /mcp-consent with a consent_code instead of straight to the client callback. Update the opaque-bearer end-to-end test to approve on the consent screen (POST /api/auth/oauth2/consent) and follow the returned redirectURI to the code, matching what a real client's browser does.
… loss) (UsefulSoftwareCo#1008) Better Auth opened its own LibsqlDialect connection (@libsql/core 0.8) to the same file as executor-core's @libsql/client (0.17). Opening second, it unlinked executor-core's -wal/-shm and created new ones, orphaning executor-core onto a deleted WAL inode. Every executor-core write (integrations, connections, tools) then vanished on the next restart while Better Auth's own tables survived — the 'reconnected account, zero tools' data-loss bug. The earlier 'single Better Auth instance' change removed only the throwaway bootstrap connection; the long-lived one unlinked the WAL just the same. Build Better Auth's LibsqlDialect on SelfHostDb's existing client ({ client }, closeClient=false) so there is one connection and one WAL. Also resolves the documented cross-connection read lag.
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…fulSoftwareCo#1010) * Drop the vendor/ fork submodules; develop emulate as a standalone repo The emulate and mcporter forks are consumed only as published npm packages (@executor-js/emulate, @executor-js/mcporter) — nothing imports vendor/ at runtime, and the submodules created persistent friction: empty in every worktree, a pnpm/node>=24 toolchain unlike executor's bun, stale-dist rebuilds, and publish/pointer drift. Remove vendor/emulate and vendor/mcporter and point the guidance at the standalone fork repos: agents develop emulate on its own main with full publish/deploy autonomy, and this repo just bumps the npm dependency. - Remove vendor/emulate, vendor/mcporter, vendor/README.md; .gitmodules keeps only integrationsdotsh - bootstrap.ts: drop the dead --forks submodule-init path - Rewrite the emulate skill (new "Changing or deploying an emulator" section), AGENTS.md, RUNNING.md, README.md, and the e2e mcp surface comment * Drop the integrationsdotsh submodule too — executor tracks zero submodules now integrationsdotsh is a downstream product that consumes executor's published SDK, not something executor imports. There was no gitlink — just a phantom .gitmodules entry plus a few "ignore this dir" guards. Develop it in its own repo (github.com/RhysSullivan/integrationsdotsh) like the forks. - Remove the integrationsdotsh stanza from .gitmodules (its last entry, so the file is deleted — executor has no submodules now) - Drop the now-stale integrationsdotsh ignores in .oxlintrc.jsonc, .oxfmtrc.json, and scripts/migrate-schema-class.ts
… flash) (UsefulSoftwareCo#1011) A multi-org user loading /<orgB> while their session cookie still points at orgA saw orgA's slug flash into the copyable `npx add-mcp .../<slug>/mcp` box before /account/me resolved. On first paint `auth.organization` comes from the SSR auth-hint, which is scoped to the COOKIE's org — but the page is scoped to the URL's org. Seed OrganizationProvider's slug (which feeds the connect card via useOrganizationSlug) from the URL's `{-$orgSlug}` param, falling back to the session slug on a bare URL (which OrgSlugGate canonicalizes onto it anyway). The URL slug is the actual request scope and is correct on the very first paint, so the wrong slug never appears. OrgSlugGate keeps using the session slug for canonicalization; only the install-URL context changes. useOrganizationSlug is consumed solely by the connect card, so there's no other blast radius.
…fulSoftwareCo#1014) * host-mcp: extract shared browser-approval primitives The browser-approval wire shape (elicitation-mode query parsing, the /resume approval URL, the resume-payload schema, and the decision acknowledgement) was copy-pasted across the in-process handler (apps/local) and the Durable Object base (@executor-js/cloudflare). Hoist it into @executor-js/host-mcp/browser-approval and refactor the existing copies onto it. Each host keeps only its transport envelope (HTTP JSON vs DO RPC result), which is the part that legitimately differs. Behaviour-preserving; no wire changes. * e2e: browser-approval scenarios on cloud + browser-mode MCP surface Adds the first end-to-end coverage of a human approving/declining a gated MCP action in the rendered console approval page — the leg unit tests structurally cannot reach. - MCP surface: a session can run in elicitation_mode=browser, exposes awaitResume (the no-action long-poll resume), and parseBrowserApproval pulls the {executionId, approvalUrl} out of a paused result. - Surface fix: give each MCP session a unique mcporter server name. mcporter caches OAuth tokens per server name, so the old constant name let a later session reuse an earlier identity's token (wrong org). A decline scenario was the first identity-sensitive flow to expose it. - cloud/browser-approval.test.ts: approve runs the gated tool to completion; decline blocks it. Moves to scenarios/ once self-host and Cloudflare gain the feature. * e2e(selfhost): drive the forced MCP OAuth consent screen The self-host serving layer forces prompt=consent on every MCP authorize, so the headless e2e target completes the /mcp-consent approval the way the page does (sign in, authorize, POST /api/auth/oauth2/consent) instead of the old direct-code cookieConsentStrategy.
Self-host (and the local app) could pause a gated MCP execution but had
no way for a human to approve it in the browser: the in-process MCP store
served model-mode only, and the shared resume page fetched paused detail
from the session-less /api/executions/:id — a different engine that never
holds the MCP pause.
- @executor-js/host-mcp: an in-process browser-approval store (the resume
long-poll <-> HTTP decision bridge, keyed by executionId), and the
in-memory session store now builds browser-mode servers, keeps each
session's engine, and serves the session-scoped paused (GET) + resume
(POST) endpoints. makeMcpBuildServer returns { mcpServer, engine }.
- host-selfhost: mounts the approval endpoints at /api/mcp-sessions/*,
gated by a Better Auth session.
- react: the shared resume page fetches paused detail session-scoped, so
the in-process hosts resolve it from the right engine (mirrors cloud).
- apps/local: migrated onto the shared approval store + the new GET
handler, dropping its duplicated waiter maps; its browser approval page
now works, not just the API-direct path.
- e2e: browser-approval scenarios moved to scenarios/ (cross-target) —
approve + decline now green on self-host too.
…fulSoftwareCo#1016) The host-cloudflare Durable Object base already owned the approval store and the paused/resume RPCs, but its buildMcpServer ignored the session's elicitation mode (model-mode only) and the worker served no approval endpoints — so a gated MCP action could never be approved in the browser. - host-cloudflare: buildMcpServer wires browser elicitation mode + approvalUrl (origin from the create request, captured by the DO base) + the base's browserApprovalStore, mirroring cloud. The worker mounts an Access-gated /api/mcp-sessions/* handler that routes the paused (GET) + resume (POST) calls to the owning session's DO RPCs. - e2e: a new 'cloudflare' target — the real worker on workerd via wrangler dev (Miniflare) with local D1 + R2 and dev-auth, the console SPA built and served as static assets. The cross-target browser-approval scenario (approve + decline) now runs green here too, driving the rendered resume page against the worker. Browser approval is now covered end-to-end on all three deployment shapes: cloud, self-host, and Cloudflare self-host.
* harden local auth + local/desktop MCP-approval e2e (squashed for stack rebase) # Conflicts: # apps/local/src/mcp.ts # apps/local/src/serve.test.ts # apps/local/src/serve.ts # e2e/src/surfaces/browser.ts # packages/react/src/api/client.tsx # packages/react/src/routes/resume.$executionId.tsx * e2e(desktop): MCP browser-approval happy path + adversarial bearer-scoping test * format e2e test files * local: fix CLI open typecheck + drop unwired approval_token - apps/cli open: reference ChildProcess error handler structurally (bun-types' ChildProcess exposes unref() but not the inherited EventEmitter on()). - Drop the approval_token query wiring from the MCP resume page + test: it was never minted or validated server-side. The /api/mcp-sessions/* endpoints are bearer-gated, which is the real guard for the resume POST.
Match the self-host open path: a per-connection PRAGMA busy_timeout=5000 gives writers a 5s retry window instead of an instant SQLITE_BUSY when a second OS process transiently holds the write lock (a CLI tool, the v1->v2 migration reader, or a launchd restart racing the old pid).
…UsefulSoftwareCo#1005) The Electron app becomes a thin client of the OS-supervised gateway: - On packaged macOS launch it attaches to a running supervised daemon, or (with a one-time consent prompt) registers one by pointing a launchd LaunchAgent at the bundled sidecar binary in supervised mode. Falls back to the existing managed sidecar everywhere else. - Quitting the app, restarting the window, updating, or resetting state no longer stops a supervised daemon -- it keeps serving MCP. The old 'another server owns the data dir -> fatal dialog' path becomes a clean attach. - The sidecar entry, when supervised, self-writes the discovery manifest and reads its password from service.key. - A crash monitor shows a reconnecting overlay while launchd restarts the daemon; regenerating the password reinstalls + re-points the window.
…sefulSoftwareCo#1013) * e2e: VM substrate for cross-OS supervised-daemon targets + build --target Foundation for codifying the supervised-daemon reboot-survival proof as e2e tests (today it's only a by-hand bash harness). Adds a VM substrate that can provision a guest, drive it over SSH, REBOOT the OS for real, and tear it down: - e2e/src/vm/{types,tart,build-binary}.ts — the tart provider (macOS + Linux guests on an Apple-Silicon host) with a reconnecting SSH forward that survives a guest reboot, plus a helper to compile the guest's `executor` binary. - apps/cli/src/build.ts: a first-class `--target <name>` flag so the harness can build a specific platform's binary (replaces a throwaway env hack). Validated standalone: provision macOS guest → ssh → real reboot (boottime changes) → clean discard. Next: the `cli` target wires this into the scenario suite so restart-persistence runs against a real reboot. * e2e: cli target — supervised daemon in a VM, restart() is a real reboot Wires the VM substrate into the scenario suite as a `cli` target (one project per guest OS: cli-macos / cli-linux / cli-windows). The globalsetup builds the guest's executor binary, provisions a VM, `executor service install`s the supervised daemon, and forwards its loopback port over a reconnecting SSH tunnel — so target.baseUrl works unchanged for the api surface. restart() reboots the guest OS for REAL and waits for the daemon to auto-start, so the existing restart-persistence scenario now proves the boot-time auto-start path (launchd RunAtLoad / systemd linger / Task Scheduler AtStartup), not a process restart. Because globalsetup (main process) and restart() (test worker) are different processes, restart() re-derives the guest address from env and reboots statelessly, mirroring how selfhost-docker's restart re-derives container ops. Verified: `vitest run --project cli-macos` boots a macOS guest, installs the daemon, addSpec's an integration, reboots the guest, and asserts the data survived — green, with a run-record in the matrix. cli-linux reuses the tart provider; cli-windows awaits the ec2 provider. * e2e: recorded film — integration survives a real reboot (terminal cast) A watchable companion to restart-persistence: drives the same register → reboot → survived flow through the chat theater so the on-screen tool spinner runs for the ACTUAL guest reboot. Produces a terminal.cast the viewer plays in the matrix — press-play evidence instead of trusting a green check. Runs against the cli-* VM targets; verified on cli-macos (357-event cast). * e2e: desktop attach film — app connects to the supervised daemon (not a sidecar) Films the desktop ATTACHING to an already-running OS-supervised daemon instead of spawning its own sidecar: starts a real cli-daemon (the desktop sidecar server in EXECUTOR_SUPERVISED mode → kind "cli-daemon") against a throwaway HOME, launches Electron pointed at the same HOME, and proves the attach — the server manifest still names OUR daemon's pid (a spawned sidecar would be a new pid + kind "desktop-sidecar"). session.mp4 is the artifact. The daemon half is verified (EXECUTOR_READY + kind cli-daemon + 401). Like the existing desktop scenario it drives Electron via Playwright, so it needs a GUI display session to run (the desktop project is already display-gated, not in the default test chain) — run on a real desktop / CI-with-display to capture the video. * e2e/cli: bearer-gated daemon target, reboot down-gate, EC2 provider for Windows - cli target authenticates with the daemon's auth.json bearer (was Basic from service.key); globalsetup reads the token from the guest and publishes it. - restart() gates on the daemon going DOWN before polling it back up, so an orderly shutdown / reconnecting tunnel can't false-pass a reboot that never happened (waitForHttpDown). - EC2 VmProvider (e2e/src/vm/ec2.ts) + cli-windows globalsetup unblock Windows: launch, key-based SSH/PowerShell drive, real boot-time-gated reboot, terminate. * e2e/desktop + cli tests: bearer auth token, not a password The supervised-attach film passed EXECUTOR_AUTH_PASSWORD, a dead env the sidecar no longer reads (it reads EXECUTOR_AUTH_TOKEN and otherwise mints/loads the bearer from auth.json). Pin a real bearer instead. Fix two stale launchd/wrapper comments that still pointed the secret at service.key.
…ised daemon (UsefulSoftwareCo#1017) The supervised-daemon attach path (ensureSupervisedConnection → attachToSupervisedDaemon) and the bundled compiled sidecar only run when app.isPackaged is true. Dev electron skips that branch and always spawns its own desktop-sidecar, so the attach behaviour can't be proven against a dev launch — the prior dev-electron attach test could never satisfy its own assertion. Test the production artifact instead. - electron-builder.e2e.config.ts: unsigned bundle (no notarize/hardenedRuntime, `dir` target) so the real package builds without Apple credentials; the release config is untouched. - desktop-packaged vitest project + globalsetup: builds web UI → compiled sidecar → electron-vite → electron-builder, then publishes the launch exe and bundled sidecar path. - desktop-packaged/supervised-attach.test.ts: starts the bundle's own compiled executor-sidecar as the supervised daemon, launches the packaged app, and asserts it attached (manifest still names the daemon's pid/kind, console renders through the bearer-gated daemon) rather than spawning a sidecar. Skips honestly where no GUI display is reachable instead of hanging on launch. - Remove the superseded dev-electron attach test; fix a stale "Basic-auth" comment (the header is bearer).
…fix packaged crash) (UsefulSoftwareCo#1018) The Electron main process imported `loadOrMintLocalAuthToken` / `rotateLocalAuthToken` from the `@executor-js/local` barrel, which also exports the full local server (`startServer`). That dragged the entire server module graph — including `ws` — into the main bundle. electron-vite's externalizeDepsPlugin then externalized `ws`, but `ws` is not a declared desktop dependency, so electron-builder never bundled it. The result: the PACKAGED app crashed on startup with `ERR_MODULE_NOT_FOUND: Cannot find package 'ws'`. Dev electron hid this — it resolves `ws` from the monorepo node_modules. The auth helpers in apps/local/src/auth.ts are pure `node:fs`/`node:crypto` with no server or Effect imports. Expose them via a dedicated `@executor-js/local/auth` subpath and import from it in the desktop main; the sidecar process (src/sidecar/server.ts) keeps the full barrel since it runs the server. `ws` references in the built main bundle drop from 547 to 0. Proven end-to-end: the packaged Linux bundle, launched headless under Xvfb in a VM, now boots and logs "attaching to supervised daemon …" and the manifest stays kind=cli-daemon with the daemon's pid (attached, not a spawned sidecar) — e2e/scripts/verify-linux-desktop-attach.ts.
UsefulSoftwareCo#1022) * Fix copy buttons on non-secure origins; move API keys into the sidebar The copy buttons (API key, MCP install command, tool IDs, …) called navigator.clipboard.writeText with no fallback. navigator.clipboard only exists in a secure context (HTTPS or localhost), so on a self-hosted console served over plain HTTP on a LAN host/IP the property is undefined and the click threw and was swallowed — copy silently did nothing. Add a shared copyToClipboard helper (@executor-js/react lib/clipboard) that tries the async Clipboard API and falls back to selecting a hidden node via a document Range + execCommand("copy"). The Range path does NOT move focus, so it still works inside a focus trap such as the dialog that shows a freshly created API key (focusing a throwaway <textarea> there gets yanked straight back, copying nothing). Route CopyButton, the code blocks, and the app update card through it. Move the API keys link out of the account dropdown and into the main sidebar nav for the hosts that serve keys in-app (self-host + cloud). Cloudflare manages API keys in Access, so it keeps the default nav without the item. Add selfhost browser e2e guards for both: API keys is a first-class sidebar item, and the copy button copies a new key on a simulated plain-HTTP origin. * Toast on copy failure instead of failing silently When a copy genuinely can't happen (no navigator.clipboard AND execCommand refuses), copyToClipboard returns false. Show a "Failed to copy to clipboard" error toast on that path across every copy site (CopyButton, the code blocks, the app update card) rather than doing nothing. Add a selfhost e2e guard that forces both copy paths to fail and asserts the toast surfaces.
… detect, actionable errors, pinned outbound URLs) (UsefulSoftwareCo#1021) * Self-host: auto-detect public URL from platform env + actionable origin error Self-host on a PaaS failed sign-in with Better Auth's bare 'Invalid origin': config.webBaseUrl defaulted to http://localhost:PORT when EXECUTOR_WEB_BASE_URL was unset, so trustedOrigins never matched the real public origin. - config.ts resolves webBaseUrl from the platform's injected origin when EXECUTOR_WEB_BASE_URL is unset (Railway/Render/Fly/Vercel/Netlify/Heroku/ Azure/Cloudflare Pages; ordering from @t3-oss/env-core's preset, minus the CRA-ambiguous PUBLIC_URL), else localhost with a one-time production warning. Pinned, never derived from the request Host (matches Windmill/n8n; keeps host-header injection out of OAuth/links). - auth/invalid-origin-help.ts rewrites the 403 'Invalid origin' into a message naming the exact EXECUTOR_WEB_BASE_URL to set, keeping code:INVALID_ORIGIN. Verified end-to-end against the prod image: RAILWAY_PUBLIC_DOMAIN set -> sign-up 200; unset -> the actionable setup message. * Cross-host: shared public-origin resolver + consistency fixes Audit of every web-URL site (202 inventoried) found no host-header-injection holes, but origin resolution was split across hosts and had a few gaps. Extract one shared resolver and apply it consistently: - packages/core/sdk/src/public-origin.ts (new, @executor-js/sdk/public-origin): getPlatformOrigin (the t3-env-preset platform-detect chain) + resolvePublicOrigin (explicit env -> platform -> undefined) + the missing-origin warning helpers. selfhost and cloudflare configs now both use it instead of diverging. - selfhost: warn on the localhost fallback unless NODE_ENV is development/test (was gated on ===production, so staging/demo stayed silent). - cloudflare: pin VITE_PUBLIC_SITE_URL via the resolver + warn once on a real deploy when it's unset (mirrors selfhost; gated on enableDevAuth). - cloudflare MCP session DO: replace the silent 'http://localhost' approval-URL fallback with a logged, visibly-invalid origin so a misconfigured deploy fails loudly instead of handing clients an unreachable localhost link. - cloud: WorkOS Admin Portal returnUrl fell back to the bare relative '/org'; use the pinned origin like every other VITE_PUBLIC_SITE_URL site. - MCP in-process session store: thread the pinned webBaseUrl so the self-host browser-approval link uses the public URL, not the internal 127.0.0.1 bind behind a reverse proxy (local/desktop keep request.url — loopback is correct). All outbound URLs stay pinned (never the request Host); request origin remains for CSRF/display only. Typecheck + lint clean; selfhost 48, host-mcp 32, cloud 128, host-cloudflare 10, sdk public-origin 4 tests green.
…Co#1024) The card title sat only 12px (pt-3) below the card's top border, leaving it visually jammed against the edge. Bump the header's top padding to pt-4 (16px) for breathing room while staying balanced against the footer's 12px bottom inset.
…d error messages, e2e contract (UsefulSoftwareCo#992) * Tool failures become visible telemetry: outcome attributes, derived error messages, recording-tracer contracts Expected tool failures (ToolResult.fail) ride the Effect success channel, so the tracer recorded healthy spans while users hit upstream error walls — in a week of prod data, zero of 19.5k plugin.openapi.invoke spans carried http.status_code (annotated inside the inner OpenApi.invoke span, not the wrapper queries target) and 562 of 804 executor.tool.execute error spans had an empty status.message (tagged errors with no message field). - Stamp executor.tool.outcome / error_code / error_status on executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult, plus tenant/subject attribution so queries no longer need a trace-id join against the outer request span. - Annotate http.status_code on the plugin.openapi.invoke wrapper span. - Derive messages for the message-less SDK tagged errors and stop formatInvocationCauseMessage rendering plain objects as [object Object]. - Add a recording Tracer to @executor-js/sdk/testing and telemetry contract tests that pin the span names, attributes, and error statuses production queries depend on. * Move product-observability plan note out of the repo * Pin the telemetry contract end-to-end instead of unit-level Replace the recording-tracer unit tests with one e2e scenario that drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream returning 502 → OTLP export — and asserts the spans as the suite's motel store actually received them: outcome/error_code/error_status + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, and ok/fail distinguishable. The unit tests verified span objects in process; the prod bug class this guards against (attribute stamped on a span the exporter never carries, failure riding the success channel) only shows up at the exported layer. Adds a Telemetry e2e service (motel span search with arrival polling), provided when the suite's motel boots (E2E_MOTEL_URL). * Document the telemetry surfaces for future work - prod-telemetry repo skill: the executor-cloud Axiom dataset layout (attributes.custom JSON map, span names and their attributes including the new outcome fields), working APL recipes for error digests and org attribution, prod-DB and PostHog access notes, and the deploy-canary procedure. - e2e/AGENTS.md: the Telemetry service — scenarios can assert on the spans the target actually exported, with the arrival-polling and fixture-tagging gotchas.
* fix supervised and local regressions * Update CLI web setup flow * Fix launchd reinstall after disable * Improve CLI install progress output * Fix local service convergence * Fix Windows service uninstall cleanup
…ce (UsefulSoftwareCo#997) * Curate integration & connection metadata; update OpenAPI specs in place Integrations and connections gain user-curated, agent-visible metadata, a way to refresh an OpenAPI spec without re-adding it, and lazy propagation of tool changes across every member's connections. Descriptions & name/description split - integration rows gain a `name` column distinct from `description` (name = display, description = agent-visible context); readers fall back to description on pre-split rows - connections carry a nullable, agent-visible `description` - both surface to agents: the execute tool's connection-prefix inventory, sources.list, and connections.list (slug/name echoes suppressed) - add flows prefill the description from the source (OpenAPI info.description, GraphQL schema description, MCP server instructions) - the integration Edit sheet has separate Name and Description fields with a live "what agents see" preview; per-connection Edit sheet for description + account label; the rename telemetry event is preserved updateSpec - openapi.updateSpec(slug, {spec?}) re-resolves the spec (stored source URL, Google Discovery bundle, or a pasted blob), rebuilds the tool catalog in place, and reports an added/removed tool diff; connections, credentials, policies, and curated metadata survive. Spec text is stored in the plugin blob store by content hash, matching addSpec. - POST /openapi/integrations/:slug/spec - spec fetches stay unauthenticated by design (a spec can be hosted by a third party; sending a credential there would leak it) — auth-gated specs use the paste path Plugin config in the Edit sheet - plugin-owned configuration moved into the integration Edit sheet via a new `editSheet` slot; OpenAPI contributes the spec-update controls, MCP its auth-method editor. One Save applies the metadata change then the staged plugin change. Multi-user tool convergence - integration `tools_revised_at` + connection `tools_synced_at` stamps; tools.list lazily rebuilds any visible connection whose catalog predates its integration's last config change, under the reader's own binding — so a spec update by one member reaches other members' personal connections on their next read (the owner policy prevents the updater from writing their rows directly) - the drizzle runtime schema-ensure now adds nullable columns to existing tables on boot (self-host / D1); cloud migration 0003 adds the four columns for Postgres Add-time description prefill, the metadata edit sheets, updateSpec, and the multi-user convergence are covered by new SDK tests (including a two-subject shared-database test) and e2e scenarios. * Evolve apps/local SQLite columns on boot like the other hosts apps/local stood up its schema with CREATE TABLE IF NOT EXISTS plus a hand-maintained list of ALTER TABLE ADD COLUMN statements, so columns added by a later schema (integration.name, integration.tools_revised_at, connection.description, connection.tools_synced_at) never reached databases created by an earlier baseline — the first query against one of them would 500. Route local through ensureDrizzleRuntimeSchemaFromTables, the same boot bring-up the self-host and Cloudflare hosts already use, so every nullable column the live schema declares is added in place. This removes the per-column drift between hosts; cloud Postgres keeps getting the same columns from its generated drizzle migration. Add a fumadb test proving an existing table gains the schema's new nullable columns on ensure, and that a second ensure tolerates the duplicates. * Cover metadata editing and multi-user spec convergence with e2e scenarios Two behaviors PR UsefulSoftwareCo#997 introduced had no watchable coverage: - metadata-editing (cross-target): an integration's name and description, and a connection's description, are editable after the fact and read back — the write half that openapi-update-spec's survives-a-refresh assertion assumes. - spec-update-convergence (cloud, real multi-user org): when one member refreshes a shared integration's spec, a co-worker's OWN connection converges to the new tool catalog on the co-worker's next read, even though the editor can never write the co-worker's rows. Proves the lazy syncStaleConnectionTools path end-to-end. * Let the e2e viewer bind a specific host (HOST env) Defaults to all-interfaces as before. Setting HOST=127.0.0.1 binds loopback so the viewer can sit behind a `tailscale serve` proxy without colliding with tailscaled's own bind on the tailnet IP:port. * Rename integration.tools_revised_at → config_revised_at Tools are materialized per connection (the tool rows are keyed by owner/integration/connection); an integration has no tools, it has the config that derives them. So the integration-side staleness marker is about the config that changed, not the tools — name it for the cause. The connection keeps tools_synced_at, which is accurate (connections do have tools). The comparison now reads tools_synced_at < config_revised_at: 'this connection's tools predate the current config'. Pre-release column, so the cloud migration + snapshot are edited in place; the libSQL/D1 hosts pick up the new name via boot schema-ensure. * Regenerate the self-host route tree the merge left stale The merge commit captured bootstrap's transient regen of the self-host routes against a stale built console-routes, dropping the {-$orgSlug} prefix (admin/api-keys routes + routeTree.gen). Regenerate them so the self-host console routes match main's org-slug scope. * Split integration name from description; backfill cloud data clean Production data confirms `integration.description` was universally the display NAME (367 rows: avg 13 chars, ~0 real sentences), never a description. - description is now nullable — an actual, optional description distinct from the name. - ONE cloud migration 0005: add name + config_revised_at (+ connection description/tools_synced_at), DROP NOT NULL on description, then backfill name = description (367/367 get a real name, verified none empty) and clear description to NULL so no row carries a duplicated title. name stays column-nullable on purpose: SQLite boot-ensure hosts (self-host, Cloudflare, local) cannot add a NOT NULL column to an existing table, so the read path keeps the name ?? description ?? slug fallback.
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
UsefulSoftwareCo#1020 moved the foreground web server behind `executor web --foreground`: plain `executor web` now opens the installed background service and prints "Executor is not running" when none exists. The release-bootstrap smoke test still invoked `executor web --port`, so it never got a served response and the v1.5.12 publish gate failed at release:check. Pass --foreground to exercise the same runForegroundSession path the test was written for.
…expiry errors (UsefulSoftwareCo#976) * Add failing e2e: paused approval must survive the session runtime idle teardown Reproduces the reported bug: with model elicitation over streamable HTTP, execute returns a paused executionId, but the session DO's 5-minute idle alarm calls closeRuntime() and rebuilds the engine from storage on the next request — the in-memory pause map (fiber + Deferred) is gone, so resume answers 'No paused execution: exec_1'. A 20-second gap passes (the runtime is still warm); the 6-minute gap fails deterministically, so the loss is the app's own idle teardown, not a workerd eviction race. * Make pause/resume honest and idempotent; ids unique across engine rebuilds Production telemetry showed two failure classes behind 'No paused execution': duplicate resumes (client retries a lost response seconds after a successful resume) and pauses outliving the session runtime (idle teardown, isolate eviction). Durable pause state is intentionally out of scope; instead: - resume() is idempotent per executionId: settled outcomes are cached (exits, so failures re-fail typed) and replayed on retry; a concurrent duplicate joins the in-flight resume via a Deferred instead of missing the consumed pause. - execution ids are exec_<uuid> instead of a per-engine exec_N counter, so a rebuilt engine (cold restore, redeploy) can never reissue an id a stale client still holds and bind its resume to a different execution's pause. - a sandbox fiber that settles without a resume (timeout, failure) drops its abandoned pause entries and records the terminal outcome, so getPausedExecution stops reporting dead pauses and the map no longer leaks. - the resume miss now explains itself: paused executions expire, recovery is re-running execute. Structured status execution_not_found, recovery re_execute. The idle-teardown e2e scenario now specs the accepted contract: an approval paused past the idle window expires with the re-run guidance and the session keeps working (fresh execute pauses under a new id and resumes to completion); a new scenario locks duplicate-resume replay end-to-end.
PR UsefulSoftwareCo#976 added a single changeset bumping both @executor-js/execution (in the fixed release group) and @executor-js/host-mcp (in the changeset ignore list). Changesets rejects a changeset that mixes ignored and non-ignored packages, so `changeset version` failed and the Release workflow could not open the Version Packages PR. Split into two homogeneous changesets, keeping both changelog entries.
## What
Drops the `_`-prefix convention for unused function parameters. They now
keep their real, descriptive names.
- **Lint:** `no-unused-vars` is configured with `{ "args": "none",
"caughtErrors": "none", "varsIgnorePattern": "^_" }`. Unused params +
catch bindings are no longer flagged (no `_` needed); unused
**variables** and **imports** are still caught.
- **Sweep:** 37 unused params renamed across 23 files (`_args` → `args`,
`_event` → `event`, `_cause` → `cause`, …). Intentional `_`-names that
are actually used (`_db`, `_open`), destructuring discards, codegen
string-literals, and vendor code were left untouched.
Pure rename — no behavior change.
## One thing to weigh in on
Destructuring discards (`const { type: _type, ...rest } = obj`) **keep**
the `_` via `varsIgnorePattern: "^_"`, because they structurally need a
name to exclude a field while spreading the rest. If you'd rather those
also drop the underscore (and find another way to omit the field), say
so and I'll follow up.
## Verification
- `bunx oxlint -c .oxlintrc.jsonc --deny-warnings packages apps` → 0
warnings, 0 errors
- `bunx turbo run typecheck` → 43/43 pass
… store + real-catalog eval (#30) Rename plugin-vectorize-search -> plugin-semantic-search and make it backend-agnostic: pluggable VectorStore (Vectorize, zvec, sqlite-vec) and FTS5 lexical store behind a shared namespace filter, transactional upserts, lazy native deps, and a real-catalog eval harness with a sample-catalog golden set for offline smoke.
…est in CI (#31) Skip the timing-sensitive scope-isolation concurrency test under CI via test.skipIf(!!process.env.CI); it intermittently exceeds the 15s budget on shared runners and reds CI for unrelated PRs. Still runs locally, so the cross-fiber identity-leak guarantee stays covered.
… peerDependencies
…eindex population (#32) Add makeD1FtsLexicalStore (async FTS5 over Cloudflare D1, sharing the schema/queries with the sync better-sqlite3/bun:sqlite adapter) and wire reconcileToolCatalog to populate an optional lexicalStore, so a Cloudflare host gets real Vectorize + FTS5/BM25 hybrid tools.search. Lexical doc ids are namespace-scoped; the lexical index is rebuilt in full per reindex (complete even when attached to an already-indexed deployment); D1 schema init is tracked via a WeakSet so a transient failure is never cached.
…rch/status API (#33) Adds the Search sidebar tab (live tools.search over the hybrid provider, index status, reindex), the ./api search+status endpoints, the shared search/status extension methods, and FtsLexicalStore.count. Greptile 5/5.
…e + sweepRemoved (#34) Splits the full-catalog reconcile into resumable, engine-agnostic primitives (page + sweep) so a durable driver can run each within its own CPU budget. Removed tools are now actually swept. Greptile 5/5.
- add an Effect KeyValueStore-backed cache primitive for derived executor artifacts - let hosts provide a durable KeyValueStore and wire Cloudflare KV through a generic optional `CACHE` binding - use that generic cache from the TypeScript schema-preview path, keyed by reachable schema roots/definitions so unused pathological definitions do not poison the preview path - bun run --filter @executor-js/sdk test -- src/executor.test.ts - bun run --filter @executor-js/sdk typecheck - bun run --filter @executor-js/api typecheck - bun run --filter @executor-js/cloudflare typecheck - dogfood host: bun run typecheck - `apps/host-cloudflare` typecheck still fails in this checkout because `@executor-js/plugin-semantic-search` is not resolvable in the current package graph; the same dogfood host typechecks when linked to the live package workspace.
## Summary Use the current agents MCP transport primitives for Cloudflare MCP sessions and persist resumable events in Durable Object storage. This removes the local reflection shim around private transport internals. ## Changes - Upgrade `agents` to `^0.16.2` in the Cloudflare host. - Pass `DurableObjectEventStore` into `WorkerTransport` for MCP session resumability. - Delegate request handling directly through the JSON-RPC request-id queue. ## Tests - `bun run --cwd packages/hosts/cloudflare typecheck` - `bun run --cwd packages/hosts/cloudflare test` - `bun run typecheck` - `bun run test` - `bunx oxfmt --check <changed files>` - `bunx oxlint -c .oxlintrc.jsonc <changed files> --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#38** 👈 current 2. #39 3. #40 <!-- stack:links:end -->
## Summary Add structured MCP session request logs and span attributes around the Cloudflare Durable Object request path. This makes long-lived stream behavior visible by method, headers, session state, response status, and elapsed wall time. ## Changes - Emit one structured `mcp_session_request` log per Cloudflare MCP session request. - Add request/transport/event-store attributes to MCP session spans. - Convert MCP session-mode logging to structured JSON. ## Tests - `bun run --cwd packages/hosts/cloudflare typecheck` - `bun run --cwd packages/hosts/cloudflare test` - `bun run --cwd packages/hosts/mcp typecheck` - `bun run --cwd packages/hosts/mcp test` - `bun run typecheck` - `bun run test` - `bunx oxfmt --check <changed files>` - `bunx oxlint -c .oxlintrc.jsonc <changed files> --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #38 2. **#39** 👈 current 3. #40 <!-- stack:links:end -->
Remove the legacy configurable SSE remote-client mode and use Streamable HTTP for remote MCP clients. This keeps the remote MCP path aligned with the transport used by the Cloudflare Durable Object session work. - Remove remote MCP transport selection from public API payloads and stored config. - Drop `SSEClientTransport` from remote client connection setup. - Update e2e payloads and expectations to the flattened remote config shape. - `bun run --cwd packages/plugins/mcp typecheck` - `bun run --cwd packages/plugins/mcp test` - `bun run --cwd e2e typecheck` - `bun run typecheck` - `bun run test` - `bunx oxfmt --check <changed files>` - `bunx oxlint -c .oxlintrc.jsonc <changed files> --deny-warnings` <!-- stack:links:start --> 1. #38 2. #39 3. **#40** 👈 current <!-- stack:links:end -->
## Summary Caches fully assembled `ToolSchemaView` results in the configured `KeyValueStore`, using a content-addressed key so repeated schema views can skip preview assembly work after the raw schema data is unchanged. ## Changes - Adds a schema-view cache entry schema and deterministic key builder that includes tool metadata, input/output schemas, definitions, and TypeScript preview cache/compiler versions. - Reads and writes the schema-view cache from `executor.tools.schema` for static and persisted tools. - Adds regression coverage for cache-key invalidation and executor cache writes. ## Tests - `bun run --cwd packages/core/sdk test src/tool-schema-view-cache.test.ts src/executor.test.ts` - `bun run --cwd packages/core/sdk typecheck` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#36** 👈 current 2. #37 <!-- stack:links:end -->
Migrates the host-cloudflare `/mcp` route to the Agents `McpAgent.serve()` Streamable HTTP bridge while keeping the existing Executor MCP server construction and approval hooks. - Adds a shared `McpAgentSessionDOBase` for Cloudflare-hosted Executor MCP sessions. - Routes host-cloudflare `/mcp` at the Worker edge so authenticated session props can flow through `ExecutionContext` into the hibernatable Durable Object. - Removes the host-cloudflare app-local `McpSessionStore` mount and its forwarded-request session-store wiring. - Updates workerd e2e coverage to accept Streamable HTTP SSE-framed JSON-RPC responses and cover initialized GET streams. - `bun run --cwd packages/hosts/cloudflare typecheck` - `bun run --cwd apps/host-cloudflare typecheck` - `bunx oxlint packages/hosts/cloudflare/src/mcp/agent-session-durable-object.ts apps/host-cloudflare/src/mcp/agent-handler.ts apps/host-cloudflare/src/mcp/session-durable-object.ts apps/host-cloudflare/src/mcp/index.ts apps/host-cloudflare/src/worker.ts apps/host-cloudflare/src/app.ts apps/host-cloudflare/src/worker.e2e.node.test.ts --deny-warnings` - `bun run --cwd apps/host-cloudflare test src/worker.e2e.node.test.ts` - `bun run --cwd apps/cloud typecheck` - `bun run --cwd apps/host-cloudflare build`
## Summary Adds a staged, fingerprint-aware semantic-search reindex pipeline so hosts can drive durable indexing work without rebuilding unchanged tool documents. ## Changes - Adds staged index collections, job planning, and snapshot/fingerprint helpers for semantic-search indexing. - Updates reindex status/API surfaces and React atoms/page wiring for the staged pipeline. - Lets schema consumers opt out of TypeScript preview generation when they only need raw schema content for fingerprinting. ## Tests - `bun run --cwd packages/plugins/semantic-search test src/sdk/fingerprint.test.ts src/sdk/indexer.test.ts` - `bun run --cwd packages/core/sdk test src/tool-schema-view-cache.test.ts src/executor.test.ts` - `bun run --cwd packages/plugins/semantic-search typecheck` - `bun run --cwd packages/core/sdk typecheck` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #36 2. **#37** 👈 current 3. #42 4. #43 <!-- stack:links:end -->
Adds a core SDK tool schema manifest that stores cheap, stable schema fingerprints during connection refresh and exposes them through `executor.tools.manifest()`. - Adds `tool_schema_manifest` to the core schema and public SDK type exports. - Writes manifest rows alongside tool/definition rows during connection refresh and removes stale rows when connections or integrations are removed. - Adds `tools.manifest()` with policy filtering and static tool support so indexers can diff without loading full schema views. - `bun run test -- src/executor.test.ts` in `packages/core/sdk` - `bun run typecheck` in `packages/core/sdk` - `bunx oxlint -c .oxlintrc.jsonc packages/core/sdk/src/types.ts packages/core/sdk/src/core-schema.ts packages/core/sdk/src/executor.ts packages/core/sdk/src/executor.test.ts packages/core/sdk/src/index.ts packages/core/sdk/src/shared.ts packages/core/sdk/src/promise.ts --deny-warnings` <!-- stack:links:start --> 1. #37 2. **#42** 👈 current 3. #43 <!-- stack:links:end -->
## Summary Rewires semantic-search indexing to use the core tool schema manifest as its scan/diff source instead of loading each tool schema during scan. ## Changes - Adds manifest listing/slicing for index scan inputs. - Makes `ToolSearchIndex.create`, `scan`, and removed-tool sweep consume manifest paths and fingerprints. - Keeps TypeScript schema/codegen work in the changed-tool chunking path only, and updates tests to assert scan does not perform raw schema reads. ## Tests - `bun run test -- src/sdk/tool-search-index.test.ts` in `packages/plugins/semantic-search` - `bun run typecheck` in `packages/plugins/semantic-search` - `bunx oxlint -c .oxlintrc.jsonc packages/plugins/semantic-search/src/sdk/documents.ts packages/plugins/semantic-search/src/sdk/tool-search-index.ts packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #37 2. #42 3. **#43** 👈 current <!-- stack:links:end -->
Recover downstream queue messages from persisted index state when chunk or embed queue turns are retried after D1 writes have already advanced job or chunk status.
## Summary Improves the semantic-search indexing path by making derived schema caches durable through Effect `KeyValueStore`, reducing queued index materialization work, and bounding Cloudflare Vectorize metadata before upsert. ## Changes - Replace large per-isolate schema `Effect.Cache` layers with direct typed `KeyValueStore` cache reads/writes and an in-memory default when hosts do not provide cache storage. - Make queued indexing use raw schema reads plus compact schema facet text instead of TypeScript preview generation. - Cache materialized index documents by tool fingerprint and batch commit work across paths. - Cap Vectorize metadata descriptions and reject oversized metadata before calling Vectorize. ## Tests - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/core/sdk test -- src/executor.test.ts src/tool-schema-view-cache.test.ts src/tool-typescript-preview-cache.test.ts` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/plugins/semantic-search test -- src/sdk/tool-search-index.test.ts src/sdk/store-cloudflare-limits.test.ts src/sdk/chunker.test.ts` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/plugins/semantic-search typecheck` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/core/sdk typecheck` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/core/api typecheck` - `bun run typecheck` in `/Users/aryasaatvik/Developer/AryaLabsHQ/executor` ## Known tradeoffs - Host queue batching is committed in the dogfood host checkout separately and is not part of this package PR. - This was not deployed and no indexing run was triggered. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#44** 👈 current 2. #45 <!-- stack:links:end -->
## Summary Batches semantic-search index storage operations so queue messages do fewer plugin-storage and D1 calls while preserving the existing indexing lifecycle. ## Changes - Adds typed bulk `getMany`, `getManyForOwner`, `putMany`, and `removeMany` operations to plugin storage collections. - Reworks scan, chunk, embed, and commit indexing paths to bulk-load and bulk-write jobs, chunks, and fingerprints. - Updates semantic-search test fixtures and plugin-storage tests to cover the new bulk facade methods. ## Tests - `bun run --cwd packages/core/sdk test -- src/plugin-storage.test.ts` - `bun run --cwd packages/plugins/semantic-search test -- src/sdk/tool-search-index.test.ts` - `bun run --cwd packages/core/sdk typecheck` - `bun run --cwd packages/plugins/semantic-search typecheck` - `bun run format:check` - `oxlint -c .oxlintrc.jsonc packages/core/sdk/src/executor.ts packages/core/sdk/src/plugin-storage.ts packages/core/sdk/src/plugin-storage.test.ts packages/plugins/semantic-search/src/sdk/tool-search-index.ts packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts --deny-warnings` ## Known tradeoffs - Root `bun run lint` still fails on pre-existing `.scratchpad` files outside this diff. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #44 2. **#45** 👈 current <!-- stack:links:end -->
Adds a bulk upsert path through FumaDB and uses it for plugin-storage `putMany`, removing the delete-then-insert write pattern from bulk plugin storage updates. - Add `upsertMany` to FumaDB query surfaces, with native Drizzle `ON CONFLICT DO UPDATE` support for SQLite/Postgres and memory/fallback adapter behavior. - Route SDK plugin-storage `putMany` through `core.upsertMany` using the plugin-storage unique key. - Reduce semantic-search index status reads by grouping job counts with plugin-storage aggregation and dedupe commit paths before job loads. - Cover policy-aware bulk upserts and update semantic-search test fixtures for grouped counts. - `bun run --cwd packages/core/fumadb typecheck` - `bun run --cwd packages/core/sdk typecheck` - `bun run --cwd packages/plugins/semantic-search typecheck` - `node ../../../node_modules/vitest/vitest.mjs run src/query/table-policy.test.ts` from `packages/core/fumadb` - `bun run --cwd packages/core/sdk test src/plugin-storage.test.ts` - `bun run --cwd packages/plugins/semantic-search test src/sdk/tool-search-index.test.ts` - `./node_modules/.bin/oxlint -c .oxlintrc.jsonc <touched files> --deny-warnings`
## Summary Preserve a reindex run's `maxTools` cap across `resume`. Previously `resume` re-enqueued scan partitions with no limit, so a run started capped (e.g. a dev/debug pass) would run **uncapped** after a resume. ## Change - `create()` stores `maxTools` on the index-run record (`indexRuns` collection; optional — absent = uncapped). - `ReconcileResult` exposes `maxTools`; `reconcile()` returns `run.data.maxTools`. - A host can now forward `reconciled.maxTools` when re-enqueuing scan partitions on resume. ## Test - New: `reconcile` preserves the run's `maxTools` (capped → value, uncapped → undefined). - `tsgo --noEmit` ✓ · `oxlint --deny-warnings` (changed files) ✓ · `vitest run` 9 passed ✓ ## Follow-up The dogfood host (`AryaLabsHQ/executor`) `resume` route forwards `reconciled.maxTools` once this lands (tracked in the host backlog). Surfaced originally by a Greptile P1 on the host queue PR.
…enant namespace (#48) ## Summary `executor.semanticSearch.search` (the operator `tools.search` surface) returned **zero results for every query**, even against a fully populated vector + lexical index. Root cause: a single `namespace` value was overloaded as two different concepts. ## Root cause The search method conflated: - **`deps.namespace`** — the *tenant* namespace (e.g. `"default"`), used for Vectorize storage isolation inside `store.query`. - **`input.namespace`** — an *optional integration/path-prefix filter* the operator can pass to narrow results (e.g. `"github"`). With no operator-supplied prefix, it defaulted the prefix filter to the tenant namespace: ```ts const namespace = input.namespace ?? deps.namespace; // ← bug ``` The provider's `matchesNamespace` then required every tool's integration/path to *start with* the tenant id (`"github_api".startsWith("default")` → `false`), so it dropped every match. This affected both the vector and lexical paths, since they share `matchesNamespace`. ## Fix Apply the integration prefix only when explicitly provided. The tenant namespace drives the store query alone and is still reported on the result page. ## Verification - New regression tests at the plugin-search level: with no prefix the provider receives `undefined` (no filter) and tenant-tagged results survive; an explicit prefix still passes through. - Suite green: `vitest` (20 passed), `tsgo --noEmit`, `oxlint --deny-warnings`. - Verified live on a self-host deploy — before: `{"items":[]}` for every query; after: `"query my postgres database"` → Neon/Planetscale SQL tools, `"github"` → `github_api` tools, and `?namespace=stripe_api` still narrows correctly. ## Notes - `makeSemanticSearchExtension` is exported for the unit test but **not** re-exported from the package index, so the public API is unchanged. - Changeset included (`patch`).
## Why
Production traces showed the web UI's tool-detail path (`GET
/api/tools/schema`, `includeTypeScript: true`) paying the json-schema →
TypeScript compile on the first view of every tool. The reindex pipeline
already visits every tool's schema, but it called `tools.schema({
includeTypeScript: false })` specifically to *skip* that compile — so it
warmed the wrong cache variant and the UI's variant stayed cold until
first manual view.
## What
- `executor.tools.schema` always renders the TypeScript preview; the
`includeTypeScript` option and the now-empty `ToolSchemaOptions` type
are removed.
- The reindex pipeline (`collectDocForTool`) calls `tools.schema`
plainly, so every reindex pre-warms exactly the KV schema-view the UI
reads → first tool-detail views become cache hits instead of a cold
compile.
- `includeTypeScript` is dropped from the schema-view cache key (content
+ manifest), collapsing the two variants into one.
## Safety
- Embedding/lexical text is unchanged: `collectDocForTool` only reads
the raw schema facets (`inputSchema` / `outputSchema` /
`schemaDefinitions`), never the rendered TypeScript — so no re-embedding
/ index churn.
- The cache-key change supersedes existing entries; they re-warm on the
next reindex or view (regenerable derived data).
## Gate
tsgo (43/43), `oxlint --deny-warnings`, oxfmt, and vitest for
`@executor-js/sdk` + `@executor-js/plugin-semantic-search` all green.
Follow-up: a reindex on the deployed instance warms the new unified
variant.
<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)
1. **#49** 👈 current
2. #50
<!-- stack:links:end -->
**Phase 1** of Tier-2 tool-catalog caching — stacked on #49. Caches `executor.tools.list` (today a full `tool` scan + `tool_policy` read on every call) behind a **content-derived revision key**, so the list stays warm and only re-reads D1 when the catalog or policies actually change. No TTL. - `produceConnectionTools` writes a per-connection **`toolset_revision`** = `sha256` of the connection's manifest rows' `(path, index_fingerprint)`, in **`plugin_storage`** (owner-scoped, **no schema migration**), at all 5 catalog-mutation sites (produce, both clear branches, connection-remove, integration-remove). - `catalogGen` combines the in-scope connections' revisions; `policyGen` combines the `tool_policy` rows. Key = `sha256(owner, integration, connection, includeBlocked, catalogGen, policyGen)`. - **Token rotation does not bust** (never calls `produceConnectionTools`); add/remove/change a tool, or any policy edit, does. - Cached value = the DB-derived, policy-filtered list. **Static tools unioned live** (never cached); **`query` applied post-cache** so distinct queries share one cached set. - Hand-authored `ToolListItem` Schema (list projection only — no input/output schema); branded ids stored as strings, rebuilt on read. - `catalogGen`/`policyGen` reads are tenant-wide → per-tenant-coarse invalidation (a catalog change re-warms all owners' list caches). Correct, just coarse. tsgo (sdk) · `oxlint --deny-warnings` · oxfmt · vitest **sdk 350/350** (new cache suite: add/remove/content invalidation, policy bust, query post-cache, static-union-on-hit). Phase 2 (stacked) moves the UI list onto the compact manifest tier. <!-- stack:links:start --> 1. #49 2. **#50** 👈 current <!-- stack:links:end -->
…51) ## Summary Introduces a shared windowed-list primitive (`@executor-js/react/components/virtual-list`) and adopts it on the web-UI surfaces that mount unbounded or long lists, so they keep a bounded DOM instead of rendering every row at once. This is **M1 of a 2-PR stack**. The follow-up adds server-side cursor pagination to `tools.list` — the one list whose API still returns the whole set — and stacks on top of this branch. ## What's here (one commit per surface) - **VirtualList primitive** — `@tanstack/react-virtual` with dynamic measurement, an `onEndReached` infinite-scroll trigger (generalized from the runs shell's hand-rolled distance-from-bottom check), optional sticky section headers, and scroll restoration keyed by id. A `useWindowedRows` hook covers surfaces that can't use `<div>` rows (real `<table>`s). - **Tool source-list** (worst case) — a broad source could mount thousands of tool rows at once, each carrying a dropdown-menu portal. The grouped account sections and dotted-name trees are flattened into one windowed list; section headers become sticky virtual items; only near-viewport rows mount. - **Runs table** — windowed via `useWindowedRows`; the real `<table>` + sticky `<thead>` stay, with top/bottom spacer `<tr>`s for off-screen rows. The two variable-width columns are already capped, so windowing a subset doesn't make `table-layout: auto` jitter. - **Run-detail drawer** — the logs and tool-calls tabs get their own windowed scroll containers; tool-call expanded state is lifted so it survives a row scrolling out of the window and back. - **Sticky-header range logic** extracted to a pure function with unit tests. ## Deliberately not virtualized The integrations catalog, the service-tokens table, and the org-members list are **page-scrolled** lists that hold dozens of rows in practice. Virtualizing them would mean introducing an independent bounded scroller — a UX change for negligible benefit. The primitive is ready if any ever grows unbounded. ## Gates - `typecheck`: 43/43 packages - tests: shared React package 162, execution-history 19 (including new windowing-logic unit tests) - `oxfmt` + `oxlint --deny-warnings` clean on all changed files ## Verification still to do (live browser) Unit tests cover the windowing math; the visual behaviour needs a real browser (the e2e browser surface is cloud-only). Worth exercising by hand: - Sticky section headers pin and swap correctly while scrolling a grouped tool list. - Runs-table column widths stay stable while scrolling deep into the keyset list. - Detail-drawer logs / tool-calls scroll and measure correctly; expanded tool calls keep their state across scroll.
…an) (#52) ## Why Production traces showed the reindex **scan** phase dominating wall-time: ~30 min for an all-skip pass over ~5,400 tools, ~3 tools/s. The cause is read amplification — `scan` calls `tools.manifest()` (a full `tool_schema_manifest` table read) **once per partition page**, then filters to its partition and slices `SCAN_PAGE` rows. With `PARTITION_COUNT=12` and `SCAN_PAGE=25` that's ~216 scan messages, each re-reading all ~5,400 rows cross-region (SIN→NRT): `O(N²/pageLimit)` ≈ **1.17M row-transfers** to scan 5,400 tools. `partition` is computed in-app (`hash(path) % count`), so it can't be pushed down to SQL without a migration. ## What - **`@executor-js/sdk`** — `createExecutor` exposes its cache `KeyValueStore` on the instance as **`executor.cache`** (Cloudflare KV in prod, in-memory fallback otherwise). Plugins reach durable cache storage without re-threading a binding; wrap with `KeyValueStore.toSchemaStore` like the internal `tools.list` / schema-view caches. - **`@executor-js/plugin-semantic-search`** — `create` reads the manifest **once**, partitions it (preserving each entry's full-list ordinal), and writes **one snapshot per partition** to `executor.cache` (`index-manifest/{runId}/{partition}`). `scan` reads only its partition's snapshot — **KV-only, no D1 fallback on the hot path**. A miss fails the message so the queue retries; the host delays the *initial* scan fan-out once to let the write propagate (separate host PR). Snapshots are deleted on run completion. ## Result - D1 manifest reads: `~1.17M → ~5,400` (a single pass at `create`). - Per-scan-message memory: full manifest (~5 MB) → one partition (~400 KB). - Scan wall-time: **~30 min → seconds–minutes** (and the host delay PR keeps the fast path certain). ## Safety / notes - KV-only by design: a snapshot miss surfaces loudly (queue retry / DLQ) rather than silently degrading to the slow D1 path. - Snapshots are partition-sharded so no single KV value approaches the 25 MB cap (scales to ~300k tools). - KV has no native TTL through the abstraction, so cleanup is explicit on completion; a stalled / terminally-failed run leaves its (small) snapshot until overwritten — a known, bounded leak. - Behavior preserved: the scan's skip-vs-changed classification, ordinal ordering, pagination, and `maxTools` cap are unchanged (the snapshot is built from the same `listToolManifests` call). ## Gate tsgo **43/43** · `oxlint --deny-warnings` 0/0 · oxfmt · vitest **sdk 350/350**, **plugin-semantic-search 78/78** (new: manifest-read-once-per-create, KV-miss-fails-no-fallback, snapshot-cleared-on-complete). Stacked deploy: needs the host PR that delays the initial scan fan-out for KV propagation.
aryasaatvik
force-pushed
the
sync/upstream-main-2026-06-22
branch
from
June 23, 2026 10:13
1b01701 to
7f22595
Compare
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
devontoupstream/mainatcb13484d6154a5ad5022ee030cfb5f093065f32a.fix(sync): resolve upstream rebase type fallout.Conflict resolution notes
tools.listcache revision tracking.Verification
bun install --frozen-lockfilegit diff --name-only --diff-filter=ACM -z upstream/main...HEAD | xargs -0 bunx oxfmt --checkgit diff --name-only --diff-filter=ACM -z upstream/main...HEAD | xargs -0 bunx oxlint -c .oxlintrc.jsonc --deny-warningsbun run typecheckbun run testReview/promotion note
This PR is for reviewing the rebase and conflict resolution in GitHub. Do not merge it with the GitHub merge button. After review, promote the reviewed branch to
devwith:Local note: full
bun run format:checkandbun run lintstill traverse copied.scratchpadfiles in this isolated worktree; the tracked source diff passesoxfmtandoxlint --deny-warnings.