Repository navigation
chore(sync): preview upstream main merge - #55
Closed
aryasaatvik wants to merge 58 commits into
Closed
aryasaatvik wants to merge 58 commits into
aryasaatvik wants to merge 58 commits into
Conversation
Adds the **execution-observer foundation** — the seam the history and metrics plugins build on. - New `ExecutionObserver` contract in the SDK: six tagged lifecycle events (`ExecutionStarted`/`Finished`, `ToolCallStarted`/`Finished`, `InteractionStarted`/`Resolved`) + `composeExecutionObservers`, with a `plugin.runtime.executionObserver` hook. - The execution engine emits the events (re-derived onto the current Queue-based pause model), exposes `Executor.owner`, and tags the run span with the execution id + trigger. - `makeExecutionStack` composes plugin observers — one seam covers every HTTP host and the MCP Durable Objects; local's fixed-execution + in-process MCP are wired directly. **Behaviour-neutral / opt-in:** with no plugin observing, every path is a no-op. Gates: `format:check`, `lint`, full-monorepo `typecheck`, and observer unit tests all green. Live app verification still to come. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#12** 👈 current 2. #13 3. #14 <!-- stack:links:end -->
Adds **`@executor-js/plugin-execution-history`** — an opt-in execution-history sink built on the observer foundation. - Records every run (plus its tool calls and interactions) into three owner-scoped `pluginStorage` collections via a buffered-batch writer — **no new tables, no migrations**; works on D1/libSQL/sqlite/postgres. - `./api`: a scoped read API (`list`/`get`/`listToolCalls`) over HttpApi (`executionHistoryHttpPlugin`). - `./client`: a Runs page at `/plugins/executionHistory/` (table, status filter, pagination, 4-tab detail drawer) — mounts via the existing plugins catch-all with zero router changes. The base `.` entry never loads the API or React layers. Fully opt-in (not registered in any default config). Stacked on the execution-observer foundation. Gates: `format:check`, `lint`, full-monorepo `typecheck`, unit + handler tests green. Runs-page live render still to verify. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #12 2. **#13** 👈 current 3. #14 <!-- stack:links:end -->
Adds **`@executor-js/plugin-execution-metrics`** — an opt-in metrics sink built on the observer foundation. - Base plugin drives Effect Metric counters + an execution-duration histogram from the event stream. - `./api`: an auth-gated `GET /execution-metrics/metrics` Prometheus scrape (`executionMetricsHttpPlugin`). - `./cloudflare`: a Workers Analytics Engine sink (`writeDataPoint`) — per-isolate Effect metrics being meaningless on a Worker fleet. Wired opt-in on host-cloudflare, gated on the `ANALYTICS` binding. Base `.` entry free of `@executor-js/api` and `@cloudflare/workers-types`. Stacked on the execution-observer foundation. Gates: `format:check`, `lint`, full-monorepo `typecheck`, and metrics-render tests green (an adversarial-verify pass caught + fixed a Prometheus histogram-bucket bug). <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #12 2. #13 3. **#14** 👈 current <!-- stack:links:end -->
Pushes JSON-document aggregation and keyset pagination down to SQL.
**FumaDB** (`packages/core/fumadb`): adds `jsonCount`, `jsonGroupCount`,
`jsonTimeBuckets`, `jsonStats` (min/max + continuous percentiles), and
`jsonPage` (cursor/keyset) over a JSON document column, on the memory
and Drizzle (SQLite/Postgres) adapters; unsupported adapters fail
loudly. A shared JS evaluator keeps memory + the SQLite percentile
fallback aligned with Postgres `percentile_cont`.
**Plugin storage** (`packages/core/sdk`): the facade exposes
`collection.aggregate.{count,groupCount,timeBuckets,stats}` and
`collection.queryKeyset(...)`, translating a collection's indexed-field
`where` into JSON-path predicates while owner/tenant scoping stays
enforced by the storage policy.
Tests: fumadb 44 (incl. a 12-test suite run against both memory and real
SQLite); sdk 341 (incl. a suite driving aggregate/keyset through the
real Drizzle+SQLite executor). Repo-wide typecheck clean.
Not included: per-collection expression indexes (declared indexes still
full-scan the partition — a perf follow-up); the Postgres path uses
standard SQL but isn't unit-tested locally (no pglite).
First of a 3-PR stack: pushdown -> runs API -> runs UI.
<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)
1. **#15** 👈 current
2. #16
3. #17
<!-- stack:links:end -->
Builds on the JSON pushdown. Reworks the runs list into a
keyset-paginated, aggregate-aware read surface.
`list` now takes an opaque keyset cursor, a sort field
(`startedAt`/`durationMs`) + direction, and a live-tail `after` floor,
and returns `{ runs, nextCursor, meta }`. The `meta` block (computed
once per filter set on the first page) carries status/trigger facet
counts (each ignoring its own filter), interaction counts, a
stacked-by-status timeline with a server-chosen bucket width,
total/filtered row counts, and duration min/max + P50-P99. All counts,
buckets, percentiles, and pages are pushed to SQL.
Tests: 9 plugin tests on real SQLite (facets, cursor pages, live
`after`).
Not included: tool-path facets + full-text code search (need a
`contains` facade op).
Stacked on the FumaDB pushdown PR.
<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)
1. #15
2. **#16** 👈 current
3. #17
<!-- stack:links:end -->
Rebuilds the runs page as an openstatus-style observability table on the keyset + aggregate API, using the plugin's Effect-atoms idiom. - Faceted filter rail (status/trigger/interaction/time with live counts), cmdk filter palette, recharts stacked timeline with drag-to-zoom, live tailing, keyset infinite scroll, sortable columns, column visibility, keyboard shortcuts, and the existing 4-tab detail drawer. - List data flows through a `useRunsList` hook that accumulates cursor pages and polls the head for live mode; the accumulation/dedup logic is a pure, unit-tested reducer. - Adds `recharts` as an optional peer dependency (provided by `@executor-js/react`). Verified: lint/format/typecheck clean, 15 plugin tests, build success, repo-wide typecheck. An adversarial review pass caught and fixed 2 bugs (a refresh that blanked page 1; a recharts axis misconfig) plus 4 correctness issues (live-divider off-by-one, live-anchor timing, drag-cancel-on-leave, filter-merge clobbering). Coverage gap: no browser/render/e2e test this round — the reducer is unit-tested and the components were adversarially reviewed, but the rendered page has not been driven live. Worth exercising by hand: live divider, timeline drag-to-zoom, the cmdk parser, infinite scroll. Live mode is append-only (existing rows' status changes are not re-pulled; meta updates live). Filters are local state (not URL-persisted yet). Stacked on the runs API PR. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #15 2. #16 3. **#17** 👈 current <!-- stack:links:end -->
## Problem
Three layout issues on the runs page (`/plugins/executionHistory`),
visible at common laptop widths:
1. **Column misalignment** — from the trigger column rightward, every
header label sat one slot left of its data.
2. **`duration` sort header wrapped** onto a second line.
3. **Timeline bars rendered as ~136px slabs** when only a few buckets
are present.
## Root cause
(1) and (2) share a cause. The shared column-slot classes encoded their
visible state as a `block` display (`2xl:block`, `md:block`). When
composed via `cn()` with an element that carries its own `inline-flex`,
tailwind-merge collapses the two `display` values to one:
- The row's trigger cell (`inline-flex`) overrode the slot's `hidden
2xl:block`, so **rows** showed the trigger column while the **header**
hid it below `2xl` — desyncing every column after it.
- The duration sort button (`md:block` over its `inline-flex`) became a
block box, so the label + sort arrow wrapped to two lines.
## Fix
- Make the trigger and duration slots **single-display flex** classes
(`xl:flex` / `md:flex`) so no composition can desync header and row.
- Lower the **trigger breakpoint `2xl → xl`** so the column appears at
common laptop widths — aligned in both header and rows.
- Cap timeline bars with `maxBarSize={28}` so few buckets render as slim
centered columns instead of band-filling slabs.
## Verification
`oxfmt --check`, `oxlint`, `turbo typecheck` (9 packages), and the
plugin test suite (16 tests) all pass. Changes are presentation-only
(className strings + one recharts prop); no logic touched.
## What Restores the old observability run-detail drawer inside the execution-history plugin, composing the shared `@executor-js/react` primitives instead of the plugin's plain-text placeholders. | Feature | Before | After | |---|---|---| | Code + JSON | monochrome `<pre>` | shared shiki `CodeBlock` (highlight, copy, collapse) — `typescript` for code, `json` for result/args/interaction payloads | | Tool calls | flat path + duration rows | **timing waterfall** — Gantt bars positioned by start-offset, sized by duration, colored by status; raw-ms label | | Interactions | separate 4th tab | **inline on Properties** (kind · purpose · status badge · request/response) → 3 tabs | | Header | id only | **prev/next execution nav** + **Copy JSON** | | Properties | flat trigger string | status dot, `via <trigger> · tools N` line, `—` durations for in-flight runs | ## Scope UI-only — entirely within `packages/plugins/execution-history/src/react/` (`detail-drawer.tsx`, `RunsPage.tsx`, `format.ts`). **No** API, store, or event-contract changes, so this is independent of the in-flight actor/trigger-attribution work in `core/sdk`. The trigger line reads the existing `triggerKind` and lights up automatically once hosts populate it. ## Verification `oxfmt --check`, `oxlint`, `turbo typecheck` (9 packages), and the plugin test suite (16) all pass. A 3-pass adversarial review (parity / correctness / idiom) ran pre-PR; its findings were folded in — notably the waterfall min-width collapse, Gantt geometry, raw-ms labels, the sentinel detail fetch (now gated behind an open drawer), and the uppercase status badge. ## Deferred (intentional) - **Live-growing bars** for a still-running execution — the drawer is a point-in-time snapshot (consistent with the 10s-TTL detail atom). No ticker. - **Distinct Created vs Started** — `RunRow` has only `startedAt`; the drawer shows Started / Completed rather than inventing a `createdAt` field.
…y actor (#20) ## What Records **who/what triggered each run** and lets the runs page filter by it. The engine's `ExecutionStarted` already carried an optional `trigger`, but no caller ever populated it — so every run rendered with an unknown trigger. This wires it end to end and adds an identity ("actor") dimension on top. ## Changes **Trigger actor plumbing (generic, carrier-agnostic)** - `ExecutionTrigger` gains an `ExecutionActor {kind, id, label}`. - The trigger is populated at both execute call sites: the HTTP execute handler (`kind: "http"`) and the MCP session build (`kind: "mcp"`), each resolving the actor from the request principal. - `Principal`/`AuthContext` gain an `actor`, derived by `executionActorFromPrincipal` (defaults to a `user` actor keyed by `accountId`). The host-mcp `Principal` schema carries it too, so it survives cross-isolate session serialization. The HTTP handler reads `AuthContext` optionally, so it adds no new requirement. A host that authenticates a machine credential acting as a human can set `principal.actor` to keep the credential distinct from the subject it acts as. **execution-history** - `RunRow` persists an indexed `actorId` plus an `actorLabel`/`actorKind` snapshot. - The runs list filters by `actorFilter`; an `actorCounts` facet (keyed on the stable `actorId`, rendering the most-recent label + kind) joins the existing facets. - UI: an **Actor** facet in the filter rail, an **Actor** column, and an Actor field in the run detail drawer. As a side effect this also fixes the long-standing "unknown trigger" — `triggerKind` is now set (`http`/`mcp`). ## Verification - `typecheck` 41/41 · `lint` + `format` clean on all changed files - tests green: execution-history 17 (incl. a new test covering actor persistence, the facet label/kind, and `actorFilter`), api 45, host-mcp 32, cloud 52 ## Follow-up The seam is fully generic. A host that aliases a machine credential to a human subject populates `principal.actor` with the credential's own id/label so the Actor column/facet distinguish it from the human — no further UI change required.
## What
Restores the user-facing affordances the execution-history plugin
dropped when it replaced the old observability runs UI (`a952d08ad`).
Scoped from a full old-vs-new audit; **Groups A + B** (all client-side —
no API/store changes).
### Timestamps
- **`HoverCardTimestamp`** (new) — hovering a timestamp reveals **epoch
/ UTC / local-tz / relative** renderings, each **click-to-copy**. On the
list-row started cell (keeps its relative-time display) and the drawer's
Started/Completed. Native `Date`/`Intl`, reusing the shared Radix
`HoverCard` (no `date-fns`).
### List
- **Log column** — optional column counting `[error]`/`[warn]` lines
from the row's `logsJson` (off by default; toggle in view options).
Case-insensitive.
- **Slow-run highlight** — durations > 5s render in the destructive
color.
- Code preview widened **80 → 160** chars.
- Toolbar **row count** ("N of M"); **end-of-history** footer;
empty-state hint line.
### Filters / controls
- **`Last 15m`** time-range preset; **title hints** on Live/Refresh
buttons.
### Chart
- **Seconds-resolution** axis + tooltip for sub-minute buckets; tooltip
escapes the view box and ignores pointer events.
### Persistence
- **Column visibility** persists across reloads (localStorage,
comma-joined keys — no `JSON.parse`).
## Deferred (called out, not done here)
- **Filter-state persistence** — needs a `RunsFilters` schema (the repo
lints against `JSON.parse`) and is better paired with URL params for
shareability.
- **`tool:` / `code:` / `duration_ms:` filters + Tools facet** — these
need API + store changes (cross-collection tool faceting especially), so
they belong in a separate PR next to the data work.
## Verification
`oxfmt --check`, `oxlint`, `turbo typecheck` (9 packages), plugin tests
(17) all pass. UI-only; no API, store, or event-contract changes —
independent of the in-flight data/actor work.
…22) ## What Threads the run **actor** through the Cloudflare MCP session Durable Object so MCP-triggered runs carry a `{kind:"mcp", actor}` trigger — the MCP half of run actor attribution (HTTP runs already got one). ## Changes - `McpSessionInit` gains an optional `actor`; the worker dispatcher stamps it from the gate's resolved principal at session create. - The shared DO base carries `actor` onto the persisted `SessionMeta` (alongside `webOrigin`), so a cold isolate rebuilds with the same attribution. - Each host's `buildMcpServer` (`apps/host-cloudflare`) stamps the trigger, falling back to a `user` actor keyed by the session user when the principal supplied none. Generic plumbing only — it threads whatever actor the principal carries (no service-token specifics here). Pairs with the actor contract from #20. ## Verification - `typecheck` 41/41 · lint + format clean on all changed files (Build/Deploy-preview checks fail on the fork's missing R2 secrets, as on prior PRs — unrelated.)
…23) ## The bug The runs list returns an **empty-body 400** for everyone. Root cause is a schema/data-evolution bug from #20, not anything resource-related: `RunRow.actorId/actorLabel/actorKind` were `Schema.NullOr(...)` — present-but-nullable, which means **the key must exist**. Runs are stored as JSON documents, and every run written *before* #20 has a `data` doc with no such key. The store passes stored docs straight to the HTTP response encoder, which rejects the newest row with `Missing key at ["runs"][0]["actorId"]`. A response-encode failure emits a bare 400 (no body, no content-type) — the empty-body signature. Since no executions have happened since the actor store deployed, 100% of stored runs are legacy and every request fails. ## The fix Make the three actor keys optional with a **decoding default of null**: ```ts actorId: Schema.optional(Schema.NullOr(Schema.String)).pipe( Schema.withDecodingDefaultType(Effect.succeed(null)), ) ``` - **Decoded type stays `string | null`** (always present) — every reader still treats the field as required; it just defaults to `null` for pre-actor runs. - **Wire/storage tolerates an absent key** — legacy docs decode *and* encode unchanged. - Immunizes the collection against the next field added the same way. No data migration, no prod writes — deploy-forward and the page is back with all run history intact. ## Verification - New regression test round-trips a legacy-shaped doc (decode → null; **encode with the keys absent succeeds** — the exact failure). - execution-history suite 20/20, typecheck 41/41, lint + format clean. (Build/Deploy-preview checks fail on the fork's missing R2 secrets, as on prior PRs — unrelated.)
…ng columns Merge the runs table header and body into a single overflow-auto scroller with a min-width grid so the column header and rows share one horizontal scroll. Every column is now always rendered; when the viewport is narrower than the table's natural width the body scrolls horizontally instead of hiding columns at breakpoints. The code column flexes to fill on wider screens. Removes the ResizeObserver/stickyTop machinery that positioned the header, which the single scroller makes unnecessary.
Fork delta for the bun-linked self-host dogfood: move effect/@effect/atom-react/react/react-dom from dependencies to peerDependencies+devDependencies in api, app, react, hosts/mcp, hosts/cloudflare, runtime-dynamic-worker, encrypted-secrets so linked packages share a single instance of each. The fumadb scoping from the original delta is now upstream and dropped.
Add readOrgPluginStorageData: reads an org-owned plugin_storage value reusing the runtime's exact query path (withQueryContext -> makeFumaClient -> makeCoreDb -> findFirst), so host code can consult plugin-written state before a scoped executor exists. Exported via host-internal and @executor-js/api/server for the Cloudflare host's auth seam.
Derive nav items from useClientPlugins() pages that declare `nav` and render them alongside the host's own links. Makes the existing-but-unwired PageDecl.nav contract appear in the sidebar; benefits any plugin and host.
New full-stack plugin: a "Service Tokens" page to map a Cloudflare Access service token's common_name to the caller's own subject. Aliases persist in org-owned plugin_storage so they are tenant-shared; storage failures stay typed and are downgraded to InternalError at the HTTP edge via `capture`.
A service token carries only common_name (no sub), so by default it lands in its own subject partition and can't see a human's Personal connections. Resolve the service-tokens plugin's common_name->subject alias at JWT-verify time so the token ACTS AS that human (same Personal partition, admin) across the API gate, account me, and the /mcp gate. Register the plugin (runtime + client bundle). Parse Access claims via Schema and give service tokens a synthetic, non-routable email (<common_name>@service-token.internal).
The alias now stores a friendly machineName plus the aliasing user's email/name, captured natively from their Access principal at creation — so the Service Tokens page renders a human "Acts as" instead of the sub UUID, and runs can show a friendly token label. The page gains an optional machine-name field; the alias handler reads the caller's email/name from AuthContext.
The alias lookup resolves the full alias now; principalFromAccessClaims gives every service token an actor keyed by its client id (labelled by the machine name when aliased) so runs attribute to the token, and surfaces the aliased human's stored email/name as the principal identity instead of the synthetic address.
The Client ID and machine-name inputs shared one flex row, so the machine-name field (flex-1) shrank to an unusable sliver next to the wide Client ID box and read as missing. Put the Client ID on its own full-width row and pair the machine-name input with the button below.
The selfhost commits were authored without running the repo formatter (oxfmt), so the service-tokens plugin and the Cloudflare Access auth seam failed format:check. Normalize them so the stack passes the mainline merge gates.
## What Splits execution-history storage so each store fits its access pattern: - **D1 `runs` row → a slim index.** Drops the bulky, drawer-only fields (full `code`, `resultJson`, `errorText`, `logsJson`, `triggerMetaJson`) and removes the `toolCalls` / `interactions` collections. Keeps the list/aggregate fields plus two bounded denormalized fields — `codePreview` and `logErrorCount`/`logWarnCount` — so the list renders entirely from D1 with **no per-row blob fetch**. - **R2 detail object → the bulky payload.** One append-only JSON object per run (via the existing plugin blob store, keyed `run-detail/<id>`): a code-only stub on `ExecutionStarted` (so a live/paused run shows its code and survives a worker restart), the full detail on `ExecutionFinished`. Serialized via Effect Schema (`fromJsonString`), never raw JSON. - **Drawer reads it in one request.** `get` flat-merges the R2 detail onto the slim run; the separate tool-calls endpoint + atom are removed. ## Why The bulky fields are write-once, never mutated, and read only by id in the drawer — the shape object storage is built for. Moving them to R2 keeps the D1 row tiny (~hundreds of bytes), removing the D1 size ceiling as a practical concern, while the slim denormalized fields keep the list a pure D1 read. ## Breaking change + cutover The runs list/detail response shapes change and **past runs are cleared (no backfill)**: - self-host: a one-off boot data migration deletes the `executionHistory` rows. - The slim fields are `optional`-with-default, so any pre-migration row still decodes/encodes (degrades to an empty preview) rather than 400-ing the list. ## Verification - typecheck: 0 errors · lint: 0/0 · format: clean · tests: 19/19 (the store test round-trips a run's detail through the blob store). ## Follow-up A separate PR adds the first execution-history e2e scenario (run → open drawer → assert code/logs/tool-calls). The optional WAE aggregate-reader remains deferred behind a documented trigger.
Each alias card gets an inline Edit (or "Add name" when unset) action that swaps the Name row for an input with Save/Cancel. Save reuses the existing `alias` upsert, so an empty value clears the name. Enter saves, Escape cancels.
…xt (#25) ## What Rebuilds the runs list as a **real semantic `<table>` with content-fit columns** (`table-layout: auto`), replacing the bespoke fixed-width flexbox layout. - **Content-fit**: each column sizes to the maximum width its rows need — the sticky `<thead>` and `<tbody>` rows share one column model, so columns size *and* stay aligned automatically (no more shared `COL_*` fixed-width constants, no `min-w` scroll floor). - **Alignment**: short / numeric / badge columns (status, duration, tools, interaction, log) are **centered**; text columns (started, trigger, actor, code) are left-aligned. - **No blowout**: the two columns that can run long — `actor` (token id / email) and `code` — are width-capped + truncated (hover shows the full value). - The row `<Button>` becomes a `<tr role="button" tabIndex onClick onKeyDown>` (click + keyboard, accessible). The live divider / loading / empty / end-of-history states become `colSpan` rows. ## Why this shape (vs openstatus) Studied `openstatus-data-table`: it renders a real shadcn `<table>` but sets **fixed per-column `size`** values (CSS vars → inline widths) — *not* content-fit. True "snap each column to its content" comes from a real table's default `table-layout: auto`, which is what this uses. Sticky-header border rendering uses `border-separate border-spacing-0` (the same browser-bug workaround openstatus uses). ## Notes - `<table>` carries a narrow `oxlint-disable react/forbid-elements` (with a boundary reason) — the runs table needs its own infinite-scroll container + sticky thead, which the shared `<Table>`'s built-in overflow wrapper would conflict with. Same suppression the shared `Table` component itself uses. - Trade-off: content-fit means the table is its natural content width — narrower than the pane → left-aligned with slack on the right; wider → horizontal scroll. ## Verification typecheck 0 · lint 0/0 · format clean · tests 19/19.
Replace the card-stack list with a minimal 3-column table (Name, Token, Acts as) using the shared Table primitives — openstatus-flavored: muted headers, font-mono token/identity, truncation, and row actions (Edit/Add name/Remove) revealed on hover. Inline machine-name edit unchanged.
Add a neutral actor-label registry in @executor-js/react: plugins publish a (kind+id)->label source atom, and the runs UI resolves through the merged map. The service-tokens plugin publishes commonName->machineName; run-row, detail-drawer, and the actor facet prefer the resolved label over the run-time snapshot, falling back to the snapshot then the id. Resolution is live, so renaming a machine name updates past and future runs alike, with no cross-plugin import (both sides depend only on @executor-js/react).
## Summary Makes `tools.search` pluggable and adds a plugin that answers it with semantic search backed by Cloudflare Vectorize + Gemini embeddings. The plugin is opt-in: until its binding and embedding key are present the engine keeps its built-in lexical search, so nothing changes for hosts that don't enable it. ## Changes **Core — a `runtime.toolDiscoveryProvider` plugin hook** - Adds `runtime.toolDiscoveryProvider` to `PluginSpec`, mirroring `runtime.executionObserver`. `composeToolDiscoveryProviders` picks the first plugin that supplies one and threads it into the engine in `makeExecutionStack`; the built-in lexical scorer stays the fallback when none do. - Relocates the tool-discovery contract (`ToolDiscoveryProvider`/`Input`/`Result`, `PagedResult`) and `ExecutionToolError` from `@executor-js/execution` into `@executor-js/sdk`, so the plugin spec (in sdk) can name the contract without an `sdk -> execution` cycle. Both are re-exported from `@executor-js/execution/core`, so existing importers are unaffected. **New plugin `@executor-js/plugin-vectorize-search`** - `runtime.toolDiscoveryProvider`: embeds the query (Gemini via the Vercel AI SDK), nearest-neighbour queries a Vectorize index, and maps matches to `ToolDiscoveryResult` straight from stored metadata (no per-tool describe). - Indexer: embeds the tool catalog (`path`/`name`/`description`/`integration`) and upserts namespaced vectors. - `/api` subpath adds `POST /vectorize-search/reindex` (the handler yields the scoped executor, since the catalog isn't on the plugin ctx). - Binding-optional: inert until both a Vectorize binding and an embedding API key are supplied. **Cloudflare host wiring** - Adds the Vectorize binding + embedding-key plumbing to the host config and threads them into the plugin tuple. The plugin is always present (stable API shape) but inert until bound — the same opt-in-by-binding shape as the metrics plugin. ## Tests - `vitest run` — new plugin 5/5 (provider mapping/pagination/blank-query; indexer projection + empty-catalog no-op). - Full suites green for the touched core packages: sdk 341, execution 34, api 45, host-mcp 32. - `typecheck` 20/20 across the host chain; lint 0/0; format clean. ## Known tradeoffs / deferred - **Not yet exercised live** — bringing it up requires provisioning a Vectorize index and an embedding API key; there is no end-to-end run in this PR. - Single-namespace scoping (per-request multi-tenant scoping is a follow-up). - Explicit reindex only (no incremental/eventful catalog sync yet). - Pagination is sliced within Vectorize's top-K cap; deep pagination is out of scope.
…ental reindex (#27) ## What Extends the Cloudflare Vectorize `tools.search` plugin (#26) with three retrieval improvements, each behind a pluggable seam. - **Structure-aware facet chunker** — a tool is chunked by its structure (an always-present identity facet, plus optional description / input-schema / output-schema facets derived from `tools.schema`) instead of a single blind vector. Long descriptions split recursively on paragraph → sentence boundaries — no fixed windows, no truncation; schema-less tools collapse to one identity chunk. `ChunkerService` makes the chunker swappable, and `makeWholeChunker` is retained as a single-vector baseline. - **Hybrid lexical + vector search (RRF)** — because a plugin discovery provider fully supersedes the engine's built-in lexical scorer (first-provider-wins), a vector-only provider would drop lexical matching entirely. `makeHybridToolDiscoveryProvider` fuses an injected lexical provider with the vector provider via Reciprocal Rank Fusion (weights 0.7 / 0.3, k = 60). The lexical provider is a plugin option, so the plugin never has to depend on the engine package — the host passes the built-in `defaultToolDiscoveryProvider` to opt into hybrid. - **Incremental fingerprint reindex** — `reconcileToolCatalog` fingerprints each tool and diffs against a per-tool side-index in plugin storage; only new and changed tools are re-chunked and re-embedded, and a changed tool's old chunk vectors are deleted before the new ones upsert. Tools absent from the live catalog are counted and logged, not deleted — per-isolate catalog variance makes deletion unsafe in v1. Also adds `withCloudflareLimits`, a store decorator that fails loudly on Cloudflare's hard limits (64-byte ids, topK ≤ 20 with full metadata) rather than surfacing opaque Vectorize 4xx errors mid-reindex; the plugin wraps its bound store with it, and the provider requests topK 20 on the full-metadata path. ## Testing `typecheck`, `lint`, `format:check`, and the package test suite (46 tests) pass on this branch in isolation. The downstream Cloudflare host typechecks against the changed plugin API. ## Stack Base of a two-part stack. The local eval harness that tunes this offline (pluggable embedder/store, zvec, golden-set eval) is stacked on top. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#27** 👈 current 2. #28 <!-- stack:links:end -->
… peerDependencies
…eindex population (#32) Add makeD1FtsLexicalStore (async FTS5 over Cloudflare D1, sharing the schema/queries with the sync better-sqlite3/bun:sqlite adapter) and wire reconcileToolCatalog to populate an optional lexicalStore, so a Cloudflare host gets real Vectorize + FTS5/BM25 hybrid tools.search. Lexical doc ids are namespace-scoped; the lexical index is rebuilt in full per reindex (complete even when attached to an already-indexed deployment); D1 schema init is tracked via a WeakSet so a transient failure is never cached.
…rch/status API (#33) Adds the Search sidebar tab (live tools.search over the hybrid provider, index status, reindex), the ./api search+status endpoints, the shared search/status extension methods, and FtsLexicalStore.count. Greptile 5/5.
…e + sweepRemoved (#34) Splits the full-catalog reconcile into resumable, engine-agnostic primitives (page + sweep) so a durable driver can run each within its own CPU budget. Removed tools are now actually swept. Greptile 5/5.
## Summary - add an Effect KeyValueStore-backed cache primitive for derived executor artifacts - let hosts provide a durable KeyValueStore and wire Cloudflare KV through a generic optional `CACHE` binding - use that generic cache from the TypeScript schema-preview path, keyed by reachable schema roots/definitions so unused pathological definitions do not poison the preview path ## Validation - bun run --filter @executor-js/sdk test -- src/executor.test.ts - bun run --filter @executor-js/sdk typecheck - bun run --filter @executor-js/api typecheck - bun run --filter @executor-js/cloudflare typecheck - dogfood host: bun run typecheck ## Notes - `apps/host-cloudflare` typecheck still fails in this checkout because `@executor-js/plugin-semantic-search` is not resolvable in the current package graph; the same dogfood host typechecks when linked to the live package workspace.
## Summary Use the current agents MCP transport primitives for Cloudflare MCP sessions and persist resumable events in Durable Object storage. This removes the local reflection shim around private transport internals. ## Changes - Upgrade `agents` to `^0.16.2` in the Cloudflare host. - Pass `DurableObjectEventStore` into `WorkerTransport` for MCP session resumability. - Delegate request handling directly through the JSON-RPC request-id queue. ## Tests - `bun run --cwd packages/hosts/cloudflare typecheck` - `bun run --cwd packages/hosts/cloudflare test` - `bun run typecheck` - `bun run test` - `bunx oxfmt --check <changed files>` - `bunx oxlint -c .oxlintrc.jsonc <changed files> --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#38** 👈 current 2. #39 3. #40 <!-- stack:links:end -->
## Summary Add structured MCP session request logs and span attributes around the Cloudflare Durable Object request path. This makes long-lived stream behavior visible by method, headers, session state, response status, and elapsed wall time. ## Changes - Emit one structured `mcp_session_request` log per Cloudflare MCP session request. - Add request/transport/event-store attributes to MCP session spans. - Convert MCP session-mode logging to structured JSON. ## Tests - `bun run --cwd packages/hosts/cloudflare typecheck` - `bun run --cwd packages/hosts/cloudflare test` - `bun run --cwd packages/hosts/mcp typecheck` - `bun run --cwd packages/hosts/mcp test` - `bun run typecheck` - `bun run test` - `bunx oxfmt --check <changed files>` - `bunx oxlint -c .oxlintrc.jsonc <changed files> --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #38 2. **#39** 👈 current 3. #40 <!-- stack:links:end -->
## Summary Remove the legacy configurable SSE remote-client mode and use Streamable HTTP for remote MCP clients. This keeps the remote MCP path aligned with the transport used by the Cloudflare Durable Object session work. ## Changes - Remove remote MCP transport selection from public API payloads and stored config. - Drop `SSEClientTransport` from remote client connection setup. - Update e2e payloads and expectations to the flattened remote config shape. ## Tests - `bun run --cwd packages/plugins/mcp typecheck` - `bun run --cwd packages/plugins/mcp test` - `bun run --cwd e2e typecheck` - `bun run typecheck` - `bun run test` - `bunx oxfmt --check <changed files>` - `bunx oxlint -c .oxlintrc.jsonc <changed files> --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #38 2. #39 3. **#40** 👈 current <!-- stack:links:end -->
## Summary Caches fully assembled `ToolSchemaView` results in the configured `KeyValueStore`, using a content-addressed key so repeated schema views can skip preview assembly work after the raw schema data is unchanged. ## Changes - Adds a schema-view cache entry schema and deterministic key builder that includes tool metadata, input/output schemas, definitions, and TypeScript preview cache/compiler versions. - Reads and writes the schema-view cache from `executor.tools.schema` for static and persisted tools. - Adds regression coverage for cache-key invalidation and executor cache writes. ## Tests - `bun run --cwd packages/core/sdk test src/tool-schema-view-cache.test.ts src/executor.test.ts` - `bun run --cwd packages/core/sdk typecheck` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#36** 👈 current 2. #37 <!-- stack:links:end -->
## Summary Migrates the host-cloudflare `/mcp` route to the Agents `McpAgent.serve()` Streamable HTTP bridge while keeping the existing Executor MCP server construction and approval hooks. ## Changes - Adds a shared `McpAgentSessionDOBase` for Cloudflare-hosted Executor MCP sessions. - Routes host-cloudflare `/mcp` at the Worker edge so authenticated session props can flow through `ExecutionContext` into the hibernatable Durable Object. - Removes the host-cloudflare app-local `McpSessionStore` mount and its forwarded-request session-store wiring. - Updates workerd e2e coverage to accept Streamable HTTP SSE-framed JSON-RPC responses and cover initialized GET streams. ## Tests - `bun run --cwd packages/hosts/cloudflare typecheck` - `bun run --cwd apps/host-cloudflare typecheck` - `bunx oxlint packages/hosts/cloudflare/src/mcp/agent-session-durable-object.ts apps/host-cloudflare/src/mcp/agent-handler.ts apps/host-cloudflare/src/mcp/session-durable-object.ts apps/host-cloudflare/src/mcp/index.ts apps/host-cloudflare/src/worker.ts apps/host-cloudflare/src/app.ts apps/host-cloudflare/src/worker.e2e.node.test.ts --deny-warnings` - `bun run --cwd apps/host-cloudflare test src/worker.e2e.node.test.ts` - `bun run --cwd apps/cloud typecheck` - `bun run --cwd apps/host-cloudflare build`
## Summary Adds a staged, fingerprint-aware semantic-search reindex pipeline so hosts can drive durable indexing work without rebuilding unchanged tool documents. ## Changes - Adds staged index collections, job planning, and snapshot/fingerprint helpers for semantic-search indexing. - Updates reindex status/API surfaces and React atoms/page wiring for the staged pipeline. - Lets schema consumers opt out of TypeScript preview generation when they only need raw schema content for fingerprinting. ## Tests - `bun run --cwd packages/plugins/semantic-search test src/sdk/fingerprint.test.ts src/sdk/indexer.test.ts` - `bun run --cwd packages/core/sdk test src/tool-schema-view-cache.test.ts src/executor.test.ts` - `bun run --cwd packages/plugins/semantic-search typecheck` - `bun run --cwd packages/core/sdk typecheck` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #36 2. **#37** 👈 current 3. #42 4. #43 <!-- stack:links:end -->
## Summary Adds a core SDK tool schema manifest that stores cheap, stable schema fingerprints during connection refresh and exposes them through `executor.tools.manifest()`. ## Changes - Adds `tool_schema_manifest` to the core schema and public SDK type exports. - Writes manifest rows alongside tool/definition rows during connection refresh and removes stale rows when connections or integrations are removed. - Adds `tools.manifest()` with policy filtering and static tool support so indexers can diff without loading full schema views. ## Tests - `bun run test -- src/executor.test.ts` in `packages/core/sdk` - `bun run typecheck` in `packages/core/sdk` - `bunx oxlint -c .oxlintrc.jsonc packages/core/sdk/src/types.ts packages/core/sdk/src/core-schema.ts packages/core/sdk/src/executor.ts packages/core/sdk/src/executor.test.ts packages/core/sdk/src/index.ts packages/core/sdk/src/shared.ts packages/core/sdk/src/promise.ts --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #37 2. **#42** 👈 current 3. #43 <!-- stack:links:end -->
## Summary Rewires semantic-search indexing to use the core tool schema manifest as its scan/diff source instead of loading each tool schema during scan. ## Changes - Adds manifest listing/slicing for index scan inputs. - Makes `ToolSearchIndex.create`, `scan`, and removed-tool sweep consume manifest paths and fingerprints. - Keeps TypeScript schema/codegen work in the changed-tool chunking path only, and updates tests to assert scan does not perform raw schema reads. ## Tests - `bun run test -- src/sdk/tool-search-index.test.ts` in `packages/plugins/semantic-search` - `bun run typecheck` in `packages/plugins/semantic-search` - `bunx oxlint -c .oxlintrc.jsonc packages/plugins/semantic-search/src/sdk/documents.ts packages/plugins/semantic-search/src/sdk/tool-search-index.ts packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts --deny-warnings` <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #37 2. #42 3. **#43** 👈 current <!-- stack:links:end -->
Recover downstream queue messages from persisted index state when chunk or embed queue turns are retried after D1 writes have already advanced job or chunk status.
## Summary Improves the semantic-search indexing path by making derived schema caches durable through Effect `KeyValueStore`, reducing queued index materialization work, and bounding Cloudflare Vectorize metadata before upsert. ## Changes - Replace large per-isolate schema `Effect.Cache` layers with direct typed `KeyValueStore` cache reads/writes and an in-memory default when hosts do not provide cache storage. - Make queued indexing use raw schema reads plus compact schema facet text instead of TypeScript preview generation. - Cache materialized index documents by tool fingerprint and batch commit work across paths. - Cap Vectorize metadata descriptions and reject oversized metadata before calling Vectorize. ## Tests - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/core/sdk test -- src/executor.test.ts src/tool-schema-view-cache.test.ts src/tool-typescript-preview-cache.test.ts` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/plugins/semantic-search test -- src/sdk/tool-search-index.test.ts src/sdk/store-cloudflare-limits.test.ts src/sdk/chunker.test.ts` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/plugins/semantic-search typecheck` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/core/sdk typecheck` - `bun run --cwd /Users/aryasaatvik/Developer/executor-worktrees/selfhost/packages/core/api typecheck` - `bun run typecheck` in `/Users/aryasaatvik/Developer/AryaLabsHQ/executor` ## Known tradeoffs - Host queue batching is committed in the dogfood host checkout separately and is not part of this package PR. - This was not deployed and no indexing run was triggered. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. **#44** 👈 current 2. #45 <!-- stack:links:end -->
## Summary Batches semantic-search index storage operations so queue messages do fewer plugin-storage and D1 calls while preserving the existing indexing lifecycle. ## Changes - Adds typed bulk `getMany`, `getManyForOwner`, `putMany`, and `removeMany` operations to plugin storage collections. - Reworks scan, chunk, embed, and commit indexing paths to bulk-load and bulk-write jobs, chunks, and fingerprints. - Updates semantic-search test fixtures and plugin-storage tests to cover the new bulk facade methods. ## Tests - `bun run --cwd packages/core/sdk test -- src/plugin-storage.test.ts` - `bun run --cwd packages/plugins/semantic-search test -- src/sdk/tool-search-index.test.ts` - `bun run --cwd packages/core/sdk typecheck` - `bun run --cwd packages/plugins/semantic-search typecheck` - `bun run format:check` - `oxlint -c .oxlintrc.jsonc packages/core/sdk/src/executor.ts packages/core/sdk/src/plugin-storage.ts packages/core/sdk/src/plugin-storage.test.ts packages/plugins/semantic-search/src/sdk/tool-search-index.ts packages/plugins/semantic-search/src/sdk/tool-search-index.test.ts --deny-warnings` ## Known tradeoffs - Root `bun run lint` still fails on pre-existing `.scratchpad` files outside this diff. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #44 2. **#45** 👈 current <!-- stack:links:end -->
## Summary Adds a bulk upsert path through FumaDB and uses it for plugin-storage `putMany`, removing the delete-then-insert write pattern from bulk plugin storage updates. ## Changes - Add `upsertMany` to FumaDB query surfaces, with native Drizzle `ON CONFLICT DO UPDATE` support for SQLite/Postgres and memory/fallback adapter behavior. - Route SDK plugin-storage `putMany` through `core.upsertMany` using the plugin-storage unique key. - Reduce semantic-search index status reads by grouping job counts with plugin-storage aggregation and dedupe commit paths before job loads. - Cover policy-aware bulk upserts and update semantic-search test fixtures for grouped counts. ## Tests - `bun run --cwd packages/core/fumadb typecheck` - `bun run --cwd packages/core/sdk typecheck` - `bun run --cwd packages/plugins/semantic-search typecheck` - `node ../../../node_modules/vitest/vitest.mjs run src/query/table-policy.test.ts` from `packages/core/fumadb` - `bun run --cwd packages/core/sdk test src/plugin-storage.test.ts` - `bun run --cwd packages/plugins/semantic-search test src/sdk/tool-search-index.test.ts` - `./node_modules/.bin/oxlint -c .oxlintrc.jsonc <touched files> --deny-warnings`
## Summary Preserve a reindex run's `maxTools` cap across `resume`. Previously `resume` re-enqueued scan partitions with no limit, so a run started capped (e.g. a dev/debug pass) would run **uncapped** after a resume. ## Change - `create()` stores `maxTools` on the index-run record (`indexRuns` collection; optional — absent = uncapped). - `ReconcileResult` exposes `maxTools`; `reconcile()` returns `run.data.maxTools`. - A host can now forward `reconciled.maxTools` when re-enqueuing scan partitions on resume. ## Test - New: `reconcile` preserves the run's `maxTools` (capped → value, uncapped → undefined). - `tsgo --noEmit` ✓ · `oxlint --deny-warnings` (changed files) ✓ · `vitest run` 9 passed ✓ ## Follow-up The dogfood host (`AryaLabsHQ/executor`) `resume` route forwards `reconciled.maxTools` once this lands (tracked in the host backlog). Surfaced originally by a Greptile P1 on the host queue PR.
…enant namespace (#48) ## Summary `executor.semanticSearch.search` (the operator `tools.search` surface) returned **zero results for every query**, even against a fully populated vector + lexical index. Root cause: a single `namespace` value was overloaded as two different concepts. ## Root cause The search method conflated: - **`deps.namespace`** — the *tenant* namespace (e.g. `"default"`), used for Vectorize storage isolation inside `store.query`. - **`input.namespace`** — an *optional integration/path-prefix filter* the operator can pass to narrow results (e.g. `"github"`). With no operator-supplied prefix, it defaulted the prefix filter to the tenant namespace: ```ts const namespace = input.namespace ?? deps.namespace; // ← bug ``` The provider's `matchesNamespace` then required every tool's integration/path to *start with* the tenant id (`"github_api".startsWith("default")` → `false`), so it dropped every match. This affected both the vector and lexical paths, since they share `matchesNamespace`. ## Fix Apply the integration prefix only when explicitly provided. The tenant namespace drives the store query alone and is still reported on the result page. ## Verification - New regression tests at the plugin-search level: with no prefix the provider receives `undefined` (no filter) and tenant-tagged results survive; an explicit prefix still passes through. - Suite green: `vitest` (20 passed), `tsgo --noEmit`, `oxlint --deny-warnings`. - Verified live on a self-host deploy — before: `{"items":[]}` for every query; after: `"query my postgres database"` → Neon/Planetscale SQL tools, `"github"` → `github_api` tools, and `?namespace=stripe_api` still narrows correctly. ## Notes - `makeSemanticSearchExtension` is exported for the unit test but **not** re-exported from the package index, so the public API is unchanged. - Changeset included (`patch`).
## Why
Production traces showed the web UI's tool-detail path (`GET
/api/tools/schema`, `includeTypeScript: true`) paying the json-schema →
TypeScript compile on the first view of every tool. The reindex pipeline
already visits every tool's schema, but it called `tools.schema({
includeTypeScript: false })` specifically to *skip* that compile — so it
warmed the wrong cache variant and the UI's variant stayed cold until
first manual view.
## What
- `executor.tools.schema` always renders the TypeScript preview; the
`includeTypeScript` option and the now-empty `ToolSchemaOptions` type
are removed.
- The reindex pipeline (`collectDocForTool`) calls `tools.schema`
plainly, so every reindex pre-warms exactly the KV schema-view the UI
reads → first tool-detail views become cache hits instead of a cold
compile.
- `includeTypeScript` is dropped from the schema-view cache key (content
+ manifest), collapsing the two variants into one.
## Safety
- Embedding/lexical text is unchanged: `collectDocForTool` only reads
the raw schema facets (`inputSchema` / `outputSchema` /
`schemaDefinitions`), never the rendered TypeScript — so no re-embedding
/ index churn.
- The cache-key change supersedes existing entries; they re-warm on the
next reindex or view (regenerable derived data).
## Gate
tsgo (43/43), `oxlint --deny-warnings`, oxfmt, and vitest for
`@executor-js/sdk` + `@executor-js/plugin-semantic-search` all green.
Follow-up: a reindex on the deployed instance warms the new unified
variant.
<!-- stack:links:start -->
### [Stack](https://github.com/kitlangton/stack)
1. **#49** 👈 current
2. #50
<!-- stack:links:end -->
**Phase 1** of Tier-2 tool-catalog caching — stacked on #49. Caches `executor.tools.list` (today a full `tool` scan + `tool_policy` read on every call) behind a **content-derived revision key**, so the list stays warm and only re-reads D1 when the catalog or policies actually change. No TTL. ## Invalidation - `produceConnectionTools` writes a per-connection **`toolset_revision`** = `sha256` of the connection's manifest rows' `(path, index_fingerprint)`, in **`plugin_storage`** (owner-scoped, **no schema migration**), at all 5 catalog-mutation sites (produce, both clear branches, connection-remove, integration-remove). - `catalogGen` combines the in-scope connections' revisions; `policyGen` combines the `tool_policy` rows. Key = `sha256(owner, integration, connection, includeBlocked, catalogGen, policyGen)`. - **Token rotation does not bust** (never calls `produceConnectionTools`); add/remove/change a tool, or any policy edit, does. ## Scope - Cached value = the DB-derived, policy-filtered list. **Static tools unioned live** (never cached); **`query` applied post-cache** so distinct queries share one cached set. - Hand-authored `ToolListItem` Schema (list projection only — no input/output schema); branded ids stored as strings, rebuilt on read. ## Notes (intentional, single-tenant self-host) - `catalogGen`/`policyGen` reads are tenant-wide → per-tenant-coarse invalidation (a catalog change re-warms all owners' list caches). Correct, just coarse. ## Gate tsgo (sdk) · `oxlint --deny-warnings` · oxfmt · vitest **sdk 350/350** (new cache suite: add/remove/content invalidation, policy bust, query post-cache, static-union-on-hit). Phase 2 (stacked) moves the UI list onto the compact manifest tier. <!-- stack:links:start --> ### [Stack](https://github.com/kitlangton/stack) 1. #49 2. **#50** 👈 current <!-- stack:links:end -->
…51) ## Summary Introduces a shared windowed-list primitive (`@executor-js/react/components/virtual-list`) and adopts it on the web-UI surfaces that mount unbounded or long lists, so they keep a bounded DOM instead of rendering every row at once. This is **M1 of a 2-PR stack**. The follow-up adds server-side cursor pagination to `tools.list` — the one list whose API still returns the whole set — and stacks on top of this branch. ## What's here (one commit per surface) - **VirtualList primitive** — `@tanstack/react-virtual` with dynamic measurement, an `onEndReached` infinite-scroll trigger (generalized from the runs shell's hand-rolled distance-from-bottom check), optional sticky section headers, and scroll restoration keyed by id. A `useWindowedRows` hook covers surfaces that can't use `<div>` rows (real `<table>`s). - **Tool source-list** (worst case) — a broad source could mount thousands of tool rows at once, each carrying a dropdown-menu portal. The grouped account sections and dotted-name trees are flattened into one windowed list; section headers become sticky virtual items; only near-viewport rows mount. - **Runs table** — windowed via `useWindowedRows`; the real `<table>` + sticky `<thead>` stay, with top/bottom spacer `<tr>`s for off-screen rows. The two variable-width columns are already capped, so windowing a subset doesn't make `table-layout: auto` jitter. - **Run-detail drawer** — the logs and tool-calls tabs get their own windowed scroll containers; tool-call expanded state is lifted so it survives a row scrolling out of the window and back. - **Sticky-header range logic** extracted to a pure function with unit tests. ## Deliberately not virtualized The integrations catalog, the service-tokens table, and the org-members list are **page-scrolled** lists that hold dozens of rows in practice. Virtualizing them would mean introducing an independent bounded scroller — a UX change for negligible benefit. The primitive is ready if any ever grows unbounded. ## Gates - `typecheck`: 43/43 packages - tests: shared React package 162, execution-history 19 (including new windowing-logic unit tests) - `oxfmt` + `oxlint --deny-warnings` clean on all changed files ## Verification still to do (live browser) Unit tests cover the windowing math; the visual behaviour needs a real browser (the e2e browser surface is cloud-only). Worth exercising by hand: - Sticky section headers pin and swap correctly while scrolling a grouped tool list. - Runs-table column widths stay stable while scrolling deep into the keyset list. - Detail-drawer logs / tool-calls scroll and measure correctly; expanded tool calls keep their state across scroll.
…an) (#52) ## Why Production traces showed the reindex **scan** phase dominating wall-time: ~30 min for an all-skip pass over ~5,400 tools, ~3 tools/s. The cause is read amplification — `scan` calls `tools.manifest()` (a full `tool_schema_manifest` table read) **once per partition page**, then filters to its partition and slices `SCAN_PAGE` rows. With `PARTITION_COUNT=12` and `SCAN_PAGE=25` that's ~216 scan messages, each re-reading all ~5,400 rows cross-region (SIN→NRT): `O(N²/pageLimit)` ≈ **1.17M row-transfers** to scan 5,400 tools. `partition` is computed in-app (`hash(path) % count`), so it can't be pushed down to SQL without a migration. ## What - **`@executor-js/sdk`** — `createExecutor` exposes its cache `KeyValueStore` on the instance as **`executor.cache`** (Cloudflare KV in prod, in-memory fallback otherwise). Plugins reach durable cache storage without re-threading a binding; wrap with `KeyValueStore.toSchemaStore` like the internal `tools.list` / schema-view caches. - **`@executor-js/plugin-semantic-search`** — `create` reads the manifest **once**, partitions it (preserving each entry's full-list ordinal), and writes **one snapshot per partition** to `executor.cache` (`index-manifest/{runId}/{partition}`). `scan` reads only its partition's snapshot — **KV-only, no D1 fallback on the hot path**. A miss fails the message so the queue retries; the host delays the *initial* scan fan-out once to let the write propagate (separate host PR). Snapshots are deleted on run completion. ## Result - D1 manifest reads: `~1.17M → ~5,400` (a single pass at `create`). - Per-scan-message memory: full manifest (~5 MB) → one partition (~400 KB). - Scan wall-time: **~30 min → seconds–minutes** (and the host delay PR keeps the fast path certain). ## Safety / notes - KV-only by design: a snapshot miss surfaces loudly (queue retry / DLQ) rather than silently degrading to the slow D1 path. - Snapshots are partition-sharded so no single KV value approaches the 25 MB cap (scales to ~300k tools). - KV has no native TTL through the abstraction, so cleanup is explicit on completion; a stalled / terminally-failed run leaves its (small) snapshot until overwritten — a known, bounded leak. - Behavior preserved: the scan's skip-vs-changed classification, ordinal ordering, pagination, and `maxTools` cap are unchanged (the snapshot is built from the same `listToolManifests` call). ## Gate tsgo **43/43** · `oxlint --deny-warnings` 0/0 · oxfmt · vitest **sdk 350/350**, **plugin-semantic-search 78/78** (new: manifest-read-once-per-create, KV-miss-fails-no-fallback, snapshot-cleared-on-complete). Stacked deploy: needs the host PR that delays the initial scan fan-out for KV propagation.
# Conflicts: # apps/host-cloudflare/package.json # apps/host-cloudflare/src/app.ts # apps/host-cloudflare/src/auth/cloudflare-access.ts # apps/host-cloudflare/src/mcp/index.ts # apps/host-cloudflare/src/mcp/session-durable-object.ts # apps/host-selfhost/src/db/data-migrations.ts # bun.lock # packages/core/api/src/server/mcp-build.ts # packages/core/execution/src/engine.ts # packages/core/sdk/src/executor.ts # packages/hosts/cloudflare/src/mcp/session-durable-object.ts # packages/plugins/mcp/src/sdk/connection.ts
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Owner
Author
|
Review artifact superseded: dev has now been force-updated to the rebased tree at 7f22595. This merge-preview PR was only used to review how the fork changed from old dev; its final tree matched the rebased branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Review-only merge preview for syncing current upstream/main into dev. This PR is not intended to be merged; it shows the upstream integration and conflict-resolution surface in GitHub's normal PR UI.
The actual landing path remains force-updating dev to the rebased branch sync/upstream-main-2026-06-22.
Important invariant
The merge-preview commit tree is identical to the refreshed rebased branch tree:
Conflict surface
The merge commit records conflicts in host Cloudflare/selfhost MCP integration, core API MCP build plumbing, execution engine imports, SDK executor wiring, MCP connection typing, and bun.lock. The final tree was resolved to match the already-rebased branch exactly.
Verification
Note: full repo format/lint scripts currently scan .scratchpad and fail on existing scratchpad-only files, so the format/lint checks above were run against the committed diff.