Skip to content

fix(plugins): in-process sandbox fallback when child-process fork is restricted - #95

Open
HareeshDaxton wants to merge 4 commits into
vencorehq:mainfrom
HareeshDaxton:feat/plugin-sandbox-in-process-fallback
Open

fix(plugins): in-process sandbox fallback when child-process fork is restricted#95
HareeshDaxton wants to merge 4 commits into
vencorehq:mainfrom
HareeshDaxton:feat/plugin-sandbox-in-process-fallback

Conversation

@HareeshDaxton

Copy link
Copy Markdown

Closes #93 (B-0018).

Problem

Vencore runs each plugin backend in an isolated forked child process. On hosts that restrict child-process creation, fork() fails: the plugin never mounts its router (404 PLUGIN_NOT_MOUNTED / "Plugin not found") and a synchronous fork throw during load surfaces as a 500.

What changed

An in-process execution strategy behind a strategy dispatcher, so no caller changes:

  • facade.ts — shared vencore facade builder parameterized by a transport, so the plugin-facing API is byte-for-byte identical in both modes and cannot drift.
  • runner.ts — builds its facade from the shared builder over an IPC transport (child behavior unchanged).
  • child-process.ts — existing fork logic extracted; fork sync-throws and pre-ready child error/exit funnel into an onForkUnavailable hook (this also closes the 500).
  • in-process.ts — runs the bundle via require (cache-busted), calls dispatchBridgeCall directly with envelope unwrap, mounts HTTP routes locally, forwards declared listen topics from the workspace bus.
  • manager.ts — thin dispatcher. PLUGIN_SANDBOX_MODE (auto | child | in-process, default auto) selects the strategy. In auto, the first detected fork failure latches into in-process for all plugins.
  • index.ts — startup log names the active mode; SIGTERM/SIGINT now call killAllSandboxes (previously dead code) so in-process bus subscriptions don't leak.

auto keeps the isolated child process wherever forking works, so the reduced isolation of in-process only applies on hosts that force it.

A second defect, found while testing

Reproducing the issue against a real fork-restricted container (exhausted pids cgroup) showed the fallback above was not sufficient.

It only fires when fork fails loudly — a synchronous fork() throw, a child error, or an exit before ready. When the host limit is a process/thread budget rather than an outright block, fork() succeeds: the child process exists but can never finish booting. It emits no error, never exits, and never signals ready. None of the three triggers fire, the spawn hangs forever, and the plugin stays unmounted — the exact 404 this PR set out to prevent.

The final commit adds a readiness deadline. A child that misses it is SIGKILLed (a process wedged before boot may never run a SIGTERM handler) and reported as fork-unavailable, so auto latches into in-process as it does elsewhere. The timer is unref'd and cleared on both ready and exit. Configurable via PLUGIN_SANDBOX_READY_TIMEOUT_MS (default 15s).

This is the failure mode a restricted host is most likely to produce — budgets exhaust silently, they don't throw EPERM.

Verification

End-to-end against a live stack (Docker Compose: TimescaleDB + Redis + API), with a seeded test plugin and the API container's process budget starved via pids_limit: 15:

Scenario Result
Fork available (baseline) count: 1, child-process sandbox mounts
Fork restricted, before readiness fix Hangs silently, plugin unmounted — bug reproduced
Fork restricted, after readiness fix Falls back, plugin mounts — bug fixed

Log from the restricted container after the fix:

WARN  Plugin child-process fork unavailable
        reason: "child did not signal ready within 15000ms"
WARN  Child-process fork unavailable — falling back to in-process plugin sandbox for all plugins
INFO  Plugin in-process sandbox setup complete
WARN  Plugin sandbox process exited          <- wedged child reaped, PIDs 15->14

Also verified: both modes honored in the deployed API (pluginSandboxMode: "auto" / "in-process"), tsc clean across packages/* and apps/api, web app renders.

28 automated tests, all passing:

  • plugin-sandbox-fallback.test.ts (12) — mode selection, the fork-unavailable fallback and its latch across plugins, crash respawn routing back through mode selection, strategy-agnostic lookups.
  • plugin-sandbox-in-process.test.ts (10) — the real in-process strategy against real bundles over a real HTTP server: routes serve, bodies pass through, a throwing handler yields a contained 500, bridge envelope unwrapping, bus/cron delivery, require-cache busting on upgrade, teardown.
  • plugin-sandbox-ready-timeout.test.ts (6) — the wedged-child case (stubbed fork + fake timers, since a unit test cannot exhaust a cgroup), including that it does not trigger the 5s crash-restart into the same dead end.

Reviewer notes

  • Security trade-off. In-process mode removes OS process isolation — plugin code shares the API process and can read process.env and process globals directly. The facade still never hands env/secrets to the plugin, and auto limits this to hosts that force it, but this matters if you run untrusted third-party plugins. Pin PLUGIN_SANDBOX_MODE=child to refuse the trade entirely (plugins stay down rather than lose isolation).
  • The 15s default is a judgment call. A genuinely slow host could exceed it and fall back unnecessarily, trading isolation for availability. That is the safer direction, but worth a look for your deployment.
  • Both new env vars are documented in .env.example and validated in apiEnvSchema.

Out of scope / not done

  • The full API test suite was not run — only the three sandbox files. There is no Node toolchain on the dev machine used here, so tests ran in a container with vitest + express only.
  • No authenticated HTTP request was made to a plugin route in the live app (first-run setup was not completed). Router mounting is evidenced by the live log line, which fires only after entry.router is assigned, and by the integration tests serving real plugin routes.
  • The worker service was not exercised.

HareeshDaxton and others added 4 commits July 27, 2026 11:35
… restricted

Vencore runs each plugin backend in an isolated forked child process. On hosts
that restrict child-process creation, fork() fails: the plugin never mounts its
router (404 PLUGIN_NOT_MOUNTED / "Plugin not found") and a synchronous fork throw
during load surfaces as a 500. (GH vencorehq#93 / B-0018)

Add an in-process execution strategy behind a strategy dispatcher so no caller
changes:

- facade.ts: shared vencore facade builder parameterized by a transport, so the
  plugin-facing API is identical in both modes.
- runner.ts: builds its facade from the shared builder over an IPC transport
  (child behavior unchanged).
- child-process.ts: existing fork logic extracted; fork sync-throws and pre-ready
  child error/exit now funnel into an onForkUnavailable hook (also closes the 500).
- in-process.ts: runs the bundle via require (cache-busted), calls
  dispatchBridgeCall directly with envelope unwrap, mounts HTTP routes locally,
  and forwards declared listen topics from the workspace bus.
- manager.ts: thin dispatcher; PLUGIN_SANDBOX_MODE (auto|child|in-process, default
  auto) selects the strategy. In auto, the first detected fork failure latches
  into in-process for all plugins.
- index.ts: startup log names the active mode; SIGTERM/SIGINT now call
  killAllSandboxes (previously dead code) so in-process bus subs don't leak.
- config: add PLUGIN_SANDBOX_MODE to apiEnvSchema for validation + docs.

auto keeps the isolated child process wherever forking works, so the reduced
isolation of in-process only applies on hosts that force it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…DBOX_MODE

Follow-up to 33e498b (GH vencorehq#93 / B-0018).

- Add plugin-sandbox-fallback.test.ts covering mode selection (auto/child/
  in-process + unrecognised values), the fork-unavailable fallback and its
  latch across plugins, crash respawn going back through mode selection, and
  the strategy-agnostic router/bus/isRunning/killAll lookups.
- .env.example: document PLUGIN_SANDBOX_MODE. It was added to apiEnvSchema but
  never surfaced to operators, so the in-process escape hatch was undiscoverable
  on hosts that block fork outright.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dispatcher tests mock both strategies, so they only prove the wiring. This
exercises the real in-process strategy against real plugin bundles over a real
HTTP server — the actual demonstration that GH vencorehq#93 / B-0018 is fixed:

- HTTP: routes mount and serve (the reported PLUGIN_NOT_MOUNTED symptom), request
  bodies reach the handler, and a throwing handler yields a contained 500 that
  does not leak the plugin's internal message.
- Bridge: the { data, error } envelope is unwrapped so plugins receive values,
  and a bridge error rejects inside the plugin.
- Lifecycle: declared listen topics forward from the workspace bus, cron pokes
  reach handlers registered via vencore.cron, the require cache is busted on
  respawn so upgrades take effect, kill unsubscribes bus handlers, and a bundle
  whose setup throws leaves no router mounted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found while reproducing GH vencorehq#93 against a real fork-restricted container
(pids_limit exhausted): the fallback added in 33e498b only fires when fork
fails loudly — a synchronous fork() throw, a child 'error', or an exit before
ready. When the host limit is a process/thread budget rather than an outright
block, fork() SUCCEEDS and the child exists but can never finish booting. It
emits no error, never exits, and never signals ready, so none of the three
triggers fire, the spawn hangs forever and the plugin stays unmounted — the
exact 404 PLUGIN_NOT_MOUNTED the fallback was meant to prevent.

Add a readiness deadline: a child that misses it is SIGKILLed (a process wedged
before boot may never run a SIGTERM handler) and reported as fork unavailable,
so `auto` latches into the in-process strategy as it already does elsewhere.
The timer is unref'd and cleared on both ready and exit. Configurable via
PLUGIN_SANDBOX_READY_TIMEOUT_MS (default 15s), documented in .env.example and
validated in apiEnvSchema.

Verified end to end: with the API container's pids budget exhausted, the plugin
previously hung with no log at all; it now logs "child did not signal ready
within 15000ms", falls back, and reaches "Plugin in-process sandbox setup
complete" with the wedged child reaped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@HareeshDaxton
HareeshDaxton force-pushed the feat/plugin-sandbox-in-process-fallback branch from faeb27a to 2b1f0bf Compare July 27, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Needs in-process sandbox fallback support when child process fork is restricted on production

1 participant