fix(plugins): in-process sandbox fallback when child-process fork is restricted - #95
Open
HareeshDaxton wants to merge 4 commits into
Open
Conversation
… restricted Vencore runs each plugin backend in an isolated forked child process. On hosts that restrict child-process creation, fork() fails: the plugin never mounts its router (404 PLUGIN_NOT_MOUNTED / "Plugin not found") and a synchronous fork throw during load surfaces as a 500. (GH vencorehq#93 / B-0018) Add an in-process execution strategy behind a strategy dispatcher so no caller changes: - facade.ts: shared vencore facade builder parameterized by a transport, so the plugin-facing API is identical in both modes. - runner.ts: builds its facade from the shared builder over an IPC transport (child behavior unchanged). - child-process.ts: existing fork logic extracted; fork sync-throws and pre-ready child error/exit now funnel into an onForkUnavailable hook (also closes the 500). - in-process.ts: runs the bundle via require (cache-busted), calls dispatchBridgeCall directly with envelope unwrap, mounts HTTP routes locally, and forwards declared listen topics from the workspace bus. - manager.ts: thin dispatcher; PLUGIN_SANDBOX_MODE (auto|child|in-process, default auto) selects the strategy. In auto, the first detected fork failure latches into in-process for all plugins. - index.ts: startup log names the active mode; SIGTERM/SIGINT now call killAllSandboxes (previously dead code) so in-process bus subs don't leak. - config: add PLUGIN_SANDBOX_MODE to apiEnvSchema for validation + docs. auto keeps the isolated child process wherever forking works, so the reduced isolation of in-process only applies on hosts that force it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…DBOX_MODE Follow-up to 33e498b (GH vencorehq#93 / B-0018). - Add plugin-sandbox-fallback.test.ts covering mode selection (auto/child/ in-process + unrecognised values), the fork-unavailable fallback and its latch across plugins, crash respawn going back through mode selection, and the strategy-agnostic router/bus/isRunning/killAll lookups. - .env.example: document PLUGIN_SANDBOX_MODE. It was added to apiEnvSchema but never surfaced to operators, so the in-process escape hatch was undiscoverable on hosts that block fork outright. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dispatcher tests mock both strategies, so they only prove the wiring. This exercises the real in-process strategy against real plugin bundles over a real HTTP server — the actual demonstration that GH vencorehq#93 / B-0018 is fixed: - HTTP: routes mount and serve (the reported PLUGIN_NOT_MOUNTED symptom), request bodies reach the handler, and a throwing handler yields a contained 500 that does not leak the plugin's internal message. - Bridge: the { data, error } envelope is unwrapped so plugins receive values, and a bridge error rejects inside the plugin. - Lifecycle: declared listen topics forward from the workspace bus, cron pokes reach handlers registered via vencore.cron, the require cache is busted on respawn so upgrades take effect, kill unsubscribes bus handlers, and a bundle whose setup throws leaves no router mounted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found while reproducing GH vencorehq#93 against a real fork-restricted container (pids_limit exhausted): the fallback added in 33e498b only fires when fork fails loudly — a synchronous fork() throw, a child 'error', or an exit before ready. When the host limit is a process/thread budget rather than an outright block, fork() SUCCEEDS and the child exists but can never finish booting. It emits no error, never exits, and never signals ready, so none of the three triggers fire, the spawn hangs forever and the plugin stays unmounted — the exact 404 PLUGIN_NOT_MOUNTED the fallback was meant to prevent. Add a readiness deadline: a child that misses it is SIGKILLed (a process wedged before boot may never run a SIGTERM handler) and reported as fork unavailable, so `auto` latches into the in-process strategy as it already does elsewhere. The timer is unref'd and cleared on both ready and exit. Configurable via PLUGIN_SANDBOX_READY_TIMEOUT_MS (default 15s), documented in .env.example and validated in apiEnvSchema. Verified end to end: with the API container's pids budget exhausted, the plugin previously hung with no log at all; it now logs "child did not signal ready within 15000ms", falls back, and reaches "Plugin in-process sandbox setup complete" with the wedged child reaped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HareeshDaxton
force-pushed
the
feat/plugin-sandbox-in-process-fallback
branch
from
July 27, 2026 06:05
faeb27a to
2b1f0bf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #93 (B-0018).
Problem
Vencore runs each plugin backend in an isolated forked child process. On hosts that restrict child-process creation,
fork()fails: the plugin never mounts its router (404PLUGIN_NOT_MOUNTED/ "Plugin not found") and a synchronous fork throw during load surfaces as a 500.What changed
An in-process execution strategy behind a strategy dispatcher, so no caller changes:
facade.ts— sharedvencorefacade builder parameterized by a transport, so the plugin-facing API is byte-for-byte identical in both modes and cannot drift.runner.ts— builds its facade from the shared builder over an IPC transport (child behavior unchanged).child-process.ts— existing fork logic extracted; fork sync-throws and pre-ready child error/exit funnel into anonForkUnavailablehook (this also closes the 500).in-process.ts— runs the bundle viarequire(cache-busted), callsdispatchBridgeCalldirectly with envelope unwrap, mounts HTTP routes locally, forwards declared listen topics from the workspace bus.manager.ts— thin dispatcher.PLUGIN_SANDBOX_MODE(auto|child|in-process, defaultauto) selects the strategy. Inauto, the first detected fork failure latches into in-process for all plugins.index.ts— startup log names the active mode; SIGTERM/SIGINT now callkillAllSandboxes(previously dead code) so in-process bus subscriptions don't leak.autokeeps the isolated child process wherever forking works, so the reduced isolation of in-process only applies on hosts that force it.A second defect, found while testing
Reproducing the issue against a real fork-restricted container (exhausted
pidscgroup) showed the fallback above was not sufficient.It only fires when fork fails loudly — a synchronous
fork()throw, a childerror, or an exit before ready. When the host limit is a process/thread budget rather than an outright block,fork()succeeds: the child process exists but can never finish booting. It emits no error, never exits, and never signalsready. None of the three triggers fire, the spawn hangs forever, and the plugin stays unmounted — the exact 404 this PR set out to prevent.The final commit adds a readiness deadline. A child that misses it is SIGKILLed (a process wedged before boot may never run a SIGTERM handler) and reported as fork-unavailable, so
autolatches into in-process as it does elsewhere. The timer isunref'd and cleared on bothreadyandexit. Configurable viaPLUGIN_SANDBOX_READY_TIMEOUT_MS(default 15s).This is the failure mode a restricted host is most likely to produce — budgets exhaust silently, they don't throw
EPERM.Verification
End-to-end against a live stack (Docker Compose: TimescaleDB + Redis + API), with a seeded test plugin and the API container's process budget starved via
pids_limit: 15:count: 1, child-process sandbox mountsLog from the restricted container after the fix:
Also verified: both modes honored in the deployed API (
pluginSandboxMode: "auto"/"in-process"),tscclean acrosspackages/*andapps/api, web app renders.28 automated tests, all passing:
plugin-sandbox-fallback.test.ts(12) — mode selection, the fork-unavailable fallback and its latch across plugins, crash respawn routing back through mode selection, strategy-agnostic lookups.plugin-sandbox-in-process.test.ts(10) — the real in-process strategy against real bundles over a real HTTP server: routes serve, bodies pass through, a throwing handler yields a contained 500, bridge envelope unwrapping, bus/cron delivery, require-cache busting on upgrade, teardown.plugin-sandbox-ready-timeout.test.ts(6) — the wedged-child case (stubbedfork+ fake timers, since a unit test cannot exhaust a cgroup), including that it does not trigger the 5s crash-restart into the same dead end.Reviewer notes
process.envand process globals directly. The facade still never hands env/secrets to the plugin, andautolimits this to hosts that force it, but this matters if you run untrusted third-party plugins. PinPLUGIN_SANDBOX_MODE=childto refuse the trade entirely (plugins stay down rather than lose isolation)..env.exampleand validated inapiEnvSchema.Out of scope / not done
vitest+expressonly.entry.routeris assigned, and by the integration tests serving real plugin routes.workerservice was not exercised.