Repository navigation
fix(nsjail): make the direct-NsJail path actually deployable - #1
Merged
Merged
Conversation
The KVM/microVM path works out of the box; the direct-NsJail path does not.
Five defects kept it from starting on a non-metal host (VI-548):
1. api/Dockerfile — the sandbox-runner target left /pkgs empty and expected the
language runtimes to arrive through the /host-packages bind mount, so every
host had to ship or compile several GB of Python/Node/Bun. Bake the rendered
/pkgs tree into the rootfs, mirroring sandbox-runner-baked.
2. start-direct-sandbox.sh — the first bind replaces /usr/sbin, and on a Debian
rootfs mount lives in /usr/bin only, so the shell lost its mount binary and
every later bind died with "No such file or directory". Stage a copy under
/run before entering the mount namespace.
3. api/Dockerfile — the runner base is Fedora, where /usr/sbin is a symlink to
bin. The /usr/sbin and /usr/bin binds therefore collided: the second
overwrote the first and nsjail (which ships in the rootfs sbin) disappeared,
so the entrypoint smoke test failed with an empty log. Replace the symlink
with a real directory so both binds land where upstream intends.
4. docker-compose.fifthseason.yml — /usr/local/bin/sandbox-runner-healthcheck.sh
is hidden by the /usr/local bind and the image ships no curl or wget, so the
healthcheck could never pass and the worker never started. Probe the sandbox
API with bash builtins instead.
5. start-direct-sandbox.sh — the Dockerfile creates /host-packages empty, so the
bare -d test always passed and the mount shadowed the baked /pkgs, leaving
the sandbox with no runtimes ("python-3.14.4 runtime is unknown"). Only bind
the host mount when it actually holds packages.
Also adds docker-compose.fifthseason.yml: upstream's compose always maps
/dev/kvm and Compose merges `devices` by appending, so an override cannot remove
it and the stack cannot start on an instance without KVM.
Verified end to end on the patched image: JWT auth (valid key 200, wrong key
401), pandas execution, and PNG/CSV generation returned as session files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Our fork's first change. Upstream's microVM (KVM) path works out of the box; the direct-NsJail path does not — it cannot start at all on a host without
/dev/kvm. Since a non-metal EC2 instance is exactly that, this branch makes that path deployable.Five defects, four of them upstream. Each is commented inline at the site.
sandbox-runnerleft/pkgsempty and expected the language runtimes via the/host-packagesbind, so every host had to ship or compile several GB of Python/Node/Bun/pkgstree into the rootfs, mirroringsandbox-runner-baked/usr/sbin; the sandbox rootfs is Debian and shipsmountonly in/usr/bin, so the shell lost itsmountbinary and every later bind died with No such file or directory/runbefore entering the mount namespace/usr/sbinis a symlink tobin. The/usr/sbinand/usr/binbinds therefore collided — the second overwrote the first andnsjail(which ships in the rootfssbin) disappeared, so the entrypoint smoke test failed with an empty log/usr/local/bin/sandbox-runner-healthcheck.shis hidden by the/usr/localbind, and the image ships nocurlorwget, so the healthcheck could never pass and the worker never started/dev/tcp)/host-packagesempty, so the bare-dtest always passed and the mount shadowed the baked/pkgs, leaving the sandbox with no runtimes ("python-3.14.4 runtime is unknown")LibreChat-AI#3 was the one that blocked everything: it is why the smoke test failed with no diagnostic output —
nsjailwas simply gone by the time it ran.Also adds
docker-compose.fifthseason.yml. Upstream's compose always maps/dev/kvm, and Compose mergesdevicesby appending, so an override file cannot remove it — the mapping has to be absent from the file itself or the stack cannot start on an instance without KVM. Images are parameterised so the same file works locally and against ECR.All five fixes are backward compatible: host-mount deployments keep working, and nothing in the KVM path is touched.
Test plan
Built
--target sandbox-runner-falseand ran the full stack withKVM_ENABLED=false:Then, with JWT auth enabled (
LOCAL_MODE=false,CODEAPI_AUTH_PROVIDER=librechat-jwt):401 {"error":"Bearer token is required"}401 {"error":"Invalid bearer token"}200, stdoutpython 3.14.4 / total: 105 / PROD OK,files: [prod.csv]Verified twice: locally, and on the deployed EC2 instance pulling these images from ECR.
Upstreaming
Fixes 1–3 and 5 are defects in
ClickHouse/code-interpreteritself, not fork-specific choices — worth sending upstream once they have settled here. Fix 4 lives in our compose file, so it is ours to keep.