Skip to content

fix(nsjail): make the direct-NsJail path actually deployable - #1

Merged
dboichuk-2 merged 1 commit into
mainfrom
vi548-nsjail-deployable
Jul 31, 2026
Merged

dboichuk-2 merged 1 commit into
mainfrom
vi548-nsjail-deployable

Conversation

@dboichuk-2

Copy link
Copy Markdown
Collaborator

Summary

Our fork's first change. Upstream's microVM (KVM) path works out of the box; the direct-NsJail path does not — it cannot start at all on a host without /dev/kvm. Since a non-metal EC2 instance is exactly that, this branch makes that path deployable.

Five defects, four of them upstream. Each is commented inline at the site.

# Defect Fix
1 sandbox-runner left /pkgs empty and expected the language runtimes via the /host-packages bind, so every host had to ship or compile several GB of Python/Node/Bun Bake the rendered /pkgs tree into the rootfs, mirroring sandbox-runner-baked
2 The first bind replaces /usr/sbin; the sandbox rootfs is Debian and ships mount only in /usr/bin, so the shell lost its mount binary and every later bind died with No such file or directory Stage a copy under /run before entering the mount namespace
3 The runner base is Fedora, where /usr/sbin is a symlink to bin. The /usr/sbin and /usr/bin binds therefore collided — the second overwrote the first and nsjail (which ships in the rootfs sbin) disappeared, so the entrypoint smoke test failed with an empty log Replace the symlink with a real directory so both binds land where upstream intends
4 /usr/local/bin/sandbox-runner-healthcheck.sh is hidden by the /usr/local bind, and the image ships no curl or wget, so the healthcheck could never pass and the worker never started Probe the sandbox API with bash builtins (/dev/tcp)
5 The Dockerfile creates /host-packages empty, so the bare -d test always passed and the mount shadowed the baked /pkgs, leaving the sandbox with no runtimes ("python-3.14.4 runtime is unknown") Only bind the host mount when it actually holds packages

LibreChat-AI#3 was the one that blocked everything: it is why the smoke test failed with no diagnostic output — nsjail was simply gone by the time it ran.

Also adds docker-compose.fifthseason.yml. Upstream's compose always maps /dev/kvm, and Compose merges devices by appending, so an override file cannot remove it — the mapping has to be absent from the file itself or the stack cannot start on an instance without KVM. Images are parameterised so the same file works locally and against ECR.

All five fixes are backward compatible: host-mount deployments keep working, and nothing in the KVM path is touched.

Test plan

Built --target sandbox-runner-false and ran the full stack with KVM_ENABLED=false:

NsJail smoke test passed
Sandbox API started (0.0.0.0:2000)

Then, with JWT auth enabled (LOCAL_MODE=false, CODEAPI_AUTH_PROVIDER=librechat-jwt):

  • no token → 401 {"error":"Bearer token is required"}
  • JWT signed with a different key → 401 {"error":"Invalid bearer token"}
  • JWT signed with the right key → 200, stdout python 3.14.4 / total: 105 / PROD OK, files: [prod.csv]

Verified twice: locally, and on the deployed EC2 instance pulling these images from ECR.

Upstreaming

Fixes 1–3 and 5 are defects in ClickHouse/code-interpreter itself, not fork-specific choices — worth sending upstream once they have settled here. Fix 4 lives in our compose file, so it is ours to keep.

The KVM/microVM path works out of the box; the direct-NsJail path does not.
Five defects kept it from starting on a non-metal host (VI-548):

1. api/Dockerfile — the sandbox-runner target left /pkgs empty and expected the
   language runtimes to arrive through the /host-packages bind mount, so every
   host had to ship or compile several GB of Python/Node/Bun. Bake the rendered
   /pkgs tree into the rootfs, mirroring sandbox-runner-baked.

2. start-direct-sandbox.sh — the first bind replaces /usr/sbin, and on a Debian
   rootfs mount lives in /usr/bin only, so the shell lost its mount binary and
   every later bind died with "No such file or directory". Stage a copy under
   /run before entering the mount namespace.

3. api/Dockerfile — the runner base is Fedora, where /usr/sbin is a symlink to
   bin. The /usr/sbin and /usr/bin binds therefore collided: the second
   overwrote the first and nsjail (which ships in the rootfs sbin) disappeared,
   so the entrypoint smoke test failed with an empty log. Replace the symlink
   with a real directory so both binds land where upstream intends.

4. docker-compose.fifthseason.yml — /usr/local/bin/sandbox-runner-healthcheck.sh
   is hidden by the /usr/local bind and the image ships no curl or wget, so the
   healthcheck could never pass and the worker never started. Probe the sandbox
   API with bash builtins instead.

5. start-direct-sandbox.sh — the Dockerfile creates /host-packages empty, so the
   bare -d test always passed and the mount shadowed the baked /pkgs, leaving
   the sandbox with no runtimes ("python-3.14.4 runtime is unknown"). Only bind
   the host mount when it actually holds packages.

Also adds docker-compose.fifthseason.yml: upstream's compose always maps
/dev/kvm and Compose merges `devices` by appending, so an override cannot remove
it and the stack cannot start on an instance without KVM.

Verified end to end on the patched image: JWT auth (valid key 200, wrong key
401), pandas execution, and PNG/CSV generation returned as session files.
@dboichuk-2
dboichuk-2 merged commit 75c9571 into main Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant