How to run NRE sensorsim without Docker? (Kubernetes GPU-as-a-Service environment)

[crom592]

Environment

• Platform: KADaP (Korea Automotive Data Platform) — a Kubernetes-based GPU-as-a-Service platform
• • GPU: NVIDIA L40S (45GB)
• • • CUDA Driver: 550.90.12 (supports up to CUDA 12.4)
• • • • Host OS: Ubuntu 22.04 (glibc 2.35)
• • • • • AlpaSim: latest main branch

Problem

I'm trying to run AlpaSim (Tutorial Level 3 — individual processes) on KADaP, which provides GPU pods via Kubernetes. Docker-in-Docker is not available on this platform due to Kubernetes security policies. This means I cannot use the standard wizard.py workflow that pulls and runs the NRE sensorsim Docker image (nvcr.io/nvidia/nre/nre-ga:26.02).

I successfully run physics, controller, and driver (VaVAM) as individual processes. However, I'm stuck on sensorsim (NRE) — this is the critical missing piece.

What I've Tried

1. Extract and run NRE from container image (using crane + proot)

I used crane export to download and extract the NRE-GA 26.02 container image as a flat filesystem (~27GB). Then I used proot to chroot into the extracted rootfs and run the NRE binary:

proot -r /path/to/nre-rootfs -b /dev -b /proc -b /sys \
  -b /usr/lib/x86_64-linux-gnu/libcuda.so.1 \
  -b /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 \
  -w /app/internal/scripts/pycena/runtime/pycena_run.runfiles/_main \
  /app/internal/scripts/pycena/runtime/pycena_run serve-grpc \
  --port=6001 --host=0.0.0.0 --artifact-glob='/mnt/nre-data/scenesets/**/*.usd' \
  --no-enable-nrend --cache-size=5 --max-workers=4

Result: The binary starts but fails with:

• No module named 'l1l1l1l1l1l1l1l1l1l1l1l1' — This appears to be a PyArmor obfuscation module that only works within the original Docker container environment.
• • No CUDA runtime is found — The NRE image requires CUDA 12.8, but the host driver only supports up to CUDA 12.4.

2. Direct execution with container's ld-linux

Used the container's ld-linux-x86-64.so.2 (glibc 2.39) to run the binary directly on the host. Same obfuscation module error.

Questions

1. Is there an officially supported way to run NRE sensorsim without Docker? For example, as a standalone binary, via enroot, Singularity/Apptainer, or any other unprivileged container runtime?
2. In Issue How to run sensorsim / NuRec container as non-root #27, it was mentioned that the NVIDIA team uses enroot for non-root environments. Could you provide guidance on how to set up enroot to run the NRE sensorsim image? Are there specific flags or environment variables needed?
3. Is there a way to disable or work around the PyArmor obfuscation when running outside Docker? The l1l1l1l1l1l1l1l1l1l1l1l1 module seems to be a protection mechanism that prevents execution outside the intended container.
4. CUDA version compatibility: The NRE-GA 26.02 image requires CUDA 12.8, but our platform's driver (550.90.12) only supports up to CUDA 12.4. Is there an older NRE image that works with CUDA 12.4, or does NRE support CUDA forward compatibility libraries?
5. Would it be possible to provide a statically-linked or less restricted NRE binary that can run in Kubernetes pods without requiring Docker-in-Docker?

Context

This is related to Issue #27 (running without root/Docker) and Issue #17 (NRE image references). Many cloud GPU platforms (like KADaP, Lambda, CoreWeave managed Kubernetes) don't allow Docker-in-Docker, which makes the sensorsim component impossible to run with the current setup.

Thank you for any guidance!

[maximilianigl]

Hi, this one is tricky. I'm aware of the restrictions of deploying NRE, but unfortunately there's not too much we can do short term.

Regarding your questions:

1.) Not that I'm aware of, but also looping in @daehyoungko , maybe he knows more?
2.) I'm not familiar with how enroot is set up in our cluster. The way we call it is through slurm srun - you can check the slurm.py for the exact commands.
3.) Not that I'm aware of.
4.) There is an older image, but it won't work with some of the other newer changes on main. You should be able to find it in the git history, together with a state of the simulator that works with it. Note that it also requires the 25_07 hugginface data revision.
5.) I don't think there's much that can be done on the NRE binary side - the obfuscations are intentional atm. This is a separate team from us, so I'm not sure what their exact plans/timelines are and whether/when they will release something less restrictive.

That all being said: My (very limited) understanding of K8n is that it can run containers in pods? Is the K8n system you have able to run a single NRE container in a separate pod? Because in that case, you could run NRE in a separate pod and then connect to it from the rest of AlpaSim, which would run in a separte pod. Would that work?

[daehyoungko]

NuRec(NRE) released as a proprietary NVIDIA product, and it is docker container on NGC. and the obfuscations are intention as proprietary product.

@crom592
Out of curiosity, Can you follow Tutorial Level 1,2 without any issue? if so, I am not sure about "docker in docker is not available" on your system. Every services are docker container.