Daytona process backend: deterministic steps each provision a separate, unsynced, anonymous sandbox

[khaliqgant]

Blocker

With RELAYFLOWS_SANDBOX_PROVIDER=daytona, a workflow's deterministic steps cannot do real work:

1. Separate sandboxes per step. process-backend-executor.ts calls backend.createEnvironment(step.name) inside every executeDeterministicStep, and createSandboxProcessBackend maps each createEnvironment to a fresh runtime.launch(). Two deterministic steps that must see the same tree (write a file, then read it back) run in two disconnected sandboxes — unlike the local path, where every step shares one machine and one cwd.
2. No source or workdir sync. Nothing uploads the runner's source into the sandbox. The step's local cwd (e.g. /Users/…/repo) is passed verbatim to runtime.exec, where it does not exist — every command fails or runs against the wrong tree.
3. No sandbox identity. The provisioned sandbox's exact ID is never exposed to the command (nor bound into step output), so evidence cannot be tied to the sandbox that produced it, and a crashed run leaves unattributed sandboxes behind.

Net effect: a clean Daytona process backend is unusable for deterministic workflows — each step silently provisions a fresh, empty, anonymous sandbox and fails.

Reproducer (released @relayflows/cli 1.1.4, Daytona provider)

A two-step deterministic workflow in a git repo: step 1 asserts the synced source is present and writes $RELAYFLOWS_SANDBOX_ID to a file; step 2 reads it back and asserts the same ID and a bound RELAYFLOWS_SOURCE_COMMIT/RELAYFLOWS_TREE_DIGEST. Baseline: FAILED at step 1 (cwd missing remotely, no source, no ID). Evidence to be attached.

Requested fix (fail-closed)

• One shared sandbox per run for deterministic steps, matching local semantics.
• The exact source (git archive of HEAD) and workdir synced into it, with the source commit and tree digest bound into every step and stamped as sandbox labels.
• The exact sandbox ID exposed to every command (RELAYFLOWS_SANDBOX_ID).
• Fail closed: if the source cannot be bound (no git repo), uploaded, or verified, the step refuses to run rather than executing in a desynced sandbox; a step whose cwd escapes the synced source root refuses too.
• Cleanup: the shared sandbox is destroyed at run end, verifiably (get-by-ID → Not Found, absent from full inventory).

[khaliqgant]

Verdict: Relayflows bug, not a sandbox source/workdir bug — reproduced live, four-step gate.

1. Released baseline FAILS. Clean install @relayflows/core@1.1.4 (+@agent-relay/sandbox@0.1.14, @daytonaio/sdk@0.205.1), RELAYFLOWS_SANDBOX_PROVIDER=daytona, two-step deterministic pipeline in a real git repo (HEAD 48d27faf…). Run failed at step 1 in 3 attempts, output fork/exec /usr/bin/zsh: no such file or directory, step 2 skipped. No RELAYFLOWS_SANDBOX_ID, no source, 3 anonymous sandboxes created+destroyed by the retry loop.

2. Attribution: the sandbox seams all work. Direct probe of @agent-relay/sandbox@0.1.14 DaytonaRuntime in fresh sandbox 0f946af0-f532-48c6-8990-71bfef0d335b: exec no-cwd OK; exec at valid remote cwd OK; git archive upload → remote sha256 matches local exactly; extract → file set equals the committed tree, reproducer-marker.txt intact. The baseline failure reproduces at the seam by passing the LOCAL macOS cwd — and equally with any nonexistent LINUX path (/home/daytona/definitely-not-here): the Toolbox's fork/exec /usr/bin/zsh ENOENT blames the shell binary (which exists, 882152 bytes) for the missing cwd. Sandbox-side paper cut only: misleading error message.

3. Candidate PASSES. Source-bound runtime taken over into worktree relayflows-worktrees/clean-sandbox-0905, branch fix/daytona-source-bound-sandbox-0905 @ 92b3661 (unpushed). Packed tarball clean-installed (sha256 97c96ce784bd5e378073c48398aa8f9c3852ad29d3af3f7592a050f35470f4dd), same reproducer, fresh sandbox 8841dace-9ef1-46c0-8992-54262a364be3: run COMPLETED 2/2 — cwd mapped to /home/daytona/relayflows-source, synced marker present, RELAYFLOWS_SANDBOX_ID=8841dace-… identical across steps, RELAYFLOWS_SOURCE_COMMIT=48d27faf… and RELAYFLOWS_TREE_DIGEST=562163b8… bound, step-1 stamp read back in step 2 (VERIFY-OK). Tests: targeted 61/61; full core suite 1089/1089, exit 0; tsc build green.

4. Cleanup proven by id. daytona info → Not Found for 0f946af0…, 4314b0a8…, 8841dace… (candidate destroyed at run end by dispose); inventory grep 0; no leftovers under the workflow's sandbox names. Note: Daytona deletion is eventually-consistent — an immediate re-get races it.

Full evidence (logs, summaries, exact scripts): chief/evidence/relayflows-cleansandbox-0905/ on this machine (REPORT.md is the index). PR deliberately not opened/pushed per lane instructions — worktree relayflows-worktrees/clean-sandbox-0905 @ 92b3661 is the candidate.

[khaliqgant]

Evidence complete — fix on branch fix/daytona-process-source-binding-0905 (PR incoming, not merged):

Baseline (released @relayflows/core@1.1.4, fresh Daytona sandbox 596a2c4a-0d99-4ac4-aa47-8dab4ef1167c, run 67a4f01c11eadeec58317fa0): FAILED. Each of the step's 3 retry attempts provisioned its own fresh, empty sandbox; the runner-local cwd was passed verbatim to the remote sandbox where it does not exist and no source was ever uploaded (fork/exec /usr/bin/bash: no such file or directory); RELAYFLOWS_SANDBOX_ID never existed as a value — no sandbox was identifiable from the run.

Candidate (packed unpublished tarball from fix commit c0b2000, sha256 97c96ce7…, installed in fresh sandbox 1c947406-2ebc-45a7-9421-c429ea84e13f, run e07ca8efd128eb67caea09ad): the same exact reproducer COMPLETED (exit 0). One sandbox for the run (ee58c343-fddf-44bd-8d32-de66b74ba399, identical in both steps, continuity verified by file round-trip); the exact committed source synced (upload sha256-verified, extraction file-set-verified); the source commit 3861e244… and tree digest a44f513c… bound into every step; the exact sandbox id exposed to every command.

Cleanup: bounded polling proves every provisioned id is Not Found AND absent from the full inventory — the engine-destroyed inner sandbox ee58c343… (844 ms) plus both outer sandboxes and the pre-flight sandbox.

Full bundle: .workflow-artifacts/daytona-process-backend-0905/ on the branch (EVIDENCE.md + raw logs + tarball + fixture).

[miyaontherelay]

Status update (2026-09-07): the live clean-sandbox diagnosis is complete. Released @relayflows/core@1.1.4 failed; the packed source-bound candidate passed with shared sandbox/source/workdir continuity and cleanup proof. PR #53 was intentionally closed after review found cwd-resolution, agent-environment continuity, immutable-HEAD race, archive/submodule file-set, and lockfile-version issues. Its replacement branch is no longer present remotely, so #52 still needs a fresh corrected implementation on current main before review, merge, and published-release clean-install proof. No Daytona rerun is needed for diagnosis.

[kjgbot]

Cross-referencing from the Relayflow v2 lane: not handled, and the reason is worth knowing

Asked whether v2 has solved this. Short answer: it cannot hit this bug today, but that is not a fix — and the more useful observation is that Cloud already solves all three symptoms, which makes this backend the odd one out.

v2 cannot hit it, because v2 has no remote execution

The v2 kernel has zero occurrences of sandbox, environment, daytona, cwd or workdir across kernel/**/*.rs. exec_det.rs runs Command::new("/bin/sh") in the daemon's own process space — one machine, one tree, by construction. "Separate sandbox per step" and "local cwd sent to a remote machine" are both structurally impossible there.

That is the absence of the feature, not a solution to it. When v2 grows remote placement it meets all three of these.

Cloud already does the three things this issue says are missing

Worth checking before writing a fourth implementation. AgentWorkforce/cloud has:

• packages/core/src/code-sync/ and storage/code-transfer.js — the run's source is uploaded and extracted into the sandbox, so a real remote path exists to cd into;
• packages/core/src/executor/sandbox-orchestrator.ts — per-root leases, one-shot sync vs continuous daemon, explicit acquire/release;
• the bootstrap (SandboxedStepExecutor, writeRunManifest) runs inside one sandbox for the whole run, so steps share a tree by default rather than by effort;
• provider runtimes under packages/core/src/runtime/ (daytona, local-http) plus JIT reclaim and drain routes.

So this is not an unsolved problem in the org — process-backend-executor.ts is a separate, thinner path that provisions per step and never joins the code-transfer path. That framing may shorten the fix: the question becomes what stops this backend using the orchestration that already works, rather than what new code is needed.

v2 has the contract for the source-binding half, unimplemented

kernel/relayflowd-core/src/entry.rs:185:

/// revision id of each declared workspace surface and the offset of each
/// declared stream. Pins are journal facts, never spec fields — the spec only
/// *declares* surfaces (rule 1).
pub struct Pins {
    pub workspace: Vec<WorkspacePin>,   // { surface, revision_id }
    pub streams:   Vec<StreamPin>,
}

That is structurally what the candidate fix in this thread achieved — "the source commit 3861e244… and tree digest a44f513c… bound into every step" — except as a journal fact with a replay contract rather than a patch. But worker.rs:70 returns Ok(Pins::default()) — empty. Declared, not populated.

Mapping to RFC-0001 gate 7

This issue is close to a ready-made acceptance test for gate 7 ("a relayflow routes to the right sandbox under the hood"), which is unstarted on the flows side — 0 hits for sandbox, requirements or placement in the authoring spec:

symptom here	v2 concept	status
separate sandbox per step	placement + continuity	gate 7, not started in flows; exists in cloud
no source sync; local cwd sent remote	WorkspacePin{surface, revision_id}	contract exists, stub
sandbox id never exposed	routing decision as a journal entry	gate 7, not started

Gate 7's own wording is "the same flow YAML runs locally and in cloud with no placement config; the routing decision is a journal entry" — i.e. exactly this issue's three symptoms, stated as requirements. I intend to cite this thread as acceptance criteria when gate 7 is scoped, so the fix here does not have to be rediscovered a third time.

No action requested — flagging the overlap so neither side solves it twice.