diff --git a/.workflow-artifacts/daytona-process-backend-0905/EVIDENCE.md b/.workflow-artifacts/daytona-process-backend-0905/EVIDENCE.md new file mode 100644 index 0000000..9282dc0 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/EVIDENCE.md @@ -0,0 +1,133 @@ +# Daytona process-backend blocker — evidence bundle + +Issue: https://github.com/AgentWorkforce/relayflows/issues/52 +Branch: `fix/daytona-process-source-binding-0905` +Fix commit: `c0b2000` (`fix(core): bind Daytona deterministic steps to one synced, named sandbox`) +Fix commit tree digest: `7dd5a231594397e5e5ed3e65027838aed0cb1fe8` +Packed candidate: `relayflows-core-candidate-1.1.3.tgz` +sha256(candidate tgz): `97c96ce784bd5e378073c48398aa8f9c3852ad29d3af3f7592a050f35470f4dd` +Baseline (released): `@relayflows/core@1.1.4` from the npm registry (verified: no source binding, per-step `createEnvironment` in its shipped `process-backend-executor.js` / `sandbox-backend.js`). + +## The exact reproducer + +A two-step deterministic workflow (`reproducer-fixture-v3.tgz`, driver +`reproducer-driver-run.mjs`) against a git source root — +fixture commit `3861e2448c39c2be11e01564dc23436f0ed5a85d`, +tree digest `a44f513c9283de2d772dbd7131e925a38e8a46d4`, identical in every run: + +- `probe-bindings`: prints `RELAYFLOWS_SANDBOX_ID` / `RELAYFLOWS_SOURCE_COMMIT` / + `RELAYFLOWS_TREE_DIGEST` (or `ABSENT`), asserts the synced source is present + (`test -f reproducer-marker.txt`), and writes the sandbox id to a file. +- `probe-continuity`: asserts it runs in the SAME sandbox (reads the id file + back) with the SAME bound commit and digest. + +Run under `RELAYFLOWS_SANDBOX_PROVIDER=daytona`, +`RELAYFLOWS_SANDBOX_HOME_DIR=/home/daytona`, +`RELAYFLOWS_SANDBOX_SNAPSHOT=relay-sandbox-lite-sdk-11.8.2-relayfile-v0.10.50-runtime-4.1.41-rf113`, +`DAYTONA_API_KEY` supplied via a mode-600 file (never printed, never committed). + +## 1. Released baseline FAILS in a fresh Daytona sandbox + +- Outer sandbox (fresh, from the same snapshot): `596a2c4a-0d99-4ac4-aa47-8dab4ef1167c` +- Engine: `npm install @relayflows/core@1.1.4` (registry) +- Run id: `67a4f01c11eadeec58317fa0` — **FAILED**, driver exit 1 +- Log: `baseline-released-1.1.4-run.log.txt` + +The failure is the blocker itself, three ways: + +1. **Separate sandboxes.** Each of the step's 3 retry attempts provisioned its + own fresh sandbox (`createEnvironment` per attempt in 1.1.4's + `process-backend-executor.js`); nothing carried between them. +2. **No source or workdir sync.** Every attempt passed the runner-local cwd + (`/home/daytona/reproducer`) verbatim to the remote sandbox, where it does + not exist and no source was uploaded: `fork/exec /usr/bin/bash: no such file + or directory` — the command never even ran. +3. **No sandbox id.** `RELAYFLOWS_SANDBOX_ID` appears in the log only inside + the command text; no value was ever exposed to any command, and no + provisioned sandbox is identifiable from the run. + +## 2. Packed unpublished candidate installed in a fresh sandbox + +- Outer sandbox (fresh): `1c947406-2ebc-45a7-9421-c429ea84e13f` +- Engine: `npm install /home/daytona/relayflows-core-candidate.tgz @daytonaio/sdk` + — tarball sha256 verified in-sandbox as + `97c96ce784bd5e378073c48398aa8f9c3852ad29d3af3f7592a050f35470f4dd` + (identical to the pack from fix commit `c0b2000`). + +## 3. Same exact reproducer PASSES + +- Run id: `e07ca8efd128eb67caea09ad` — **COMPLETED**, driver exit 0 +- Log: `candidate-packed-run.log.txt` + +Step outputs (both steps): + +``` +sid=ee58c343-fddf-44bd-8d32-de66b74ba399 +commit=3861e2448c39c2be11e01564dc23436f0ed5a85d +digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 +CONTINUITY_OK sid=ee58c343-fddf-44bd-8d32-de66b74ba399 commit=3861e2448c39c2be11e01564dc23436f0ed5a85d digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 +``` + +- **One sandbox for the run**: both steps saw `ee58c343-fddf-44bd-8d32-de66b74ba399`; + step 2 read back the id file step 1 wrote (continuity). +- **Exact source and workdir synced**: `test -f reproducer-marker.txt` passed — + the committed tree was uploaded (`git archive HEAD`), digest-verified + (sha256), extracted, and file-set-verified against the committed tree. +- **Source commit and tree digest bound into every step**: both steps report + the fixture's exact `HEAD` commit and tree digest (values identical to the + baseline run's fixture). +- **Exact sandbox id available to the process**: `RELAYFLOWS_SANDBOX_ID` was + set (and identical) in both steps. + +A pre-flight of the same candidate from the orchestrator's machine (before the +sandbox-in-sandbox run) is in `candidate-local-preflight-run.log.txt` +(inner sandbox `9eca9416-48e1-4cb9-a501-8c18b3aa7612`, same PASS shape). + +## 4. Cleanup — same IDs Not Found and absent from full inventory after bounded polling + +`poll-gone` = bounded polling (deadline-bounded loop, 3s interval) requiring +BOTH get-by-id = Not Found AND absence from the full sandbox inventory in the +same round: + +| Sandbox | Role | Evidence file | Result | +| --- | --- | --- | --- | +| `ee58c343-fddf-44bd-8d32-de66b74ba399` | run-shared inner sandbox (candidate run, destroyed by the engine at run end) | `candidate-inner-sandbox-poll-gone.txt` | NOT_FOUND, inInventory=false, inventory 124, 844 ms | +| `1c947406-2ebc-45a7-9421-c429ea84e13f` | outer candidate sandbox (destroyed after evidence) | `candidate-outer-sandbox-poll-gone.txt` | NOT_FOUND, inInventory=false, inventory 123, 10.8 s, sawPresentBefore=true | +| `596a2c4a-0d99-4ac4-aa47-8dab4ef1167c` | outer baseline sandbox (destroyed after evidence) | `baseline-outer-sandbox-poll-gone.txt` | NOT_FOUND, inInventory=false, inventory 125, 4.3 s, sawPresentBefore=true | +| `9eca9416-48e1-4cb9-a501-8c18b3aa7612` | pre-flight inner sandbox | `candidate-local-preflight-inner-sandbox-poll-gone.txt` | NOT_FOUND, inInventory=false, inventory 124, 1.0 s | + +The baseline run's own per-step sandboxes were destroyed inline by the released +engine's per-step `finally` (inventory went 128 → 126 across the run window; +no run leftovers: `base-inventory-before.txt` / `base-inventory-after.txt`). + +## Unit tests + +`packages/core/src/__tests__/sandbox-source-sync.test.ts` (16 tests) pins the +contract exactly: binding to `HEAD` commit + tree digest; label stamping; +archive upload + digest + extraction + file-set verification; every fail-closed +refusal (non-git root, digest mismatch, extraction failure, file-set mismatch — +with nothing provisioned); one shared sandbox per run (including concurrent +first steps); binding env in every command; cwd mapping and escape refusal; +per-step behavior preserved byte-for-byte on unbound backends; dispose teardown; +and the full runner-level contract through a real `WorkflowRunner.execute`. + +## Commands (reproducible skeleton) + +```bash +# pack the candidate from the fix branch +npm run build --workspace=packages/core && npm pack --workspace=packages/core + +# fresh outer sandbox +daytona create --snapshot "$SNAPSHOT" --name rf-evidence-base +# install engine, upload fixture + driver, then run: +export DAYTONA_API_KEY=$(cat /home/daytona/.daytona-key) +export RELAYFLOWS_SANDBOX_PROVIDER=daytona \ + RELAYFLOWS_SANDBOX_HOME_DIR=/home/daytona \ + RELAYFLOWS_SANDBOX_SNAPSHOT=$SNAPSHOT +node run.mjs /home/daytona/reproducer # baseline: exit 1 / FAILED + # candidate: exit 0 / COMPLETED + +# cleanup proof (bounded polling, both conditions) +daytona info # -> not Found +daytona list # -> id absent from full inventory +``` diff --git a/.workflow-artifacts/daytona-process-backend-0905/base-create.log.txt b/.workflow-artifacts/daytona-process-backend-0905/base-create.log.txt new file mode 100644 index 0000000..1bf6cfe --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/base-create.log.txt @@ -0,0 +1 @@ +CREATED id=596a2c4a-0d99-4ac4-aa47-8dab4ef1167c name=rf-evidence-base attempts=1 diff --git a/.workflow-artifacts/daytona-process-backend-0905/base-inventory-after.txt b/.workflow-artifacts/daytona-process-backend-0905/base-inventory-after.txt new file mode 100644 index 0000000..af9b369 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/base-inventory-after.txt @@ -0,0 +1 @@ +INVENTORY count=126 diff --git a/.workflow-artifacts/daytona-process-backend-0905/base-inventory-before.txt b/.workflow-artifacts/daytona-process-backend-0905/base-inventory-before.txt new file mode 100644 index 0000000..22df5a7 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/base-inventory-before.txt @@ -0,0 +1 @@ +INVENTORY count=128 diff --git a/.workflow-artifacts/daytona-process-backend-0905/baseline-outer-sandbox-poll-gone.txt b/.workflow-artifacts/daytona-process-backend-0905/baseline-outer-sandbox-poll-gone.txt new file mode 100644 index 0000000..2565379 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/baseline-outer-sandbox-poll-gone.txt @@ -0,0 +1 @@ +POLL_GONE id=596a2c4a-0d99-4ac4-aa47-8dab4ef1167c getById=NOT_FOUND inInventory=false inventoryCount=125 elapsedMs=4339 sawPresentBefore=true diff --git a/.workflow-artifacts/daytona-process-backend-0905/baseline-released-1.1.4-run.log.txt b/.workflow-artifacts/daytona-process-backend-0905/baseline-released-1.1.4-run.log.txt new file mode 100644 index 0000000..c0a81f4 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/baseline-released-1.1.4-run.log.txt @@ -0,0 +1,40 @@ +EXEC exit=0 +EXEC_OUT_BEGIN +[workflow 00:00] Starting workflow "default" (2 steps) +[workflow] run 67a4f01c11eadeec58317fa0 +[workflow 00:00] Executing 2 steps (pattern: pipeline) + ● probe-bindings — started +[workflow 00:00] [probe-bindings] Running: echo "sid=${RELAYFLOWS_SANDBOX_ID:-ABSENT}" && echo "commit=${RELAYFLOWS_SOURCE_COMMIT:-ABSENT}" && echo "digest=${RELAYFLOWS_TREE_DIGEST:-ABSENT}" && test -f reproducer-marker.txt && printf '%s\n' "$... +[workflow 00:01] [probe-bindings] Command failed (exit code -1) +[workflow 00:01] [probe-bindings] Output: +fork/exec /usr/bin/bash: no such file or directory + ↻ probe-bindings — retrying (attempt 1) + ● probe-bindings — started +[workflow 00:02] [probe-bindings] Running: echo "sid=${RELAYFLOWS_SANDBOX_ID:-ABSENT}" && echo "commit=${RELAYFLOWS_SOURCE_COMMIT:-ABSENT}" && echo "digest=${RELAYFLOWS_TREE_DIGEST:-ABSENT}" && test -f reproducer-marker.txt && printf '%s\n' "$... +[workflow 00:04] [probe-bindings] Command failed (exit code -1) +[workflow 00:04] [probe-bindings] Output: +fork/exec /usr/bin/bash: no such file or directory + ↻ probe-bindings — retrying (attempt 2) + ● probe-bindings — started +[workflow 00:05] [probe-bindings] Running: echo "sid=${RELAYFLOWS_SANDBOX_ID:-ABSENT}" && echo "commit=${RELAYFLOWS_SOURCE_COMMIT:-ABSENT}" && echo "digest=${RELAYFLOWS_TREE_DIGEST:-ABSENT}" && test -f reproducer-marker.txt && printf '%s\n' "$... +[workflow 00:06] [probe-bindings] Command failed (exit code -1) +[workflow 00:06] [probe-bindings] Output: +fork/exec /usr/bin/bash: no such file or directory +[workflow 00:06] [probe-bindings] Output (FAILED): +``` +fork/exec /usr/bin/bash: no such file or directory +``` + ✗ probe-bindings — FAILED: Command failed with exit code -1: fork/exec /usr/bin/bash: no such file or directory + ○ probe-continuity — skipped +[workflow] FAILED: Step "probe-bindings" failed: Step "probe-bindings" failed: Command failed with exit code -1: fork/exec /usr/bin/bash: no such file or directory +[workflow 00:07] Shutting down broker... +RUN_STATUS=failed +RUN_ERROR=Step "probe-bindings" failed: Step "probe-bindings" failed: Command failed with exit code -1: fork/exec /usr/bin/bash: no such file or directory +STEP probe-bindings status=failed +STEP probe-bindings OUTPUT BEGIN +fork/exec /usr/bin/bash: no such file or directory +STEP probe-bindings OUTPUT END +STEP probe-bindings ERROR: Command failed with exit code -1: fork/exec /usr/bin/bash: no such file or directory +STEP probe-continuity status=skipped + +EXEC_OUT_END diff --git a/.workflow-artifacts/daytona-process-backend-0905/cand-create.log.txt b/.workflow-artifacts/daytona-process-backend-0905/cand-create.log.txt new file mode 100644 index 0000000..7fea875 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/cand-create.log.txt @@ -0,0 +1 @@ +CREATED id=1c947406-2ebc-45a7-9421-c429ea84e13f name=rf-evidence-cand attempts=1 diff --git a/.workflow-artifacts/daytona-process-backend-0905/cand-inventory-after.txt b/.workflow-artifacts/daytona-process-backend-0905/cand-inventory-after.txt new file mode 100644 index 0000000..2293f25 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/cand-inventory-after.txt @@ -0,0 +1 @@ +INVENTORY count=124 diff --git a/.workflow-artifacts/daytona-process-backend-0905/cand-inventory-before.txt b/.workflow-artifacts/daytona-process-backend-0905/cand-inventory-before.txt new file mode 100644 index 0000000..0351b24 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/cand-inventory-before.txt @@ -0,0 +1 @@ +INVENTORY count=125 diff --git a/.workflow-artifacts/daytona-process-backend-0905/candidate-inner-sandbox-poll-gone.txt b/.workflow-artifacts/daytona-process-backend-0905/candidate-inner-sandbox-poll-gone.txt new file mode 100644 index 0000000..5213e99 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/candidate-inner-sandbox-poll-gone.txt @@ -0,0 +1 @@ +POLL_GONE id=ee58c343-fddf-44bd-8d32-de66b74ba399 getById=NOT_FOUND inInventory=false inventoryCount=124 elapsedMs=844 sawPresentBefore=false diff --git a/.workflow-artifacts/daytona-process-backend-0905/candidate-local-preflight-inner-sandbox-poll-gone.txt b/.workflow-artifacts/daytona-process-backend-0905/candidate-local-preflight-inner-sandbox-poll-gone.txt new file mode 100644 index 0000000..b0ebcc5 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/candidate-local-preflight-inner-sandbox-poll-gone.txt @@ -0,0 +1 @@ +POLL_GONE id=9eca9416-48e1-4cb9-a501-8c18b3aa7612 getById=NOT_FOUND inInventory=false inventoryCount=124 elapsedMs=1014 sawPresentBefore=false diff --git a/.workflow-artifacts/daytona-process-backend-0905/candidate-local-preflight-run.log.txt b/.workflow-artifacts/daytona-process-backend-0905/candidate-local-preflight-run.log.txt new file mode 100644 index 0000000..20d5742 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/candidate-local-preflight-run.log.txt @@ -0,0 +1,51 @@ +[workflow 00:00] Starting workflow "default" (2 steps) +[workflow] run 6821541f46f60bb11f29b3bd +[workflow 00:00] Executing 2 steps (pattern: pipeline) + ● probe-bindings — started +[workflow 00:00] [probe-bindings] Running: echo "sid=${RELAYFLOWS_SANDBOX_ID:-ABSENT}" && echo "commit=${RELAYFLOWS_SOURCE_COMMIT:-ABSENT}" && echo "digest=${RELAYFLOWS_TREE_DIGEST:-ABSENT}" && test -f reproducer-marker.txt && printf '%s\n' "$... +[workflow 00:07] [probe-bindings] Output: +``` +sid=9eca9416-48e1-4cb9-a501-8c18b3aa7612 +commit=3861e2448c39c2be11e01564dc23436f0ed5a85d +digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 +``` + ✓ probe-bindings — completed + ● probe-continuity — started +[workflow 00:07] [probe-continuity] Running: test "$(cat .reproducer-sid)" = "$RELAYFLOWS_SANDBOX_ID" && test "$RELAYFLOWS_SOURCE_COMMIT" != "ABSENT" && test "$RELAYFLOWS_TREE_DIGEST" != "ABSENT" && echo "CONTINUITY_OK sid=$RELAYFLOWS_SANDBOX_ID... +[workflow 00:07] [probe-continuity] Output: +``` +CONTINUITY_OK sid=9eca9416-48e1-4cb9-a501-8c18b3aa7612 commit=3861e2448c39c2be11e01564dc23436f0ed5a85d digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 +``` + ✓ probe-continuity — completed +[workflow 00:07] Workflow completed successfully +[workflow] completed + +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + Workflow "default" — COMPLETED + 2 passed, 0 failed, 0 skipped +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + Step Status Model Tokens Duration Errors + probe-bindings pass -- -- 7s -- + probe-continuity pass -- -- 141ms -- + ────────────────────────────────────────────────────────────────────────────────── + Total -- 7s + + Run ID: 6821541f46f60bb11f29b3bd + Step output: /var/folders/6d/0x5fkt8d01gfmmjdzkxqzwnh0000gn/T/opencode/daytona-evidence/fixture/.agent-relay/step-outputs/6821541f46f60bb11f29b3bd + Agent logs: /var/folders/6d/0x5fkt8d01gfmmjdzkxqzwnh0000gn/T/opencode/daytona-evidence/fixture/.agent-relay/team/worker-logs +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +[workflow 00:07] Shutting down broker... +RUN_STATUS=completed +STEP probe-bindings status=completed +STEP probe-bindings OUTPUT BEGIN +sid=9eca9416-48e1-4cb9-a501-8c18b3aa7612 +commit=3861e2448c39c2be11e01564dc23436f0ed5a85d +digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 + +STEP probe-bindings OUTPUT END +STEP probe-continuity status=completed +STEP probe-continuity OUTPUT BEGIN +CONTINUITY_OK sid=9eca9416-48e1-4cb9-a501-8c18b3aa7612 commit=3861e2448c39c2be11e01564dc23436f0ed5a85d digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 + +STEP probe-continuity OUTPUT END diff --git a/.workflow-artifacts/daytona-process-backend-0905/candidate-outer-sandbox-poll-gone.txt b/.workflow-artifacts/daytona-process-backend-0905/candidate-outer-sandbox-poll-gone.txt new file mode 100644 index 0000000..c8b9369 --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/candidate-outer-sandbox-poll-gone.txt @@ -0,0 +1 @@ +POLL_GONE id=1c947406-2ebc-45a7-9421-c429ea84e13f getById=NOT_FOUND inInventory=false inventoryCount=123 elapsedMs=10757 sawPresentBefore=true diff --git a/.workflow-artifacts/daytona-process-backend-0905/candidate-packed-run.log.txt b/.workflow-artifacts/daytona-process-backend-0905/candidate-packed-run.log.txt new file mode 100644 index 0000000..181cabf --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/candidate-packed-run.log.txt @@ -0,0 +1,55 @@ +EXEC exit=0 +EXEC_OUT_BEGIN +[workflow 00:00] Starting workflow "default" (2 steps) +[workflow] run e07ca8efd128eb67caea09ad +[workflow 00:00] Executing 2 steps (pattern: pipeline) + ● probe-bindings — started +[workflow 00:00] [probe-bindings] Running: echo "sid=${RELAYFLOWS_SANDBOX_ID:-ABSENT}" && echo "commit=${RELAYFLOWS_SOURCE_COMMIT:-ABSENT}" && echo "digest=${RELAYFLOWS_TREE_DIGEST:-ABSENT}" && test -f reproducer-marker.txt && printf '%s\n' "$... +[workflow 00:01] [probe-bindings] Output: +``` +sid=ee58c343-fddf-44bd-8d32-de66b74ba399 +commit=3861e2448c39c2be11e01564dc23436f0ed5a85d +digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 +``` + ✓ probe-bindings — completed + ● probe-continuity — started +[workflow 00:01] [probe-continuity] Running: test "$(cat .reproducer-sid)" = "$RELAYFLOWS_SANDBOX_ID" && test "$RELAYFLOWS_SOURCE_COMMIT" != "ABSENT" && test "$RELAYFLOWS_TREE_DIGEST" != "ABSENT" && echo "CONTINUITY_OK sid=$RELAYFLOWS_SANDBOX_ID... +[workflow 00:01] [probe-continuity] Output: +``` +CONTINUITY_OK sid=ee58c343-fddf-44bd-8d32-de66b74ba399 commit=3861e2448c39c2be11e01564dc23436f0ed5a85d digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 +``` + ✓ probe-continuity — completed +[workflow 00:01] Workflow completed successfully +[workflow] completed + +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + Workflow "default" — COMPLETED + 2 passed, 0 failed, 0 skipped +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + Step Status Model Tokens Duration Errors + probe-bindings pass -- -- 1s -- + probe-continuity pass -- -- 50ms -- + ────────────────────────────────────────────────────────────────────────────────── + Total -- 1s + + Run ID: e07ca8efd128eb67caea09ad + Step output: /home/daytona/reproducer/.agent-relay/step-outputs/e07ca8efd128eb67caea09ad + Agent logs: /home/daytona/reproducer/.agent-relay/team/worker-logs +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +[workflow 00:02] Shutting down broker... +RUN_STATUS=completed +STEP probe-bindings status=completed +STEP probe-bindings OUTPUT BEGIN +sid=ee58c343-fddf-44bd-8d32-de66b74ba399 +commit=3861e2448c39c2be11e01564dc23436f0ed5a85d +digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 + +STEP probe-bindings OUTPUT END +STEP probe-continuity status=completed +STEP probe-continuity OUTPUT BEGIN +CONTINUITY_OK sid=ee58c343-fddf-44bd-8d32-de66b74ba399 commit=3861e2448c39c2be11e01564dc23436f0ed5a85d digest=a44f513c9283de2d772dbd7131e925a38e8a46d4 + +STEP probe-continuity OUTPUT END + +EXEC_OUT_END diff --git a/.workflow-artifacts/daytona-process-backend-0905/relayflows-core-candidate-1.1.3.tgz b/.workflow-artifacts/daytona-process-backend-0905/relayflows-core-candidate-1.1.3.tgz new file mode 100644 index 0000000..a057e3f Binary files /dev/null and b/.workflow-artifacts/daytona-process-backend-0905/relayflows-core-candidate-1.1.3.tgz differ diff --git a/.workflow-artifacts/daytona-process-backend-0905/reproducer-driver-run.mjs b/.workflow-artifacts/daytona-process-backend-0905/reproducer-driver-run.mjs new file mode 100644 index 0000000..476d38f --- /dev/null +++ b/.workflow-artifacts/daytona-process-backend-0905/reproducer-driver-run.mjs @@ -0,0 +1,23 @@ +// run.mjs — the exact reproducer driver, identical for baseline and candidate. +// Runs INSIDE the outer Daytona sandbox. Exit 0 iff the run completed. +import { runWorkflow, JsonFileWorkflowDb } from '@relayflows/core'; + +const fixtureDir = process.argv[2] ?? '/home/daytona/reproducer'; +const result = await runWorkflow(`${fixtureDir}/reproducer.yaml`, { cwd: fixtureDir }); + +console.log(`RUN_STATUS=${result.status}`); +if (result.error) console.log(`RUN_ERROR=${result.error.slice(0, 500)}`); + +const db = new JsonFileWorkflowDb(`${fixtureDir}/.agent-relay/workflow-runs.jsonl`); +const steps = await db.getStepsByRunId(result.id); +for (const step of steps) { + console.log(`STEP ${step.stepName} status=${step.status}`); + if (step.output) { + console.log(`STEP ${step.stepName} OUTPUT BEGIN`); + console.log(step.output); + console.log(`STEP ${step.stepName} OUTPUT END`); + } + if (step.error) console.log(`STEP ${step.stepName} ERROR: ${String(step.error).slice(0, 500)}`); +} + +process.exit(result.status === 'completed' ? 0 : 1); diff --git a/.workflow-artifacts/daytona-process-backend-0905/reproducer-fixture-v3.tgz b/.workflow-artifacts/daytona-process-backend-0905/reproducer-fixture-v3.tgz new file mode 100644 index 0000000..97a19c4 Binary files /dev/null and b/.workflow-artifacts/daytona-process-backend-0905/reproducer-fixture-v3.tgz differ diff --git a/README.md b/README.md index d247241..9a66663 100644 --- a/README.md +++ b/README.md @@ -1014,6 +1014,35 @@ produces no backend at all, so nothing about the default path changes. An explicit `executor` or `processBackend` still wins over sandbox config, so a host that injects its own backend today keeps it. +**Deterministic steps under `daytona`.** A deterministic workflow only makes +sense when its steps share one world, so the Daytona provider provisions ONE +sandbox per run for deterministic steps — not one per step — matching local +semantics where every step runs on the same machine in the same tree. On first +use it syncs the exact source into that sandbox: it binds the runner's cwd to +its git `HEAD` commit and tree digest, uploads `git archive HEAD`, and +verifies the upload by digest and the extracted file set against the committed +tree. Every deterministic command then runs against that synced workdir and +receives: + +| Env var | Meaning | +| --- | --- | +| `RELAYFLOWS_SANDBOX_ID` | The exact id of the sandbox the command runs in. | +| `RELAYFLOWS_SOURCE_COMMIT` | The source commit the sandbox is bound to. | +| `RELAYFLOWS_TREE_DIGEST` | The tree digest of the synced source. | +| `RELAYFLOWS_SOURCE_WORKDIR` | The directory inside the sandbox the source was synced to. | + +The commit and digest are also stamped as sandbox labels +(`relayflows/source-commit`, `relayflows/tree-digest`), so every provisioned +sandbox is attributable to the tree it runs. + +This is fail-closed by construction: a source root that is not a git repo, an +upload that cannot be verified, or an extraction whose file set does not match +the committed tree refuses to provision a sandbox rather than handing steps a +desynced one; a step whose `cwd` escapes the source root refuses to run. The +shared sandbox is destroyed when the run ends (completed, failed, or +cancelled). A step's local `cwd` inside the source root is mapped to the same +relative path inside the synced workdir. + **Custom providers.** Register a runtime under any name, or hand one in directly. This is the seam a host uses to plug in a runtime that does not live in this repo: diff --git a/packages/core/src/__tests__/process-backend-executor.test.ts b/packages/core/src/__tests__/process-backend-executor.test.ts index ddc6d28..5e4f9fe 100644 --- a/packages/core/src/__tests__/process-backend-executor.test.ts +++ b/packages/core/src/__tests__/process-backend-executor.test.ts @@ -119,7 +119,9 @@ describe('createProcessBackendExecutor', () => { expect(result).toEqual({ output: 'ok', exitCode: 0 }); const [, opts] = exec.mock.calls[0]!; expect(opts?.cwd).toBe('/work'); - expect(opts?.env).toEqual({ RELAY_WORKSPACE: 'ws_123' }); + // The exact sandbox id rides along on every deterministic command so the + // process (and the run's evidence) can name the sandbox it ran in. + expect(opts?.env).toEqual({ RELAY_WORKSPACE: 'ws_123', RELAYFLOWS_SANDBOX_ID: 'env-1' }); expect(opts?.timeoutSeconds).toBe(5); }); }); diff --git a/packages/core/src/__tests__/sandbox-source-sync.test.ts b/packages/core/src/__tests__/sandbox-source-sync.test.ts new file mode 100644 index 0000000..992ca97 --- /dev/null +++ b/packages/core/src/__tests__/sandbox-source-sync.test.ts @@ -0,0 +1,527 @@ +/** + * The Daytona process-backend contract, exactly. + * + * These tests pin the three guarantees a sandboxed deterministic workflow + * cannot work without, and the fail-closed refusals that keep a desynced + * sandbox from ever running a step: + * + * 1. one run-shared sandbox for deterministic steps (not one per step); + * 2. the exact committed source synced and verified in it, with the source + * commit and tree digest bound into every step; + * 3. the exact sandbox id exposed to every command. + * + * The source binding is real: tests build a real git repo in a temp dir and + * run real `git` against it, because the binding is only honest if it is + * computed the way production computes it. The remote side is a fake runtime + * that runs real `sh` for step commands (so file continuity between steps is + * a fact, not an assertion) and implements the exact verification command + * surface the wrapper uses (sha256sum, tar extract, find). + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { execFile } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { mkdir, mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { promisify } from 'node:util'; + +import { InMemoryWorkflowDb } from '../memory-db.js'; +import { + SOURCE_COMMIT_LABEL, + TREE_DIGEST_LABEL, + createSourceBoundSandboxRuntime, + resolveSourceBinding, +} from '../sandbox-source-sync.js'; +import { + SANDBOX_ID_VAR, + SOURCE_COMMIT_VAR, + SOURCE_WORKDIR_VAR, + TREE_DIGEST_VAR, + createProcessBackendExecutor, +} from '../process-backend-executor.js'; +import { registerSandboxProvider } from '../sandbox-backend.js'; +import type { SandboxWorkflowRuntime } from '../sandbox-backend.js'; +import type { ProcessBackend, ProcessEnvironment, WorkflowStep, WorkflowStepRow } from '../types.js'; +import { WorkflowRunner } from '../runner.js'; +import { workflow } from '../builder.js'; + +const execFileAsync = promisify(execFile); + +// ── Fixtures ──────────────────────────────────────────────────────────────── + +/** A real git repo whose committed tree is the "exact source". */ +async function makeSourceRepo(): Promise<{ + root: string; + commit: string; + digest: string; + files: string[]; +}> { + const root = await mkdtemp(path.join(tmpdir(), 'relayflows-srcsync-')); + const { writeFile } = await import('node:fs/promises'); + await writeFile(path.join(root, 'reproducer-marker.txt'), 'marker\n'); + await mkdir(path.join(root, 'nested'), { recursive: true }); + await writeFile(path.join(root, 'nested', 'deep.txt'), 'deep\n'); + await execFileAsync('git', ['-C', root, 'init', '-q']); + await execFileAsync('git', ['-C', root, 'add', '-A']); + await execFileAsync('git', ['-C', root, '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-q', '-m', 'init']); + const commit = (await execFileAsync('git', ['-C', root, 'rev-parse', 'HEAD'])).stdout.trim(); + const digest = (await execFileAsync('git', ['-C', root, 'rev-parse', 'HEAD^{tree}'])).stdout.trim(); + return { root, commit, digest, files: ['nested/deep.txt', 'reproducer-marker.txt'] }; +} + +async function runSh( + command: string, + opts: { cwd?: string; env?: Record } = {} +): Promise<{ output: string; exitCode: number }> { + const cwd = opts.cwd ?? (await mkdtemp(path.join(tmpdir(), 'relayflows-sh-'))); + await mkdir(cwd, { recursive: true }); + const env: Record = { PATH: process.env.PATH ?? '/usr/bin:/bin' }; + Object.assign(env, opts.env ?? {}); + try { + const { stdout, stderr } = await execFileAsync('sh', ['-c', command], { cwd, env }); + return { output: stdout + stderr, exitCode: 0 }; + } catch (error) { + const err = error as { stdout?: string; stderr?: string; code?: number }; + return { output: (err.stdout ?? '') + (err.stderr ?? ''), exitCode: err.code ?? 1 }; + } +} + +/** + * A fake remote runtime implementing the verification command surface the + * source-bound wrapper uses, running real `sh` for everything else, with + * injectable failure modes. + */ +interface FakeRemoteOptions { + /** Files `find` reports after extraction. Defaults to the committed set. */ + reportedFiles?: string[]; + /** Override the sha256 the runtime reports for the uploaded archive. */ + reportedSha?: string; + /** Exit code for the extraction command. */ + extractExitCode?: number; + /** Home dir the fake reports. Default `/home/daytona` (no real fs needed). */ + homeDir?: string; +} + +function createFakeRemoteRuntime(options: FakeRemoteOptions = {}) { + const uploads = new Map(); + const commands: { command: string; cwd?: string; env?: Record }[] = []; + const launchedLabels: Record[] = []; + let destroyed = 0; + let seq = 0; + let root: string | undefined; + + const runtime: SandboxWorkflowRuntime = { + id: 'fake-remote', + async launch(launchOptions = {}) { + launchedLabels.push({ ...(launchOptions.labels ?? {}) }); + root = await mkdtemp(path.join(tmpdir(), 'relayflows-fakeremote-')); + return { id: `fake-sbx-${++seq}`, homeDir: options.homeDir ?? '/home/daytona' }; + }, + async exec(handle, command, execOptions = {}) { + commands.push({ command, cwd: execOptions.cwd, env: execOptions.env }); + if (command.startsWith('sha256sum ')) { + const tarPath = command.split(/\s+/)[1]; + const uploaded = uploads.get(tarPath); + const sha = + options.reportedSha ?? (uploaded ? createHash('sha256').update(uploaded).digest('hex') : 'missing'); + return { output: `${sha}\n`, exitCode: 0 }; + } + if (command.includes('tar -xf')) { + return { output: '', exitCode: options.extractExitCode ?? 0 }; + } + if (command.includes('find .')) { + return { output: `${(options.reportedFiles ?? []).join('\n')}\n`, exitCode: 0 }; + } + // A step command: run it for real, inside this sandbox's own directory, + // with exactly the env the wrapper handed over. + const cwd = execOptions.cwd ?? root; + const result = await runSh(command, { cwd, env: execOptions.env }); + return result; + }, + async uploadFile(_handle, source, destination) { + uploads.set(destination, Buffer.isBuffer(source) ? source : Buffer.from(source)); + }, + async getHomeDir() { + return '/home/daytona'; + }, + async destroy() { + destroyed += 1; + if (root) await rm(root, { recursive: true, force: true }).catch(() => undefined); + }, + }; + + return { runtime, uploads, commands, launchedLabels, getDestroyed: () => destroyed, launchCount: () => seq }; +} + +/** A fake ProcessBackend handing out source-bound or unbound environments. */ +function createFakeProcessBackend(bound: boolean) { + const createdLabels: string[] = []; + const execs: { env: Record; cwd?: string }[] = []; + const envRoots: string[] = []; + let destroyed = 0; + let seq = 0; + + const backend: ProcessBackend = { + async createEnvironment(label: string): Promise { + createdLabels.push(label); + const id = `fake-env-${++seq}`; + const envRoot = await mkdtemp(path.join(tmpdir(), 'relayflows-fakeenv-')); + envRoots.push(envRoot); + return { + id, + homeDir: envRoot, + ...(bound + ? { sourceCommit: 'a'.repeat(40), treeDigest: 'b'.repeat(40), sourceWorkdir: envRoot } + : {}), + async exec(command, opts) { + // Record the contract (the cwd/env the executor handed over); run + // the command for real at this environment's own root so file + // continuity between steps in the SAME environment is a fact. + execs.push({ env: opts?.env ?? {}, cwd: opts?.cwd }); + return runSh(command, { cwd: envRoot, env: opts?.env }); + }, + async uploadFile() { + return; + }, + async destroy() { + destroyed += 1; + await rm(envRoot, { recursive: true, force: true }).catch(() => undefined); + }, + }; + }, + }; + + return { backend, createdLabels, execs, getDestroyed: () => destroyed }; +} + +function fakeStep(name: string): WorkflowStep { + return { name, type: 'deterministic', command: 'true' } as unknown as WorkflowStep; +} + +// ── Source binding ────────────────────────────────────────────────────────── + +describe('resolveSourceBinding', () => { + let repo: Awaited>; + + beforeEach(async () => { + repo = await makeSourceRepo(); + }); + afterEach(async () => { + await rm(repo.root, { recursive: true, force: true }); + }); + + it('binds a git root to the exact HEAD commit and tree digest', async () => { + const binding = await resolveSourceBinding(repo.root); + expect(binding.sourceCommit).toBe(repo.commit); + expect(binding.treeDigest).toBe(repo.digest); + }); + + it('fails closed on a root that is not a git repo', async () => { + const bare = await mkdtemp(path.join(tmpdir(), 'relayflows-notgit-')); + try { + await expect(resolveSourceBinding(bare)).rejects.toThrow(/Sandbox source binding failed.*git/); + } finally { + await rm(bare, { recursive: true, force: true }); + } + }); +}); + +// ── Source-bound runtime wrapper ──────────────────────────────────────────── + +describe('createSourceBoundSandboxRuntime', () => { + let repo: Awaited>; + + beforeEach(async () => { + repo = await makeSourceRepo(); + }); + afterEach(async () => { + await rm(repo.root, { recursive: true, force: true }); + }); + + it('syncs the exact source, labels the sandbox, and binds the handle', async () => { + const fake = createFakeRemoteRuntime({ reportedFiles: repo.files }); + const runtime = createSourceBoundSandboxRuntime(fake.runtime, { sourceRoot: repo.root }); + + const handle = await runtime.launch({ label: 'step-one' }); + + expect(handle.id).toBe('fake-sbx-1'); + expect(handle.sourceCommit).toBe(repo.commit); + expect(handle.treeDigest).toBe(repo.digest); + expect(handle.workdir).toBe('/home/daytona/relayflows-source'); + expect(handle.homeDir).toBe('/home/daytona'); + // The provisioned sandbox is attributable: commit and digest are labels. + expect(fake.launchedLabels[0][SOURCE_COMMIT_LABEL]).toBe(repo.commit); + expect(fake.launchedLabels[0][TREE_DIGEST_LABEL]).toBe(repo.digest); + // The exact archive bytes were uploaded... + const uploaded = fake.uploads.get('/home/daytona/.relayflows-source.tar'); + expect(uploaded).toBeDefined(); + const localArchive = await execFileAsync( + 'git', + ['-C', repo.root, 'archive', '--format=tar', 'HEAD'], + { encoding: 'buffer', maxBuffer: 64 * 1024 * 1024 } as never + ); + expect(uploaded!.equals(localArchive.stdout as unknown as Buffer)).toBe(true); + // ...verified by digest, extracted, then file-set compared. + expect(fake.commands.map((c) => c.command)).toEqual([ + expect.stringContaining('sha256sum /home/daytona/.relayflows-source.tar'), + expect.stringContaining('tar -xf /home/daytona/.relayflows-source.tar -C /home/daytona/relayflows-source'), + expect.stringContaining('find .'), + ]); + }); + + it('fails closed when the uploaded archive digest does not match', async () => { + const fake = createFakeRemoteRuntime({ reportedFiles: repo.files, reportedSha: '0'.repeat(64) }); + const runtime = createSourceBoundSandboxRuntime(fake.runtime, { sourceRoot: repo.root }); + + await expect(runtime.launch({ label: 'step' })).rejects.toThrow(/digest mismatch.*refusing an unsynced sandbox/); + }); + + it('fails closed when extraction fails remotely', async () => { + const fake = createFakeRemoteRuntime({ reportedFiles: repo.files, extractExitCode: 2 }); + const runtime = createSourceBoundSandboxRuntime(fake.runtime, { sourceRoot: repo.root }); + + await expect(runtime.launch({ label: 'step' })).rejects.toThrow(/extracting the source archive.*exited 2/); + }); + + it('fails closed when the extracted file set does not match the committed tree', async () => { + const fake = createFakeRemoteRuntime({ reportedFiles: ['only-one-file.txt'] }); + const runtime = createSourceBoundSandboxRuntime(fake.runtime, { sourceRoot: repo.root }); + + await expect(runtime.launch({ label: 'step' })).rejects.toThrow( + /file set.*does not match the committed tree.*refusing an unsynced sandbox/ + ); + }); + + it('fails closed when the source root is not a git repo — and provisions nothing', async () => { + const bare = await mkdtemp(path.join(tmpdir(), 'relayflows-notgit2-')); + try { + const fake = createFakeRemoteRuntime(); + const runtime = createSourceBoundSandboxRuntime(fake.runtime, { sourceRoot: bare }); + await expect(runtime.launch({ label: 'step' })).rejects.toThrow(/Sandbox source binding failed/); + expect(fake.launchCount()).toBe(0); + } finally { + await rm(bare, { recursive: true, force: true }); + } + }); + + it('passes exec, uploadFile, getHomeDir, and destroy straight through', async () => { + const fake = createFakeRemoteRuntime({ reportedFiles: repo.files }); + const runtime = createSourceBoundSandboxRuntime(fake.runtime, { sourceRoot: repo.root }); + const handle = await runtime.launch({}); + + await runtime.exec(handle, 'echo hi'); + await runtime.uploadFile(handle, 'x', '/home/daytona/x.txt'); + await expect(runtime.getHomeDir(handle)).resolves.toBe('/home/daytona'); + await runtime.destroy(handle); + + expect(fake.commands.at(-1)?.command).toBe('echo hi'); + expect(fake.uploads.get('/home/daytona/x.txt')?.toString()).toBe('x'); + expect(fake.getDestroyed()).toBe(1); + }); +}); + +// ── Executor: one run-shared sandbox for deterministic steps ──────────────── + +describe('createProcessBackendExecutor — deterministic steps', () => { + it('source-bound: steps share ONE sandbox, see the same id, get the binding, keep file continuity', async () => { + const { backend, createdLabels, execs, getDestroyed } = createFakeProcessBackend(true); + const executor = createProcessBackendExecutor(backend, { sourceRoot: '/repo' }); + + const first = await executor.executeDeterministicStep!( + fakeStep('one'), + `printf '%s\\n' "\${${SANDBOX_ID_VAR}}" > stamp`, + '/repo' + ); + const second = await executor.executeDeterministicStep!(fakeStep('two'), 'cat stamp', '/repo'); + + // File continuity across steps — the separate-sandbox defect, directly. + expect(first.exitCode).toBe(0); + expect(second.exitCode).toBe(0); + expect(second.output).toContain(execs[0].env[SANDBOX_ID_VAR]); + // One environment for the whole run — not one per step. + expect(createdLabels).toEqual(['one']); + // Both commands saw the exact same sandbox id and the full binding. + for (const exec of execs) { + expect(exec.env[SANDBOX_ID_VAR]).toBe('fake-env-1'); + expect(exec.env[SOURCE_COMMIT_VAR]).toBe('a'.repeat(40)); + expect(exec.env[TREE_DIGEST_VAR]).toBe('b'.repeat(40)); + expect(exec.env[SOURCE_WORKDIR_VAR]).toBeDefined(); + } + // The shared sandbox is NOT destroyed between steps; dispose tears it down once. + expect(getDestroyed()).toBe(0); + await executor.dispose!(); + expect(getDestroyed()).toBe(1); + }); + + it('source-bound: a subdirectory cwd maps inside the synced workdir', async () => { + const { backend, execs } = createFakeProcessBackend(true); + const executor = createProcessBackendExecutor(backend, { sourceRoot: '/repo' }); + + await executor.executeDeterministicStep!(fakeStep('sub'), 'true', path.join('/repo', 'packages', 'core')); + + expect(execs[0].cwd).toBe(`${execs[0].env[SOURCE_WORKDIR_VAR]}/packages/core`); + }); + + it('source-bound: a cwd escaping the source root fails closed', async () => { + const { backend } = createFakeProcessBackend(true); + const executor = createProcessBackendExecutor(backend, { sourceRoot: '/repo' }); + + await expect(executor.executeDeterministicStep!(fakeStep('escape'), 'true', '/elsewhere')).rejects.toThrow( + /escapes the synced source root.*refusing to run/ + ); + }); + + it('source-bound: no source root configured fails closed instead of guessing', async () => { + const { backend } = createFakeProcessBackend(true); + const executor = createProcessBackendExecutor(backend); + + await expect(executor.executeDeterministicStep!(fakeStep('noroot'), 'true', '/repo')).rejects.toThrow( + /no source root is configured/ + ); + }); + + it('unbound backends keep per-step environments, pass-through cwd, no binding vars', async () => { + const { backend, createdLabels, execs, getDestroyed } = createFakeProcessBackend(false); + const executor = createProcessBackendExecutor(backend, { sourceRoot: '/repo' }); + + await executor.executeDeterministicStep!(fakeStep('one'), 'true', '/repo'); + await executor.executeDeterministicStep!(fakeStep('two'), 'true', '/repo'); + + // Today's contract, byte for byte: an environment per step, destroyed + // inline, cwd untouched, no source binding invented. + expect(createdLabels).toEqual(['one', 'two']); + expect(execs.map((e) => e.cwd)).toEqual(['/repo', '/repo']); + expect(execs.map((e) => e.env[SANDBOX_ID_VAR])).toEqual(['fake-env-1', 'fake-env-2']); // own id, still exposed + for (const exec of execs) { + expect(exec.env[SOURCE_COMMIT_VAR]).toBeUndefined(); + expect(exec.env[TREE_DIGEST_VAR]).toBeUndefined(); + } + expect(getDestroyed()).toBe(2); + await executor.dispose!(); // nothing shared: a no-op + expect(getDestroyed()).toBe(2); + }); + + it('concurrent first steps do not double-provision the shared sandbox', async () => { + const { backend, createdLabels } = createFakeProcessBackend(true); + const executor = createProcessBackendExecutor(backend, { sourceRoot: '/repo' }); + + await Promise.all([ + executor.executeDeterministicStep!(fakeStep('a'), 'true', '/repo'), + executor.executeDeterministicStep!(fakeStep('b'), 'true', '/repo'), + executor.executeDeterministicStep!(fakeStep('c'), 'true', '/repo'), + ]); + + expect(createdLabels).toEqual(['a']); + }); + + it('after dispose, the next run provisions a fresh shared sandbox', async () => { + const { backend, createdLabels } = createFakeProcessBackend(true); + const executor = createProcessBackendExecutor(backend, { sourceRoot: '/repo' }); + + await executor.executeDeterministicStep!(fakeStep('one'), 'true', '/repo'); + await executor.dispose!(); + await executor.executeDeterministicStep!(fakeStep('two'), 'true', '/repo'); + + expect(createdLabels).toEqual(['one', 'two']); + }); +}); + +// ── Runner integration: the guarantees hold through a real run ───────────── + +describe('WorkflowRunner + source-bound provider — the full contract', () => { + const PROVIDER = 'srcsync-fake-provider'; + let repo: Awaited>; + + interface FakeRunState { + destroyed: number; + sandboxIds: string[]; + sourceRootSeen?: string; + } + + beforeEach(async () => { + repo = await makeSourceRepo(); + }); + afterEach(async () => { + await rm(repo.root, { recursive: true, force: true }); + }); + + async function runTwoStepWorkflow(): Promise<{ + status: string; + steps: WorkflowStepRow[]; + state: FakeRunState; + }> { + const state: FakeRunState = { destroyed: 0, sandboxIds: [] }; + registerSandboxProvider(PROVIDER, async (config) => { + state.sourceRootSeen = config.sourceRoot; + const homeDir = await mkdtemp(path.join(tmpdir(), 'relayflows-runhome-')); + const fake = createFakeRemoteRuntime({ reportedFiles: repo.files, homeDir }); + const inner = fake.runtime; + const bound: SandboxWorkflowRuntime = { + id: inner.id, + launch: async (launchOptions = {}) => { + const handle = await inner.launch(launchOptions); + state.sandboxIds.push(handle.id); + return { + ...handle, + workdir: handle.homeDir, + sourceCommit: repo.commit, + treeDigest: repo.digest, + }; + }, + exec: (handle, command, execOptions) => inner.exec(handle, command, execOptions), + uploadFile: (handle, source, destination) => inner.uploadFile(handle, source, destination), + getHomeDir: (handle) => inner.getHomeDir(handle), + destroy: async (handle) => { + state.destroyed += 1; + return inner.destroy(handle); + }, + }; + return bound; + }); + + const config = workflow('srcsync-probe') + .pattern('pipeline') + .step('probe-one', { + type: 'deterministic', + command: `echo "sid=\${${SANDBOX_ID_VAR}:-ABSENT} commit=\${${SOURCE_COMMIT_VAR}:-ABSENT}"`, + }) + .step('probe-two', { + type: 'deterministic', + command: `echo "sid2=\${${SANDBOX_ID_VAR}:-ABSENT} digest=\${${TREE_DIGEST_VAR}:-ABSENT}"`, + dependsOn: ['probe-one'], + }) + .toConfig(); + + const db = new InMemoryWorkflowDb(); + const runner = new WorkflowRunner({ + cwd: repo.root, + db, + sandbox: { provider: PROVIDER, homeDir: '/home/daytona', sourceRoot: repo.root }, + }); + const run = await runner.execute(config, 'srcsync-probe-workflow'); + const steps = await db.getStepsByRunId(run.id); + return { status: run.status, steps, state }; + } + + it('one sandbox for the run, binding in every step, destroyed at run end', async () => { + const { status, steps, state } = await runTwoStepWorkflow(); + + expect(status).toBe('completed'); + // The runner wired its cwd through as the source root. + expect(state.sourceRootSeen).toBe(repo.root); + // ONE sandbox provisioned for the whole run — the separate-sandbox defect. + expect(state.sandboxIds).toHaveLength(1); + const sandboxId = state.sandboxIds[0]; + + const one = steps.find((s) => s.stepName === 'probe-one')!; + const two = steps.find((s) => s.stepName === 'probe-two')!; + // The exact sandbox id is available to each command and identical across steps. + expect(one.output).toContain(`sid=${sandboxId}`); + expect(two.output).toContain(`sid2=${sandboxId}`); + // The source commit and tree digest are bound into every step. + expect(one.output).toContain(`commit=${repo.commit}`); + expect(two.output).toContain(`digest=${repo.digest}`); + // Run end tears the shared sandbox down. + expect(state.destroyed).toBe(1); + }); +}); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 9d100d0..343c4b4 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -7,10 +7,15 @@ export * from './observer-token.js'; export * from './process-spawner.js'; export { createProcessBackendExecutor, + SANDBOX_ID_VAR, + SOURCE_COMMIT_VAR, + SOURCE_WORKDIR_VAR, + TREE_DIGEST_VAR, type ProcessBackendExecutorOptions, } from './process-backend-executor.js'; export * from './sandbox-backend.js'; export * from './sandbox-local-runtime.js'; +export * from './sandbox-source-sync.js'; export * from './run-summary-table.js'; export * from './template-resolver.js'; export * from './verification.js'; diff --git a/packages/core/src/process-backend-executor.ts b/packages/core/src/process-backend-executor.ts index ba3d972..f20acbf 100644 --- a/packages/core/src/process-backend-executor.ts +++ b/packages/core/src/process-backend-executor.ts @@ -8,10 +8,33 @@ * `processBackend` without an explicit `executor`, so every existing * `executor.executeAgentStep(...)` call site transparently flows through * the backend (e.g. a cloud sandbox) without any further plumbing. + * + * Deterministic steps hold a stronger contract than agent steps, because a + * deterministic workflow only makes sense when its steps share one world: + * + * - **One environment per run.** The first deterministic step provisions the + * environment; its siblings reuse it. This mirrors the local path, where + * every step runs on the same machine in the same tree, so "write a file in + * step 1, read it in step 2" works identically sandboxed and not. The + * environment is destroyed by {@link RunnerStepExecutor.dispose}, which the + * runner calls when the run ends. + * - **Sandbox identity is visible.** Every deterministic command receives + * `RELAYFLOWS_SANDBOX_ID` (and, on a source-bound backend, + * `RELAYFLOWS_SOURCE_COMMIT` / `RELAYFLOWS_TREE_DIGEST` / + * `RELAYFLOWS_SOURCE_WORKDIR`), so output can be tied to the exact sandbox + * that produced it. + * - **Source-bound cwd mapping, fail closed.** On a backend that synced the + * source (env reports `sourceCommit`/`treeDigest`/`sourceWorkdir`), a step's + * local cwd is remapped into the synced workdir; a cwd outside the source + * root refuses to run rather than executing against a path that does not + * exist remotely. Backends that do not bind source (local-process, injected + * hosts) keep today's per-step, pass-through behavior byte for byte. */ +import path from 'node:path'; + import { buildCommand } from './process-spawner.js'; -import type { ProcessBackend, AgentDefinition, WorkflowStep, RunnerStepExecutor } from './types.js'; +import type { ProcessBackend, ProcessEnvironment, AgentDefinition, WorkflowStep, RunnerStepExecutor } from './types.js'; function shellEscape(value: string): string { if (value === '') return "''"; @@ -23,9 +46,28 @@ function commandToShell(argv: string[]): string { return argv.map(shellEscape).join(' '); } +/** Env var carrying the exact sandbox id to every deterministic command. */ +export const SANDBOX_ID_VAR = 'RELAYFLOWS_SANDBOX_ID'; +/** Env var carrying the bound source commit, on source-bound backends. */ +export const SOURCE_COMMIT_VAR = 'RELAYFLOWS_SOURCE_COMMIT'; +/** Env var carrying the bound source tree digest, on source-bound backends. */ +export const TREE_DIGEST_VAR = 'RELAYFLOWS_TREE_DIGEST'; +/** Env var carrying the synced workdir, on source-bound backends. */ +export const SOURCE_WORKDIR_VAR = 'RELAYFLOWS_SOURCE_WORKDIR'; + +function isSourceBound(env: ProcessEnvironment): boolean { + return Boolean(env.sourceCommit && env.treeDigest && env.sourceWorkdir); +} + export interface ProcessBackendExecutorOptions { /** Env vars injected into every step (e.g. auth tokens, relayfile config). */ env?: Record; + /** + * Absolute local root whose tree source-bound backends sync into the + * sandbox. Deterministic step cwds inside it are remapped into the synced + * workdir; cwds outside it fail closed. The runner passes its own cwd. + */ + sourceRoot?: string; } export function createProcessBackendExecutor( @@ -33,6 +75,85 @@ export function createProcessBackendExecutor( options: ProcessBackendExecutorOptions = {} ): RunnerStepExecutor { const baseEnv = options.env ?? {}; + const sourceRoot = options.sourceRoot ? path.resolve(options.sourceRoot) : undefined; + + /** The run-shared deterministic environment, once a source-bound backend provisions one. */ + let sharedEnv: ProcessEnvironment | undefined; + /** Serializes provisioning so concurrent first steps cannot double-launch. */ + let provisioning: Promise | undefined; + + /** + * Acquire the environment a deterministic step runs in. + * + * Source-bound backends: one shared environment per executor (per run), + * released only by `dispose`. Unbound backends: one environment per step, + * destroyed inline — exactly the behavior before run-sharing existed. + */ + async function acquireDeterministicEnvironment( + label: string + ): Promise<{ env: ProcessEnvironment; release: (() => Promise) | undefined }> { + if (sharedEnv) return { env: sharedEnv, release: undefined }; + + // Serialize the first provisioning; a fan-out's concurrent first steps + // must not each launch their own sandbox. + while (provisioning) await provisioning; + if (sharedEnv) return { env: sharedEnv, release: undefined }; + + let unlock: () => void = () => undefined; + provisioning = new Promise((resolve) => { + unlock = resolve; + }); + try { + const env = await backend.createEnvironment(label); + if (isSourceBound(env)) { + sharedEnv = env; + return { env, release: undefined }; + } + return { + env, + release: () => + env.destroy().catch(() => { + // A step-scoped teardown failure must not mask the step's own + // result; the shared path is the one that must be airtight. + }), + }; + } finally { + unlock(); + provisioning = undefined; + } + } + + /** + * Map a local step cwd into the synced sandbox workdir. Fails closed when + * the cwd escapes the source root: the path exists only on the runner's + * machine, and running against it remotely is the desync this seam exists + * to prevent. + */ + function mapSourceBoundCwd(env: ProcessEnvironment, cwd: string): string { + if (!sourceRoot) { + throw new Error( + `Deterministic step cwd "${cwd}" cannot be mapped into source-bound sandbox "${env.id}": ` + + `no source root is configured. Pass ProcessBackendExecutorOptions.sourceRoot (the runner does).` + ); + } + const rel = path.relative(sourceRoot, path.resolve(cwd)); + if (rel.startsWith('..') || path.isAbsolute(rel)) { + throw new Error( + `Deterministic step cwd "${cwd}" escapes the synced source root "${sourceRoot}"; ` + + `refusing to run it in sandbox "${env.id}" against a path that exists only on the runner. ` + + `Move the step's cwd inside the source root.` + ); + } + return rel === '' ? env.sourceWorkdir! : path.posix.join(env.sourceWorkdir!, rel.split(path.sep).join('/')); + } + + function bindingEnv(env: ProcessEnvironment): Record { + const binding: Record = { [SANDBOX_ID_VAR]: env.id }; + if (env.sourceCommit) binding[SOURCE_COMMIT_VAR] = env.sourceCommit; + if (env.treeDigest) binding[TREE_DIGEST_VAR] = env.treeDigest; + if (env.sourceWorkdir) binding[SOURCE_WORKDIR_VAR] = env.sourceWorkdir; + return binding; + } return { async executeAgentStep( @@ -87,20 +208,42 @@ export function createProcessBackendExecutor( resolvedCommand: string, cwd: string ): Promise<{ output: string; exitCode: number }> { - const env = await backend.createEnvironment(step.name); + const { env, release } = await acquireDeterministicEnvironment(step.name); try { const execOpts: { cwd?: string; env?: Record; timeoutSeconds?: number; - } = { cwd }; - if (Object.keys(baseEnv).length > 0) execOpts.env = baseEnv; + } = {}; + if (isSourceBound(env)) { + execOpts.cwd = mapSourceBoundCwd(env, cwd); + } else { + execOpts.cwd = cwd; + } + execOpts.env = { ...baseEnv, ...bindingEnv(env) }; if (step.timeoutMs && step.timeoutMs > 0) { execOpts.timeoutSeconds = Math.max(1, Math.ceil(step.timeoutMs / 1000)); } return await env.exec(resolvedCommand, execOpts); } finally { - await env.destroy().catch(() => undefined); + await release?.(); + } + }, + + async dispose(): Promise { + const env = sharedEnv; + sharedEnv = undefined; + if (env) { + await env.destroy().catch((error: unknown) => { + // Run-end teardown is already the last act of the run; surface the + // failure as a rejected dispose so the runner can log it, but never + // let it retroactively fail a completed run. + throw new Error( + `Failed to destroy the run-shared deterministic sandbox "${env.id}": ${ + error instanceof Error ? error.message : String(error) + }` + ); + }); } }, }; diff --git a/packages/core/src/runner.ts b/packages/core/src/runner.ts index 16ef7e1..23b978c 100644 --- a/packages/core/src/runner.ts +++ b/packages/core/src/runner.ts @@ -1035,12 +1035,16 @@ export class WorkflowRunner { // Only reached when the caller injected neither an executor nor a // backend. The config's provider defaults to `none`, which yields // `undefined` here and leaves the local child-process path intact. - const sandboxConfig = options.sandbox ?? resolveSandboxConfigFromEnv(); + const sandboxConfig = { ...(options.sandbox ?? resolveSandboxConfigFromEnv()) }; + // The runner's cwd is the source root a source-bound provider (Daytona) + // syncs into every sandbox — unless the caller or environment named one. + if (!sandboxConfig.sourceRoot) sandboxConfig.sourceRoot = this.cwd; this.processBackend = createLazySandboxProcessBackend(sandboxConfig); } if (!this.executor && this.processBackend) { this.executor = createProcessBackendExecutor(this.processBackend, { env: this.envSecrets, + sourceRoot: this.cwd, }); } this.templateResolver = new TemplateResolver(); @@ -4565,6 +4569,10 @@ export class WorkflowRunner { await this.stopRelayfileEventSubscriptions(); + // Run-scoped executor resources — the run-shared deterministic sandbox + // on sandboxed runs — belong to this run and must not outlive it. + await this.disposeStepExecutor(); + this.log('Shutting down broker...'); await this.shutdownRelay(); this.currentBrokerContext = undefined; @@ -4597,6 +4605,21 @@ export class WorkflowRunner { this.paused = true; } + /** + * Tear down run-scoped executor resources — the run-shared deterministic + * sandbox, on sandboxed runs. Best-effort by design: teardown must never + * retroactively fail a run that already has its final status. + */ + private async disposeStepExecutor(): Promise { + try { + await this.executor?.dispose?.(); + } catch (error) { + this.log( + `Executor teardown failed: ${error instanceof Error ? error.message : String(error)}` + ); + } + } + /** Resume after a pause(). */ unpause(): void { this.paused = false; diff --git a/packages/core/src/sandbox-backend.ts b/packages/core/src/sandbox-backend.ts index dce41ef..7befd7e 100644 --- a/packages/core/src/sandbox-backend.ts +++ b/packages/core/src/sandbox-backend.ts @@ -35,6 +35,10 @@ export interface SandboxRuntimeHandle { id: string; homeDir?: string; workdir?: string; + /** Commit of the exact source synced into this sandbox, when the runtime binds source. */ + sourceCommit?: string; + /** Tree digest of the synced source, when the runtime binds source. */ + treeDigest?: string; } /** Options a provider accepts when creating a sandbox. */ @@ -112,6 +116,13 @@ export interface SandboxBackendConfig { homeDir?: string; /** Working directory inside the sandbox. */ workdir?: string; + /** + * Local git root whose exact committed source is synced into every sandbox + * (Daytona). The runner sets this to its own cwd; the Daytona provider + * fails closed without a boundable git root, because a remote sandbox + * without the source is a sandbox every deterministic step fails in. + */ + sourceRoot?: string; /** Env injected at sandbox creation (per-step env is layered on top). */ env?: Record; /** Provider labels stamped on each created sandbox. */ @@ -173,6 +184,9 @@ export function resolveSandboxConfigFromEnv( const workdir = env.RELAYFLOWS_SANDBOX_WORKDIR?.trim(); if (workdir) config.workdir = workdir; + const sourceRoot = env.RELAYFLOWS_SANDBOX_SOURCE_ROOT?.trim(); + if (sourceRoot) config.sourceRoot = sourceRoot; + const createTimeout = Number(env.RELAYFLOWS_SANDBOX_CREATE_TIMEOUT_SECONDS); if (Number.isFinite(createTimeout) && createTimeout > 0) { config.createTimeoutSeconds = createTimeout; @@ -260,6 +274,9 @@ export function createSandboxProcessBackend( return { id: handle.id, homeDir, + ...(handle.sourceCommit ? { sourceCommit: handle.sourceCommit } : {}), + ...(handle.treeDigest ? { treeDigest: handle.treeDigest } : {}), + ...(handle.sourceCommit && handle.workdir ? { sourceWorkdir: handle.workdir } : {}), async exec(command, execOpts) { const sandboxOpts: SandboxExecOptions = {}; if (execOpts?.cwd) sandboxOpts.cwd = execOpts.cwd; @@ -405,5 +422,13 @@ registerSandboxProvider('daytona', async (config) => { }; if (config.snapshot) runtimeOptions.snapshot = config.snapshot; - return new DaytonaRuntime(runtimeOptions) as unknown as SandboxWorkflowRuntime; + const daytonaRuntime = new DaytonaRuntime(runtimeOptions) as unknown as SandboxWorkflowRuntime; + // Source-bound: every sandbox this runtime provisions carries the exact + // committed source (git archive of HEAD), its commit and tree digest on the + // handle and as labels, and refuses to launch when the source cannot be + // bound or verified. See sandbox-source-sync.ts. + const { createSourceBoundSandboxRuntime } = await import('./sandbox-source-sync.js'); + return createSourceBoundSandboxRuntime(daytonaRuntime, { + sourceRoot: config.sourceRoot ?? process.cwd(), + }); }); diff --git a/packages/core/src/sandbox-source-sync.ts b/packages/core/src/sandbox-source-sync.ts new file mode 100644 index 0000000..8472662 --- /dev/null +++ b/packages/core/src/sandbox-source-sync.ts @@ -0,0 +1,252 @@ +/** + * Source-bound sandbox provisioning. + * + * A remote sandbox is useless to a deterministic step unless three facts hold: + * the step runs in the SAME sandbox as its siblings (one per run, like the + * local path shares one machine), the EXACT source tree is present in it, and + * the process can see WHICH sandbox it is in. Historically none held: each + * step launched its own empty sandbox, was pointed at a cwd path that only + * exists on the runner's machine, and was never told the sandbox ID. + * + * This module wraps a {@link SandboxWorkflowRuntime} (the Daytona provider + * among them) and makes those three facts a launch-time invariant — or refuses + * to launch. "Fail closed" is the contract: if the source cannot be identified + * (not a git repo), transported (upload), or verified (digest + file set), no + * sandbox is handed back, so no step can run in a desynced environment. + * + * The source identity is a git commit and its tree digest; the bytes synced + * are `git archive HEAD` of that commit, so the digest names exactly the tree + * that was uploaded. Both are stamped as sandbox labels and carried on the + * handle, which is how every step gets them bound into its environment. + */ + +import { execFile } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { promisify } from 'node:util'; + +import type { + SandboxExecOptions, + SandboxExecResult, + SandboxLaunchOptions, + SandboxRuntimeHandle, + SandboxWorkflowRuntime, +} from './sandbox-backend.js'; + +const execFileAsync = promisify(execFile); + +/** Label carrying the source commit on every provisioned sandbox. */ +export const SOURCE_COMMIT_LABEL = 'relayflows/source-commit'; +/** Label carrying the source tree digest on every provisioned sandbox. */ +export const TREE_DIGEST_LABEL = 'relayflows/tree-digest'; + +/** Directory name (under the sandbox home) the source archive is extracted to. */ +export const SOURCE_WORKDIR_NAME = 'relayflows-source'; + +/** Bounded time for each verification command run inside the sandbox. */ +const SYNC_EXEC_TIMEOUT_MS = 300_000; + +/** The exact source identity a sandbox is bound to. */ +export interface SourceBinding { + /** Full commit hash of `HEAD` in the source root. */ + sourceCommit: string; + /** Tree digest of that commit (`git rev-parse HEAD^{tree}`). */ + treeDigest: string; +} + +export interface SourceBoundRuntimeOptions { + /** + * Local git working tree whose committed HEAD is the exact source. Required + * and must be a git repo with at least one commit — there is no honest way + * to bind a sandbox to "whatever files happen to be here". + */ + sourceRoot: string; +} + +function gitError(context: string, error: unknown): Error { + const detail = error instanceof Error ? error.message : String(error); + return new Error( + `Sandbox source binding failed (${context}). The sandbox provider requires a git source root with at least one commit; ` + + `refusing to provision an unsynced sandbox. Detail: ${detail}` + ); +} + +async function git(sourceRoot: string, args: string[]): Promise { + try { + const { stdout } = await execFileAsync('git', ['-C', sourceRoot, ...args], { + maxBuffer: 64 * 1024 * 1024, + }); + return stdout.trim(); + } catch (error) { + throw gitError(`git ${args.join(' ')}`, error); + } +} + +/** + * Resolve the exact source identity of a source root: the HEAD commit and its + * tree digest. Throws (fail closed) when the root is not a boundable git repo. + */ +export async function resolveSourceBinding(sourceRoot: string): Promise { + const sourceCommit = await git(sourceRoot, ['rev-parse', 'HEAD']); + const treeDigest = await git(sourceRoot, ['rev-parse', 'HEAD^{tree}']); + if (!/^[0-9a-f]{40,64}$/.test(sourceCommit) || !/^[0-9a-f]{40,64}$/.test(treeDigest)) { + throw gitError('unexpected rev-parse output', `${sourceCommit} ${treeDigest}`); + } + return { sourceCommit, treeDigest }; +} + +/** The exact bytes of a source root's HEAD tree, as a tar archive. */ +export async function archiveSource(sourceRoot: string): Promise { + try { + // `encoding: 'buffer'` keeps the archive bytes intact; the default string + // decoding would corrupt them before they ever reach the sandbox. + const { stdout } = await execFileAsync('git', ['-C', sourceRoot, 'archive', '--format=tar', 'HEAD'], { + encoding: 'buffer', + maxBuffer: 512 * 1024 * 1024, + }); + // promisify's signature types stdout as string; with `encoding: 'buffer'` + // the runtime value is the raw Buffer we need. + return stdout as unknown as Buffer; + } catch (error) { + throw gitError('git archive HEAD', error); + } +} + +/** Sorted list of file paths the archive writes (blobs in the HEAD tree). */ +async function archivedFilePaths(sourceRoot: string): Promise { + const output = await git(sourceRoot, ['ls-tree', '-r', '--name-only', 'HEAD']); + return output + .split('\n') + .map((line) => line.trim()) + .filter((line) => line.length > 0) + .sort(); +} + +/** + * Wrap a runtime so every launch is bound to the exact source of + * `options.sourceRoot` before the handle is returned. + * + * The wrapper owns only provisioning; `exec`/`uploadFile`/`getHomeDir`/ + * `destroy` pass straight through to the underlying runtime. + */ +export function createSourceBoundSandboxRuntime( + runtime: SandboxWorkflowRuntime, + options: SourceBoundRuntimeOptions +): SandboxWorkflowRuntime { + const { sourceRoot } = options; + + async function syncSource( + handle: SandboxRuntimeHandle, + homeDir: string, + binding: SourceBinding, + archive: Buffer + ): Promise { + const tarPath = `${homeDir}/.relayflows-source.tar`; + const workdir = `${homeDir}/${SOURCE_WORKDIR_NAME}`; + + await runtime.uploadFile(handle, archive, tarPath); + + // Transport integrity: the uploaded bytes must be the archived bytes. + const expectedSha = createHash('sha256').update(archive).digest('hex'); + const shaResult = await runtime.exec( + handle, + `sha256sum ${tarPath} | cut -d ' ' -f1`, + { timeoutMs: SYNC_EXEC_TIMEOUT_MS } + ); + if (shaResult.exitCode !== 0 || shaResult.output.trim() !== expectedSha) { + throw new Error( + `Sandbox source sync failed: uploaded archive digest mismatch for sandbox "${handle.id}" ` + + `(expected ${expectedSha}, got ${shaResult.output.trim() || `exit ${shaResult.exitCode}`}); refusing an unsynced sandbox.` + ); + } + + // Extraction into the workdir every step will run against. + const extract = await runtime.exec( + handle, + `mkdir -p ${workdir} && tar -xf ${tarPath} -C ${workdir}`, + { timeoutMs: SYNC_EXEC_TIMEOUT_MS } + ); + if (extract.exitCode !== 0) { + throw new Error( + `Sandbox source sync failed: extracting the source archive into "${workdir}" in sandbox "${handle.id}" ` + + `exited ${extract.exitCode}: ${extract.output.slice(0, 500)}` + ); + } + + // Exactness: the extracted file set must equal the committed file set. + const expected = await archivedFilePaths(sourceRoot); + const listing = await runtime.exec( + handle, + `cd ${workdir} && find . \\( -type f -o -type l \\) | sed 's|^\\./||' | sort`, + { timeoutMs: SYNC_EXEC_TIMEOUT_MS } + ); + if (listing.exitCode !== 0) { + throw new Error( + `Sandbox source sync failed: verifying the extracted file set in sandbox "${handle.id}" ` + + `exited ${listing.exitCode}: ${listing.output.slice(0, 500)}` + ); + } + const actual = listing.output + .split('\n') + .map((line) => line.trim()) + .filter((line) => line.length > 0); + if (actual.length !== expected.length || actual.some((p, i) => p !== expected[i])) { + throw new Error( + `Sandbox source sync failed: extracted file set in sandbox "${handle.id}" does not match the committed tree ` + + `${binding.treeDigest} (${actual.length} files extracted, ${expected.length} committed); refusing an unsynced sandbox.` + ); + } + + return workdir; + } + + return { + id: runtime.id, + + async launch(launchOptions: SandboxLaunchOptions = {}): Promise { + const binding = await resolveSourceBinding(sourceRoot); + const archive = await archiveSource(sourceRoot); + + const labels: Record = { + ...(launchOptions.labels ?? {}), + [SOURCE_COMMIT_LABEL]: binding.sourceCommit, + [TREE_DIGEST_LABEL]: binding.treeDigest, + }; + const handle = await runtime.launch({ ...launchOptions, labels }); + const homeDir = handle.homeDir ?? (await runtime.getHomeDir(handle)); + + const workdir = await syncSource(handle, homeDir, binding, archive); + + return { + ...handle, + homeDir, + workdir, + sourceCommit: binding.sourceCommit, + treeDigest: binding.treeDigest, + }; + }, + + async exec( + handle: SandboxRuntimeHandle, + command: string, + execOptions?: SandboxExecOptions + ): Promise { + return runtime.exec(handle, command, execOptions); + }, + + async uploadFile( + handle: SandboxRuntimeHandle, + source: string | Buffer, + destination: string + ): Promise { + return runtime.uploadFile(handle, source, destination); + }, + + async getHomeDir(handle: SandboxRuntimeHandle): Promise { + return runtime.getHomeDir(handle); + }, + + async destroy(handle: SandboxRuntimeHandle): Promise { + return runtime.destroy(handle); + }, + }; +} diff --git a/packages/core/src/schema.ts b/packages/core/src/schema.ts index d3ea554..80b6be8 100644 --- a/packages/core/src/schema.ts +++ b/packages/core/src/schema.ts @@ -532,4 +532,11 @@ export interface RunnerStepExecutor { }) => Promise; } ): Promise<{ output: string; success: boolean }>; + + /** + * Release run-scoped resources when the run ends (completed, failed, or + * cancelled). The runner calls it once per run; executors that hold nothing + * (one environment per step, destroyed inline) may omit it. + */ + dispose?(): Promise; } diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index b944d13..4a4413a 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -676,6 +676,19 @@ export interface ProcessEnvironment { id: string; /** Home directory inside the environment. */ homeDir: string; + /** + * Commit of the exact source synced into this environment, when the backend + * binds source (the Daytona provider does; a remote sandbox without the + * source is one every deterministic step fails in). + */ + sourceCommit?: string; + /** Tree digest of the synced source, when the backend binds source. */ + treeDigest?: string; + /** + * Directory inside the environment the synced source was materialized in. + * Set only on source-bound environments, whose local cwd is remapped into it. + */ + sourceWorkdir?: string; /** Execute a shell command in the environment. */ exec( command: string,