fix(sdk): reuse authored CLI probes for each run so concurrent f.llm never outlives its lease (#561)

evidence/561/README.md

@@ -0,0 +1,112 @@
+# #561 verification commands and captured output
+
+Commands ran from the repository root unless a `cd` is shown. Output files are literal stdout/stderr captures, not rewritten summaries. The initial pre-fix tests were committed in `7533229`; the fix and corrected assertions are in `55caf19`. The first baseline predates the corrected journal entry spelling and timing headroom; the mutation transcript uses the final assertions.
+
+## Regression and mutation
+
+For `baseline.txt`, `fixed.txt`, `mutation-red.txt`, and `mutation-green.txt`:
+
+```sh
+cd packages/sdk
+RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd npx vitest run tests/authored-probe-cache.test.ts
+```
+
+[Baseline](baseline.txt), [fixed](fixed.txt), [mutation failure](mutation-red.txt), [restored pass](mutation-green.txt).
+
+To reproduce the mutation from the fixed checkout:
+
+```sh
+git apply evidence/561/mutation.patch
+# Run the regression command above (exit 1).
+git restore -- packages/sdk/src/authored-worker-step.ts
+git diff --exit-code -- packages/sdk/src/authored-worker-step.ts
+# Run the regression command above again (exit 0).
+```
+
+The mutation removes only the fifth `checkAuthoredFlow` argument, as captured in [mutation.patch](mutation.patch). After the failure, `git restore -- packages/sdk/src/authored-worker-step.ts` restores the committed bytes. [restore.txt](restore.txt) captures `git diff --exit-code` and the SHA-256 comparison with HEAD.
+
+### Re-verified at `100cd17` (branch head)
+
+The mutation was run once more at the branch head, after the artifact-gate fix
+(`1ccaaed`) landed, so the transcript matches the bytes a reviewer checks out:
+
+```sh
+git apply evidence/561/mutation.patch
+cd packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd \
+  ./node_modules/.bin/vitest run tests/authored-probe-cache.test.ts   # exit 1
+cd .. && git restore -- packages/sdk/src/authored-worker-step.ts
+git diff --exit-code -- packages/sdk/src/authored-worker-step.ts      # clean
+sha256sum packages/sdk/src/authored-worker-step.ts
+#   b9cd463d9f1ee274175a0b659119262c0ba5b18f1afc11b866c3e0f7f34023b7
+cd packages/sdk && RELAYFLOWD_BIN=... ./node_modules/.bin/vitest run \
+  tests/authored-probe-cache.test.ts                                  # exit 0
+```
+
+[Mutation failure at head](mutation-red-at-head.txt) — all 5 cases fail, and the
+capacity-1 and capacity-4 journals carry the issue's exact shape: attempt 1
+`"completionReason":"lease_expired"` with `"wallclock_ms":30011`, a
+`retry_backoff` sleep, then attempt 2 `"completionReason":"success"`.
+[Restored pass at head](mutation-green-at-head.txt) — 5 passed, exit 0.
+
+The spread bound allows one 300 ms synchronous probe plus process startup under load (1,000 ms total); the original nine probes take over 3 seconds. Exact auth and identify-only counts additionally pin the cache independently of timing. Worker identification sessions are counted separately from preflight probes. Tests also cover failures, agents, run isolation, capacity, and every child journal.
+
+## Broader checks
+
+```sh
+cd packages/sdk
+npm test
+```
+
+[sdk-suite.txt](sdk-suite.txt) is the full output, including failures. It includes production and existing test typechecks and the SDK build. It is **not green**. No gates or existing tests were weakened to accommodate this environment.
+
+```sh
+cd kernel
+sh ../ops/cargo.sh test
+```
+
+[kernel.txt](kernel.txt) contains the full kernel result.
+
+```sh
+cd packages/sdk
+npm run typecheck
+```
+
+[typecheck.txt](typecheck.txt).
+
+```sh
+cd packages/sdk
+npx vitest run tests/preflight-run-cache.test.ts
+```
+
+[cache-keys.txt](cache-keys.txt).
+
+## Live reproduction
+
+The [repro source](runtime-parallel-llm-repro.flow.ts) is copied verbatim from `origin/feat/examples-prompt-lab:examples/prompt-lab/evidence/runtime-findings/runtime-parallel-llm-repro.flow.ts`. Adjacent `flows.json` chooses Claude and allows its declared model; `package.json` declares ESM.
+
+```sh
+RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd node packages/sdk/dist/cli.js run --local-agent --no-observer-link --data-dir /tmp/flows-561-live-default evidence/561/runtime-parallel-llm-repro.flow.ts --input '{"n":3}'
+RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd node packages/sdk/dist/cli.js run --local-agent --agent-capacity 1 --no-observer-link --data-dir /tmp/flows-561-live-one evidence/561/runtime-parallel-llm-repro.flow.ts --input '{"n":3}'
+python3 evidence/561/check-live-journals.py
+```
+
+[Default capacity](live-default.txt), [capacity 1](live-one.txt), [journal assertions and outputs](live-journals.txt). Both CLI invocations exited 0. Journals live in the named `/tmp` directories in this environment; they are not committed. Reproduction needs authenticated Claude. `npm run build` in `packages/sdk` regenerates `dist` if the test runner has pruned it.
+
+```sh
+python3 - <<'PY'
+from pathlib import Path
+for p in ['examples/prompt-lab/jobs/shared.ts', 'examples/prompt-lab/jobs/new-agency.ts']:
+    print(p + ': ' + ('present' if Path(p).exists() else 'absent'))
+PY
+```
+
+[prompt-lab.txt](prompt-lab.txt) records why the example restoration remains open.
+
+## Typecheck the new regression files
+
+```sh
+cd packages/sdk
+npx tsc -p ../../evidence/561/tsconfig.tests.json
+```
+
+[regression-typecheck.txt](regression-typecheck.txt) captures output and exit status. This separate config includes the new tests without changing the repository's check configuration.

evidence/561/artifact-gates-green.txt

@@ -0,0 +1,11 @@
+
+ RUN  v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk
+
+ ✓ tests/artifact-gates.test.ts (27 tests) 259ms
+
+ Test Files  1 passed (1)
+      Tests  27 passed (27)
+   Start at  06:37:46
+   Duration  1.89s (transform 840ms, setup 0ms, collect 1.46s, tests 259ms, environment 0ms, prepare 44ms)
+
+EXIT=0

evidence/561/artifact-gates-red.txt

@@ -0,0 +1,36 @@
+
+ RUN  v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk
+
+ ❯ tests/artifact-gates.test.ts (27 tests | 1 failed) 248ms
+   × artifact_exists named gate: static scan coverage > does not refuse a gate the bundled worker itself can satisfy through JSON output 10ms
+     → expected [ 'invalid_spec' ] to deeply equal [ 'gate_path_unscanned' ]
+
+⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
+
+ FAIL  tests/artifact-gates.test.ts > artifact_exists named gate: static scan coverage > does not refuse a gate the bundled worker itself can satisfy through JSON output
+AssertionError: expected [ 'invalid_spec' ] to deeply equal [ 'gate_path_unscanned' ]
+
+- Expected
++ Received
+
+  Array [
+-   "gate_path_unscanned",
++   "invalid_spec",
+  ]
+
+ ❯ tests/artifact-gates.test.ts:318:50
+    316|     const checked = preflight(authored as never, { probes: { ...probes…
+    317|       cli: () => ({ exists: true, authenticated: true, modelAvailable:…
+    318|     expect(checked.diagnostics.map(d => d.kind)).toEqual(['gate_path_u…
+       |                                                  ^
+    319|     expect(checked.ok).toBe(true);
+    320| 
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
+
+ Test Files  1 failed (1)
+      Tests  1 failed | 26 passed (27)
+   Start at  06:37:40
+   Duration  1.90s (transform 820ms, setup 0ms, collect 1.48s, tests 248ms, environment 0ms, prepare 42ms)
+
+EXIT=1

evidence/561/baseline.txt

@@ -0,0 +1,115 @@
+
+ RUN  v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk
+
+(node:16742) [FLOWS_WORKER_LEASE_LOST] Warning: test-llm: run_id=01M390MBFVS1E4P7XHG40XXVT3 step_id=llm-1 attempt=1: WorkerLeaseLostError: Agent lease is already expired for 01M390MBFVS1E4P7XHG40XXVT3/llm-1.
+(Use `node --trace-warnings ...` to show where the warning was created)
+(node:16742) [FLOWS_WORKER_LEASE_LOST] Warning: test-llm: run_id=01M390NG2Q7NEX3EPDJJCAE5T7 step_id=llm-1 attempt=1: WorkerLeaseLostError: Agent lease is already expired for 01M390NG2Q7NEX3EPDJJCAE5T7/llm-1.
+ ❯ tests/authored-probe-cache.test.ts (5 tests | 5 failed) 86829ms
+   × authored run CLI probe cache > nine calls have no expired attempts at capacity 1 37494ms
+     → step_failed: journal step "llm-1" completed with lease_expired
+Inspect: flows replay 01M390MBFVS1E4P7XHG40XXVT3 --at llm-1
+   × authored run CLI probe cache > nine calls have no expired attempts at capacity 4 37529ms
+     → step_failed: journal step "llm-1" completed with lease_expired
+Inspect: flows replay 01M390NG2Q7NEX3EPDJJCAE5T7 --at llm-1
+   × authored run CLI probe cache > does not serialize nine starts behind nine probes, and probes again on a new run 6453ms
+     → expected 3250.5002949999907 to be less than 450
+   × authored run CLI probe cache > caches probe failures while refusing all nine calls 3299ms
+     → expected [ { kind: 'auth' }, …(8) ] to have a length of 1 but got 9
+   × authored run CLI probe cache > shares probe results across agent calls too 2052ms
+     → expected [ { kind: 'auth' }, …(2) ] to have a length of 1 but got 3
+
+⎯⎯⎯⎯⎯⎯⎯ Failed Tests 5 ⎯⎯⎯⎯⎯⎯⎯
+
+ FAIL  tests/authored-probe-cache.test.ts > authored run CLI probe cache > nine calls have no expired attempts at capacity 1
+AuthoredFlowExecutionError: step_failed: journal step "llm-1" completed with lease_expired
+Inspect: flows replay 01M390MBFVS1E4P7XHG40XXVT3 --at llm-1
+ ❯ stepFailure src/authored-step-output.ts:123:10
+    121|     ...edges,
+    122|   });
+    123|   return new AuthoredFlowExecutionError('step_failed', message, reason…
+       |          ^
+    124| }
+    125| 
+ ❯ Module.readCompletedStepOutput src/authored-step-output.ts:70:11
+ ❯ WorkerSlots.run src/worker-slots.ts:44:14
+ ❯ Object.llm src/authored-worker-step.ts:233:22
+ ❯ Module.observeStep src/progress.ts:48:20
+ ❯ AuthoredFlowOperation.begin src/authored-flow-operation.ts:174:23
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/5]⎯
+
+ FAIL  tests/authored-probe-cache.test.ts > authored run CLI probe cache > nine calls have no expired attempts at capacity 4
+AuthoredFlowExecutionError: step_failed: journal step "llm-1" completed with lease_expired
+Inspect: flows replay 01M390NG2Q7NEX3EPDJJCAE5T7 --at llm-1
+ ❯ stepFailure src/authored-step-output.ts:123:10
+    121|     ...edges,
+    122|   });
+    123|   return new AuthoredFlowExecutionError('step_failed', message, reason…
+       |          ^
+    124| }
+    125| 
+ ❯ Module.readCompletedStepOutput src/authored-step-output.ts:70:11
+ ❯ WorkerSlots.run src/worker-slots.ts:44:14
+ ❯ Object.llm src/authored-worker-step.ts:233:22
+ ❯ Module.observeStep src/progress.ts:48:20
+ ❯ AuthoredFlowOperation.begin src/authored-flow-operation.ts:174:23
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/5]⎯
+
+ FAIL  tests/authored-probe-cache.test.ts > authored run CLI probe cache > does not serialize nine starts behind nine probes, and probes again on a new run
+AssertionError: expected 3250.5002949999907 to be less than 450
+ ❯ tests/authored-probe-cache.test.ts:102:41
+    100|     expect(starts).toHaveLength(9);
+    101|     // F1 permits one synchronous probe (~300ms), not nine (~3s).
+    102|     expect(starts.at(-1)! - starts[0]!).toBeLessThan(450);
+       |                                         ^
+    103|     expectOneProbe(logs());
+    104|     await executeAuthoredFlow(nine, client, undefined, options);
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/5]⎯
+
+ FAIL  tests/authored-probe-cache.test.ts > authored run CLI probe cache > caches probe failures while refusing all nine calls
+AssertionError: expected [ { kind: 'auth' }, …(8) ] to have a length of 1 but got 9
+
+- Expected
++ Received
+
+- 1
++ 9
+
+ ❯ expectOneProbe tests/authored-probe-cache.test.ts:70:47
+     68| 
+     69| function expectOneProbe(logs: ReturnType<Awaited<ReturnType<typeof slo…
+     70|   expect(logs.filter(l => l.kind === 'auth')).toHaveLength(1);
+       |                                               ^
+     71|   expect(logs.filter(l => l.kind === 'identify').length - logs.filter(…
+     72| }
+ ❯ tests/authored-probe-cache.test.ts:121:5
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[4/5]⎯
+
+ FAIL  tests/authored-probe-cache.test.ts > authored run CLI probe cache > shares probe results across agent calls too
+AssertionError: expected [ { kind: 'auth' }, …(2) ] to have a length of 1 but got 3
+
+- Expected
++ Received
+
+- 1
++ 3
+
+ ❯ expectOneProbe tests/authored-probe-cache.test.ts:70:47
+     68| 
+     69| function expectOneProbe(logs: ReturnType<Awaited<ReturnType<typeof slo…
+     70|   expect(logs.filter(l => l.kind === 'auth')).toHaveLength(1);
+       |                                               ^
+     71|   expect(logs.filter(l => l.kind === 'identify').length - logs.filter(…
+     72| }
+ ❯ tests/authored-probe-cache.test.ts:134:5
+
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[5/5]⎯
+
+ Test Files  1 failed (1)
+      Tests  5 failed (5)
+   Start at  06:11:11
+   Duration  88.74s (transform 1.05s, setup 0ms, collect 1.73s, tests 86.83s, environment 0ms, prepare 44ms)
+

evidence/561/cache-keys.txt

@@ -0,0 +1,10 @@
+
+ RUN  v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk
+
+ ✓ tests/preflight-run-cache.test.ts (1 test) 12ms
+
+ Test Files  1 passed (1)
+      Tests  1 passed (1)
+   Start at  06:15:07
+   Duration  1.02s (transform 390ms, setup 0ms, collect 836ms, tests 12ms, environment 0ms, prepare 43ms)
+

evidence/561/check-live-journals.py

@@ -0,0 +1,23 @@
+import glob
+import json
+import sqlite3
+
+for directory in ['/tmp/flows-561-live-default', '/tmp/flows-561-live-one']:
+    journals = sorted(glob.glob(directory + '/runs/*.sqlite3'))
+    assert journals
+    llms = 0
+    for path in journals:
+        with sqlite3.connect('file:' + path + '?mode=ro', uri=True) as db:
+            rows = db.execute('select entry_type, step_id, payload from entries').fetchall()
+        assert all('lease_expired' not in payload for _, _, payload in rows), path
+        starts = [step for kind, step, _ in rows if kind == 'step.attempt.started' and step.startswith('llm-')]
+        if starts:
+            assert len(starts) == 1, (path, starts)
+            llms += 1
+        for kind, step, payload in rows:
+            if kind == 'step.completed' and step and step.startswith('llm-'):
+                value = json.loads(payload)
+                assert value['completionReason'] == 'success', value
+                print(step, value['output'])
+    assert llms == 9, llms
+    print(directory, f'{len(journals)} journals, {llms} LLM children, one attempt each, no lease_expired')

evidence/561/fixed.txt

@@ -0,0 +1,15 @@
+
+ RUN  v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk
+
+ ✓ tests/authored-probe-cache.test.ts (5 tests) 22896ms
+   ✓ authored run CLI probe cache > nine calls have no expired attempts at capacity 1 7405ms
+   ✓ authored run CLI probe cache > nine calls have no expired attempts at capacity 4 5425ms
+   ✓ authored run CLI probe cache > does not serialize nine starts behind nine probes, and probes again on a new run 8357ms
+   ✓ authored run CLI probe cache > caches probe failures while refusing all nine calls 401ms
+   ✓ authored run CLI probe cache > shares probe results across agent calls too 1306ms
+
+ Test Files  1 passed (1)
+      Tests  5 passed (5)
+   Start at  06:13:43
+   Duration  24.51s (transform 872ms, setup 0ms, collect 1.45s, tests 22.90s, environment 0ms, prepare 46ms)
+

evidence/561/flows.json

@@ -0,0 +1 @@
+{"cli":"claude","models":["claude-haiku-4-5-20251001"]}