Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .shakedown-actions/ACTIONS.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
flows OPEN-PR fix/cli-help-shakedown-0910 .shakedown-actions/help-pr-body.md
flows OPEN-PR fix/docs-inline-model-shakedown-0910 .shakedown-actions/docs-pr-body.md
flows FILE-ISSUE .shakedown-actions/issues/llm-local.title.txt .shakedown-actions/issues/llm-local.md
flows FILE-ISSUE .shakedown-actions/issues/yaml-worker.title.txt .shakedown-actions/issues/yaml-worker.md
flows FILE-ISSUE .shakedown-actions/issues/yaml-binding.title.txt .shakedown-actions/issues/yaml-binding.md
flows FILE-ISSUE .shakedown-actions/issues/runtime-diagnostic.title.txt .shakedown-actions/issues/runtime-diagnostic.md
flows FILE-ISSUE .shakedown-actions/issues/dependency-audit.title.txt .shakedown-actions/issues/dependency-audit.md
flows FILE-ISSUE .shakedown-actions/issues/observer-origins.title.txt .shakedown-actions/issues/observer-origins.md
1 change: 1 addition & 0 deletions .shakedown-actions/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
COMPLETED: all actions delivered by the coordinator. Do not re-run ACTIONS.txt. Actual issue/PR/comment URLs and public API verification are in evidence/shakedown-0910/DELIVERY.md and publication.json. Report PR #281 is a draft. No merge was performed.
21 changes: 21 additions & 0 deletions .shakedown-actions/docs-pr-body.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The surface docs still say all models require a project registry, contradicting #266. Document the actual split: inline named-agent models reach the live readiness probe when no `flows.json` exists, while direct step models still need an allowlist; an existing config still enforces its policy.

Validated against `preflight.ts:180` and two actual `flows run` outcomes from the launch shakedown (both outside any config ancestry):

```text
agent-inline.flow.yaml: EXIT 2
REFUSED [cli_unauthenticated] Step "draft" declares CLI "claude", but "claude auth status" exited non-zero; authenticate it or repair that adapter's authentication probe.

chained.flow.yaml with model directly on llm step: EXIT 2
REFUSED [model_unknown] Step "extract" declares model "claude-sonnet-4-6" for CLI "claude", but it is not listed in the nearest project config (no model registry was found); add the exact model only after verifying that project is allowed to use it.
```

A forward-only PATH shim captured the real inline-model probe before the auth refusal:

```text
auth status --help
-p --model claude-sonnet-4-6 --tools --no-session-persistence Reply with exactly RELAYFLOWS_MODEL_READY and nothing else.
auth status
```

`git diff --check` exited 0 with no output. Documentation-only change.
127 changes: 127 additions & 0 deletions .shakedown-actions/help-pr-body.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
`flows --help` and `flows -h` now print one usage form per line to stdout and exit 0; unknown options still exit 2. Single-step summaries now say `(1 step)` instead of `(1 steps)`.

Reproduced during the launch shakedown on main a42ca16. This small PR is based directly on main; it does not modify #268/#269 branches.

Validation on this change (full captured output in the shakedown report):

```text
CHECKOUT: /Users/khaliqgant/flows-help-shakedown-0910
COMMAND: npx tsc --noEmit
EXIT: 0

```

```text
CHECKOUT: /Users/khaliqgant/flows-help-shakedown-0910
COMMAND: npx vitest run tests/cli.test.ts
EXIT: 0

RUN v2.1.9 /Users/khaliqgant/flows-help-shakedown-0910/packages/sdk

✓ tests/cli.test.ts (63 tests) 2691ms
✓ flows check CLI > binds a checked relative wrapper to the flow directory for worker execution 704ms

Test Files 1 passed (1)
Tests 63 passed (63)
Start at 16:21:14
Duration 3.14s (transform 133ms, setup 0ms, collect 236ms, tests 2.69s, environment 0ms, prepare 34ms)


```

```text
COMMAND: node packages/sdk/dist/cli.js --help
EXIT: 0
STDOUT:
Usage:
flows check [--json] <flow.yaml|spec.json>
flows run [--json] [--no-spawn] [--data-dir <dir>] <flow.yaml|spec.json>
flows run --cloud [--json] [--wait] <flow.yaml|spec.json>
flows run [--json] [--no-spawn] [--data-dir <dir>] [--local-agent] <flow.ts> --input <inline-json-or-file>
flows tick start --schedule-id <id> --interval-ms <ms> [--epoch-ms <ms>] [--max-catch-up <n>] [--poll-interval-ms <ms>] [--data-dir <dir>] <spec.json>
flows resume [--json] [--no-spawn] [--data-dir <dir>] <run-id>
flows hn-monitor start [--data-dir <dir>] [--poll-interval-ms <n>] <spec.json>

STDERR:

COMMAND: node packages/sdk/dist/cli.js -h
EXIT: 0
STDOUT:
Usage:
flows check [--json] <flow.yaml|spec.json>
flows run [--json] [--no-spawn] [--data-dir <dir>] <flow.yaml|spec.json>
flows run --cloud [--json] [--wait] <flow.yaml|spec.json>
flows run [--json] [--no-spawn] [--data-dir <dir>] [--local-agent] <flow.ts> --input <inline-json-or-file>
flows tick start --schedule-id <id> --interval-ms <ms> [--epoch-ms <ms>] [--max-catch-up <n>] [--poll-interval-ms <ms>] [--data-dir <dir>] <spec.json>
flows resume [--json] [--no-spawn] [--data-dir <dir>] <run-id>
flows hn-monitor start [--data-dir <dir>] [--poll-interval-ms <n>] <spec.json>

STDERR:

COMMAND: node packages/sdk/dist/cli.js --definitely-invalid
EXIT: 2
STDOUT:

STDERR:
REFUSED [invalid_invocation] Usage:
flows check [--json] <flow.yaml|spec.json>
flows run [--json] [--no-spawn] [--data-dir <dir>] <flow.yaml|spec.json>
flows run --cloud [--json] [--wait] <flow.yaml|spec.json>
flows run [--json] [--no-spawn] [--data-dir <dir>] [--local-agent] <flow.ts> --input <inline-json-or-file>
flows tick start --schedule-id <id> --interval-ms <ms> [--epoch-ms <ms>] [--max-catch-up <n>] [--poll-interval-ms <ms>] [--data-dir <dir>] <spec.json>
flows resume [--json] [--no-spawn] [--data-dir <dir>] <run-id>
flows hn-monitor start [--data-dir <dir>] [--poll-interval-ms <n>] <spec.json>

```

```text
CWD: /Users/khaliqgant/flows-help-shakedown-0910
COMMAND: node /Users/khaliqgant/flows-help-shakedown-0910/packages/sdk/dist/cli.js run /Users/khaliqgant/flows-shakedown-0910/testdata/shakedown/hello-world.flow.yaml --data-dir /tmp/rfd-96g6toyf
ENV OVERRIDES: {}
EXIT: 0
ELAPSED: 0.249s
STDOUT:
RUN 01M25V99YWG6MCDTKWNG9PB3HK completed (1 step) completionReason: success

STDERR:
WARNING [unprovable_effects] Step "hello" command "printf" resolves, but its effects cannot be proven before execution.


```

```text
CWD: /Users/khaliqgant/flows-help-shakedown-0910
COMMAND: git rev-parse HEAD
EXIT: 0
179092173be91d4e4fcc9245be17371bca08360a

CWD: /Users/khaliqgant/flows-help-shakedown-0910
COMMAND: git ls-remote origin refs/heads/fix/cli-help-shakedown-0910
EXIT: 0
179092173be91d4e4fcc9245be17371bca08360a refs/heads/fix/cli-help-shakedown-0910


```

`cargo test -p relayflowd` exited 0. Final literal output:

```text

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.16s

Running tests/subscription_liveness.rs (target/debug/deps/subscription_liveness-9506d11f678a9586)

running 3 tests
test submit_event_upserts_subscription_row_and_sweep_flags_it_stale_after_budget ... ok
test stale_transition_is_journaled_as_subscription_stale_entry_in_the_last_known_run ... ok
test a_fresh_arrival_re_arms_the_latch_and_the_next_silence_can_stale_again ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s

Doc-tests relayflowd

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

```
41 changes: 41 additions & 0 deletions .shakedown-actions/issues/dependency-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
## Summary

**Low launch severity for this local CLI shakedown; reachability not assessed.** The documented fresh `packages/sdk` install succeeds but immediately reports `6 vulnerabilities (4 moderate, 1 high, 1 critical)`. This is visible first-build friction and merits dependency maintenance. The high/critical findings are in development tooling, not the production-only audit.

## Repro

From main `a42ca16` plus #268/#269, using Node 25.8.1:

```sh
cd packages/sdk
npm ci
npm audit --json
npm audit --omit=dev --json
```

Install output:

```text
added 58 packages, and audited 59 packages in 1s

16 packages are looking for funding
run `npm fund` for details

6 vulnerabilities (4 moderate, 1 high, 1 critical)
```

Full audit: `vitest` critical, `vite` high, `@vitest/mocker`, `vite-node`, `esbuild`, and `ajv` moderate. The suggested test-tooling upgrade crosses a major version to Vitest 4.1.11. Production-only audit contains just the moderate Ajv `$data` ReDoS advisory (GHSA-2g4f-4pwh-qvx6). This report does not establish exploitability or assert that `$data` is enabled.

## Expected

The documented clean SDK build has an evaluated dependency baseline and no unexplained high/critical install banner.

## Suggested direction

Evaluate compatible test-tooling and Ajv updates, capture reachability where upgrades must be deferred, and rerun the SDK tests. Do not use blanket audit suppression or `npm audit fix --force` without checking the resulting major-version changes.

## Acceptance criteria

- Fresh install and full/production-only audits are captured after maintenance.
- The selected dependency changes pass SDK type checks and relevant tests.
- Any remaining advisory has an explicit, evidence-based disposition.
1 change: 1 addition & 0 deletions .shakedown-actions/issues/dependency-audit.title.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
flows: fresh SDK npm ci reports six dependency advisories
59 changes: 59 additions & 0 deletions .shakedown-actions/issues/llm-local.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
## Summary

**BLOCKER for the 2026-09-11 launch's requested `llm → agent → deterministic` demo.** With a real authenticated Codex CLI, declarative YAML parks at its first `llm` step with no worker; authored TypeScript with `--local-agent` reports `unsupported_verb` for `f.llm`. A standalone TypeScript `f.agent` using the same installation succeeds, so this is not a missing provider login.

Observed against main `a42ca16` composed with #268 `33c460a` and #269 `f3dc7ce`, on macOS arm64 / Node 25.8.1.

## Repro

Build kernel and SDK, then run this file outside any `flows.json` ancestry with a fresh data directory and an authenticated `codex` on PATH:

```yaml
version: "0.1.0"
name: llm-first
steps:
- id: extract
type: llm
cli: codex
prompt: 'Return only JSON: {"message":"hello"}'
output:
type: object
required: [message]
properties:
message: {type: string}
```

```sh
node packages/sdk/dist/cli.js run /tmp/llm-first.flow.yaml --data-dir /tmp/llm-first-fresh
```

Captured from the three-step variant:

```
EXIT: 3
RUN 01M25TMMN68GZPYJ7JK85CSH2B parked (0 steps)
PARKED [run_parked] Run "01M25TMMN68GZPYJ7JK85CSH2B" parked at step "extract" (llm): no worker is attached for step type "llm".
```

Trying the imperative path (`await f.llm(...)`) with installed `@relayflows/surface` and `flows run chained.flow.ts --local-agent --input '{}'` instead gives:

```
EXIT: 1
FAILED [protocol_error] relayflowd could not complete the run request: unsupported_verb: the initial authored executor does not lower f.llm
```

## Expected

The supported local CLI can execute a real `llm` request, validate its JSON output, and advance through the advertised three-step chain. If a worker must be started separately, ship and document the exact supported command and refuse before creating a run when a required executor is absent (RFC-0001 covenant 2).

## Suggested direction

Implement authored `f.llm` lowering and a real local LLM worker/runner path using existing kernel `llm` dispatch and typed completion. Cover the declarative entry point as well. Do not disguise this as `agent` or remove the typed gate to get a green demo.

## Acceptance criteria

- A documented local command runs `llm → agent → deterministic` with real provider output and a fresh data directory.
- Invalid JSON and schema mismatch fail with a typed completion reason.
- The deterministic final step consumes verified upstream data and writes the expected artifact.
- Missing runtime capabilities are reported before run submission with a concrete remediation command.
- Capture a real-provider CLI transcript; a stubbed worker alone is not acceptance evidence.
1 change: 1 addition & 0 deletions .shakedown-actions/issues/llm-local.title.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
flows: flagship llm chain has no executable local path in YAML or TypeScript
12 changes: 12 additions & 0 deletions .shakedown-actions/issues/observer-264-comment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Launch shakedown against #269 `f3dc7ce` found a real endpoint mismatch, independent of the stale-cookie browser issue. Please handle on the existing observer branch; this shakedown is not editing that PR.

1. `RELAYCAST_WORKSPACE_KEY=rk_live_test flows run observer.flow.yaml` succeeds as a flow but emits `[observer] token mint failed: mint API returned HTTP 404; skipping observer link`. Default `https://agentrelay.com/v1/observer-tokens` is the dashboard origin, not the canonical API.
2. The active workspace reports `relaycastUrl: https://cast.agentrelay.com`. Setting `RELAYCAST_API_URL` to that URL makes the dummy key fail with the expected HTTP401.
3. With the existing real canonical workspace key (never logged), minting succeeds, but the CLI prints `Observer: https://cast.agentrelay.com/observer?key=ot_live_REDACTED`.
4. Actually loading that printed URL with `curl -L` returns **HTTP404**, body `{"ok":false,"error":{"code":"not_found","message":"Route not found"}}`.

The code derives both `/v1/observer-tokens` and `/observer` from the same base. The API and dashboard are different origins in the canonical deployment. Acceptance: separate the API mint origin from the human dashboard origin, and load a freshly printed real-key URL before declaring the link delivered. Dummy-key failure must remain best-effort and leave a successful local run at exit0.

At this tested head, `flows observer` itself is still absent (`REFUSED [invalid_invocation]`, exit2); I will retest if the follow-up lands. A first real-key mint attempt also hit HTTP429, so the final successful mint was done after backing off.

Follow-up at `0a0bca4`: rebuilt and exercised new `flows observer` and the actual existing cloud-login store. With no env key, the default mint still404s; canonical API override mints successfully, and the printed URL still loads404. Suppressed and absent-store refusals are actionable. `npx tsc --noEmit` passes. `npx vitest run tests/observer-link.test.ts` is **34 passed / 4 failed** when composed with #268: `startCliLoopback` still binds `join(dataDir, 'relayflowd.sock')` at line74, so the CLI connects to a freshly spawned real daemon on its hashed path instead of the test loopback; expected sentinel run IDs are replaced by real ULIDs. Please reconcile test socket setup with #268 on your branch.
53 changes: 53 additions & 0 deletions .shakedown-actions/issues/observer-origins.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
## Summary

**High launch severity; #269 remains with its existing owner.** The default API base is the dashboard origin (`https://agentrelay.com`), so minting returns 404. Overriding to the canonical API (`https://cast.agentrelay.com`) allows minting, but the generated dashboard URL uses that API origin too and also returns 404 when loaded.

Confirmed both on initial #269 `f3dc7ce` and its on-demand/cloud-login follow-up `0a0bca4`, composed with main `a42ca16` and #268 `33c460a`.

## Repro

```sh
RELAYCAST_WORKSPACE_KEY=rk_live_test flows observer
```

```
EXIT: 2
REFUSED [observer_link_unavailable] mint API returned HTTP 404
```

The canonical API reported by `agent-relay workspace active --json` is `https://cast.agentrelay.com`. A dummy key there gives HTTP401 as expected. With an existing real cloud-login workspace key and **no** RELAYCAST_WORKSPACE_KEY env var:

```sh
RELAYCAST_API_URL=https://cast.agentrelay.com flows observer
```

```
EXIT: 0
https://cast.agentrelay.com/observer?key=ot_live_REDACTED
```

Actually loading the printed URL:

```text
curl --max-time 15 -sS -L '<printed URL>'
HTTP: 404
{"ok":false,"error":{"code":"not_found","message":"Route not found"}}
```

`flows run` also prints an unusable URL; a mint failure itself correctly leaves a successful deterministic run at exit0. Live credential material is intentionally redacted.

## Expected

Default cloud-login or explicit-key observer commands mint against the canonical API and emit a dashboard URL that actually loads.

## Suggested direction

Separate the mint API base and dashboard URL. Default the API to the canonical Relaycast service and the dashboard to its UI origin; respect deployment overrides independently. Do not conflate this with the separate stale-cookie fix.

## Acceptance criteria

- Dummy key against the default mint endpoint yields an auth failure, not 404.
- A real existing workspace key mints a URL through both `flows run` and `flows observer`.
- Load the printed URL with a browser or curl and capture a non-404 dashboard response.
- Cloud-login fallback works without exposing the admin key in URLs/logs.
- Observer failures remain nonfatal to otherwise successful runs.
1 change: 1 addition & 0 deletions .shakedown-actions/issues/observer-origins.title.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
flows: observer API and dashboard origins produce HTTP 404
49 changes: 49 additions & 0 deletions .shakedown-actions/issues/runtime-diagnostic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
## Summary

**High launch friction.** When a deterministic command exits nonzero, `flows run` reports only the run-level `step_failed`. The user's stderr and the actual exit code are absent, and the message gives no inspection command. A first-time author must reach into the journal to learn why a command failed.

## Repro

```yaml
version: "0.1.0"
name: runtime-error
steps:
- id: fail-command
type: deterministic
command: 'printf "shakedown intentional failure" >&2; exit 7'
```

Run with a fresh data directory:

```sh
flows run runtime-error.flow.yaml --data-dir /tmp/runtime-error-fresh
```

Captured output (main `a42ca16` + #268/#269):

```text
EXIT: 1
STDOUT:
RUN 01M25VBQ8W3TCRWTM15KYAM7AA failed (1 steps) completionReason: step_failed

STDERR:
WARNING [unprovable_effects] Step "fail-command" command "printf" resolves, but its effects cannot be proven before execution.
FAILED [step_failed] Run "01M25VBQ8W3TCRWTM15KYAM7AA" failed with completionReason: step_failed.
```

Neither `exit 7` nor `shakedown intentional failure` appears. The warning happens to name the step here, but it is unrelated to the failure and cannot substitute for a failed-step diagnostic.

## Expected

The refusal/failure output names the failed step and exit code, includes an appropriately bounded stderr tail, and gives an executable inspection command when additional context is available. Preserve the typed run completion reason.

## Suggested direction

Read the journaled failed attempt result at the CLI reporting boundary and render the already captured result, applying the project's output bounds and credential-redaction conventions. Do not re-run the command to diagnose it.

## Acceptance criteria

- The repro identifies `fail-command`, exit code 7, and its captured stderr.
- Successful summaries remain concise.
- `--json` exposes the same useful failure context in structured form.
- Long stderr is bounded, with clear truncation and an inspection path.
1 change: 1 addition & 0 deletions .shakedown-actions/issues/runtime-diagnostic.title.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
flows: failed deterministic runs hide the command exit code and stderr
Loading
Loading