Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docs/BUDGET.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Budget headers and spend

`budget` is optional. A flow may declare `"$0.10/run"`, `"$20/day"`, or
`{ tokens: 10000, dollars: 0.10, wallclock: "2m" }`. Objects default to a run
window. Tokens count input plus output; wallclock sums attempt durations from
journaled start to completion. Duration units are `ms`, `s`, `m`, `h`, and `d`.
Limits are non-negative; dollars support up to six decimal places.

A day is a UTC calendar day within a run. Completed spend resets for admission
at the next UTC day; unrelated runs do not share a global account. Header
syntax errors refuse as `budget_syntax_invalid`. Declared models without a
frozen price refuse as `budget_missing_price`; dollar budgets also require a
model on each worker step. Existing project model allowlist checks still apply.

Every newly written `step.completed` includes:

```json
"spend": {
"tokens_input": 1000,
"tokens_output": 200,
"dollars": 0.006,
"wallclock_ms": 25
}
```

The frozen table in `packages/sdk/src/model-pricing.ts` quotes dollars per
million tokens. Workers compute integer microdollars:
`inputTokens * inputPrice + outputTokens * outputPrice`. The existing `budget`
field retains exact decimal dollars; only the journal's `spend.dollars` becomes
a JSON number. Memory-injected costs retain their existing single charge.
Deterministic steps have zero model tokens and dollars, with measured duration.

The kernel checks accumulated spend before each new attempt. Equality is
permitted. Crossing a limit keeps that completion valid and refuses the next
start with `run.completed.completionReason: "budget_exceeded"`. Running peers
may finish; no new peers start. A final successful step may cross a limit and
still complete its run successfully. Replay reconstructs accounting from the
journal, including retries and prior epochs.

The internal TypeScript executor currently lowers each authored step to its
own kernel run. For budgeted flows it serializes admission and carries exact
journaled costs into the next run's `budget.prior_spend`. Its generated terminal
marker does not consume the author's step budget. This does not add a durable
TypeScript root or change that runner's existing resume contract.

Raw Claude/Codex adapters request structured output to extract usage. A custom
wrapper may return an explicit result envelope after its execution handshake:

```json
{"protocol":"relayflows-agent-cli-v1-result","output":"answer","usage":{"input_tokens":1000,"output_tokens":200}}
```

A priced model with missing or malformed usage produces a journaled worker
error. Legacy `maxTokensIn` / `maxTokensOut` / `maxDollars` envelopes keep their
worker-supplied pricing contract, including existing synthetic test models.
New surface headers carry `pricing: "frozen"` through compiled artifacts so
checking a compiled flow preserves the same missing-price refusal.

`testdata/budget-guarded.flow.yaml` is a local smoke using a zero-duration budget:
its first completed command exceeds the ceiling and its dependent command
never starts.
2 changes: 1 addition & 1 deletion docs/SURFACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ No process runs between events: the handler wakes, executes to its next await, p
[the generator notes](../packages/surface/src/helpers/README.md).

4. **`{{prev}}` / return-value chaining.** Output flows downward implicitly; naming steps is for reaching back, not bookkeeping.
5. **Headers are optional escalation.** identity, memory, budget, tools appear only when used. The empty header is the common case.
5. **Headers are optional escalation.** identity, memory, budget, tools appear only when used. The empty header is the common case. [Budget headers and spend](BUDGET.md) specifies parsing, prices, journal attribution, and admission limits.
6. **Agent definitions escalate by composition** — and a reusable agent *is* a flow:
```yaml
- agent: Review this diff for security issues. # 1. anonymous
Expand Down
36 changes: 36 additions & 0 deletions evidence/spec-G-budget/PR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
Title: feat(surface,sdk,kernel): budget header + spend attribution (SURFACE §2 rule 5)

The `budget:` header was declared in surface examples but never enforced. This
change accepts string and object headers, refuses invalid syntax and missing
model prices before execution, and records tokens, dollars, and elapsed attempt
time on every completion. Once completed spend crosses a limit, the kernel
preserves completed work and refuses the next start with `budget_exceeded`.

Parent issue: the lead will insert the issue link when opening this PR.

The SDK uses the frozen model-pricing table and integer microdollars; the kernel
keeps exact decimal accounting through replay. UTC day windows, total-token and
wallclock limits are supported. The internal authored executor carries journaled
spend between its existing per-step kernel runs. Legacy explicit envelopes keep
their worker-supplied price contract. See [the budget contract](docs/BUDGET.md).

Fixture: [budget-guarded.flow.yaml](testdata/budget-guarded.flow.yaml). Its
`measured` command completes; `guarded` never starts. Actual journal excerpt:

```json
{"entry_type":"step.completed","step_id":"measured","payload":{"completionReason":"success","spend":{"tokens_input":0,"tokens_output":0,"dollars":0,"wallclock_ms":19}}}
{"entry_type":"run.completed","payload":{"completionReason":"budget_exceeded"}}
```

Validation commands and complete captured output are in
[evidence/spec-G-budget](evidence/spec-G-budget/README.md). Kernel workspace,
surface tests, and focused SDK tests pass. Plain `npm test` remains blocked by
this host's missing Claude login for the existing live analyzer acceptance test;
that failure has not been skipped or weakened. A timing-sensitive existing
lease-wait test failed in one full run and passed in the subsequent focused run.
The two new exhaustive refusal scenarios were added with lead approval; the
existing assertion is unchanged.

The lead owns PR creation, CI verification, and bot review because GitHub
credentials on the worker return HTTP 401. Do not mark ready for merge until
those checks and the live analyzer acceptance requirement are satisfied.
97 changes: 97 additions & 0 deletions evidence/spec-G-budget/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Spec G verification and handoff

This branch implements budget headers, frozen SDK pricing, per-completion spend,
and kernel admission limits. The lead approved adding two scenarios to the
existing exhaustive refusal test without changing its assertion, and preserving
legacy explicit-budget synthetic model behavior (Relay message
`224082264437923840`). The same message directs the worker to push and hand off
PR creation because fleet GitHub credentials return HTTP 401.

## Commands and captured output

Only trailing whitespace was normalized in the captured text logs for git.

From `kernel`:

```sh
PATH=/Users/khaliqgant/.cargo/bin:$PATH cargo test --workspace
```

Exit 0. Full literal output: [cargo-test.log](cargo-test.log). New integration
coverage output:

```text
running 3 tests
test daily_windows_reset_and_exact_limits_do_not_refuse ... ok
test crossing_completion_is_durable_and_next_step_is_refused ... ok
test deterministic_spend_and_wallclock_limit_gate_parallel_batch_starts ... ok
```

From `packages/sdk`:

```sh
PATH=/Users/khaliqgant/.cargo/bin:$PATH npm test
```

The full suite is not green on this host. Full output is retained in
[sdk-npm-test.log](sdk-npm-test.log):

```text
Test Files 1 failed | 53 passed | 1 skipped (55)
Tests 1 failed | 984 passed | 3 skipped (988)
```

The real analyzer requires a Claude login;
`claude auth status` returned exit 1; its full output is
[claude-auth-status.log](claude-auth-status.log).

An [earlier full run](sdk-npm-test-earlier.log) also failed the existing lease-wait timing assertion. The
entire CLI test file subsequently passed in the focused rerun below. Neither
assertion was modified and the analyzer was not skipped with an environment
flag.

```sh
PATH=/Users/khaliqgant/.cargo/bin:$PATH ./node_modules/.bin/vitest run tests/cli.test.ts tests/budget-preflight.test.ts tests/budget-attribution.test.ts tests/budget-authored-live.test.ts tests/preflight.test.ts
```

Final focused output: [sdk-focused.log](sdk-focused.log).

```text
Test Files 5 passed (5)
Tests 109 passed (109)
```

From `packages/surface`, `npm test` could not launch because `bun` is absent.
Its build, test typecheck, and test commands were run directly:

```sh
npm run build
./node_modules/.bin/tsc -p tsconfig.test.json
./node_modules/.bin/vitest run
```

All three exited 0; test output: [surface-test.log](surface-test.log).

```text
Test Files 1 passed (1)
Tests 7 passed (7)
```

## Declarative smoke

The fixture `testdata/budget-guarded.flow.yaml` was compiled with the SDK's
`compileYaml` and `toKernelSpec` into `/tmp/spec-g-smoke.json`, then run from the
repository root:

```sh
kernel/target/debug/relayflowd --data-dir /tmp/spec-g-smoke-data run /tmp/spec-g-smoke.json
```

Exit 1 is the expected budget refusal. The emitted spec and captured outcome are
[smoke.spec.json](smoke.spec.json) and [smoke.log](smoke.log). The actual SQLite
journal excerpts are [smoke-journal.json](smoke-journal.json): `measured` completed
successfully with 19ms spend; `guarded` never started; the run completed with
`budget_exceeded`.

PR creation, CI checks and review-bot triage are handed to the lead. This evidence
does not claim required CI passed or that the branch is ready for merge.
Loading
Loading