Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,20 @@ hooks:
fi
```

Post-hooks also receive the run's recorded usage without parsing `metadata.json`: `CONVOY_RUN_COST` (executor plus advisor spend) and `CONVOY_RUN_ADVISOR_COST` are USD decimals with exactly four fractional digits; `CONVOY_RUN_TOKENS_INPUT`, `CONVOY_RUN_TOKENS_OUTPUT`, `CONVOY_RUN_TOKENS_REASONING`, `CONVOY_RUN_TOKENS_CACHE_READ`, `CONVOY_RUN_TOKENS_CACHE_WRITE`, and `CONVOY_RUN_TOKENS_TOTAL` are integer token counts; and `CONVOY_RUN_DURATION_MS` is the integer sum of recorded phase durations. Cost appears only after a phase reports executor or advisor spend, token variables appear together only after a phase reports executor usage, advisor cost appears only when advisor spend is positive, and duration appears only after a phase records one — absent facts are not represented as zero. For example, a post-hook can comment the result on its PR and enforce a decimal budget:

```yaml
hooks:
post:
- name: report run usage
command: gh pr comment --body "Convoy run: \$${CONVOY_RUN_COST} in ${CONVOY_RUN_DURATION_MS}ms"
- name: enforce budget
when: always
command: 'awk "BEGIN { exit !(${CONVOY_RUN_COST:-0} <= 5.0000) }"'
```

Use `awk` for the budget guard because shell integer comparisons cannot compare decimal USD amounts. A failing post-hook fails the run unless `continueOnError: true` is set.

Success post-hooks also receive the compaction outcome — `CONVOY_FINALIZATION_STATE` plus the produced commit (`CONVOY_FINALIZATION_SHA`, `CONVOY_FINALIZATION_SUBJECT`) and `CONVOY_FINALIZATION_REASON` when it did not complete — so a publish hook can require `completed` before pushing.

`convoy publish` is the explicit headless publication request. It composes the same run-aware title and Why / What / How-tested body the dashboard's **Create PR** action composes — the branch's conventional prefix plus the OpenSpec proposal title, grounded in the proposal, the run recap (`reports/run-report.md`), and the validation reports — prints the disclosed branch/remote/base and the exact text, and pushes and creates the PR **only under explicit `--yes`**. Without `--yes`, or with `--dry-run`, it prints the review and performs no effect, so a run completing never publishes on its own. It is deliberately distinct from `convoy worktrees pr`, which is worktree-only and composes a generic body from the branch slug and commit subjects.
Expand All @@ -386,7 +400,6 @@ convoy publish --run-dir ~/.convoy/runs/<id> --worktree . --dry-run
# compose, push, and create/report the PR
convoy publish --run-dir ~/.convoy/runs/<id> --worktree . --yes
```

The dashboard shows the goal, the current iteration, and the trajectory (`◆ convoy · goal 90 · iter 2/4 · 71 → …`), and when the cycle ends — goal met, plateau, iteration cap, no score, or a failure — the dashboard holds its finish screen **once**, with the verdict in place of the live goal readout (`✓ goal 92/100`, `plateau 86/100`, `cap 88/100`, `no score`, or `✗ run failed`) and the full trajectory (`71 → 84 → 92`); the terminal prints the trajectory and why it stopped after the dashboard closes. Goal fragment phases appear under the parent pipeline with their iteration-qualified names (for example `goal-measure-1-score-report`); the whole cycle runs in one run, so the dashboard never remounts between rounds.

## Requirements
Expand Down Expand Up @@ -850,7 +863,7 @@ The rules:
- **Resume is frozen**: the resolved pipeline is persisted in the run's `metadata.json`; `--resume` replays it even if the config changed since.
- **Dirty-tree recovery**: a writable phase interrupted before its commit (Ctrl+C, a failed commit step, a killed process) leaves uncommitted work in the tree, which normally blocks `--resume`. In an interactive terminal, resume offers to commit that work as the interrupted phase (`convoy(<phase>): …` with the resumed run's `Convoy-Run` trailer), mark it done, and continue with the following phases. If the interrupted phase had already accepted a structured commit description through `write_report`, recovery reuses it; otherwise the message describes the staged paths or says plainly what happened. Read-only phases are never recoverable as agent output: preserved changes must be resolved manually, and resume also verifies their recorded HEAD/branch baseline. Decline (or a non-TTY resume) keeps the old "commit/stash first" behavior.
- **Permissions are additive**: `permissions.deny` extends the hard denylist, `permissions.allow` extends the allowlist, deny always wins, and there is deliberately no way for a repo to grant itself `--yolo`.
- **Hooks are trusted local shell commands**: `hooks.pre` runs after the run workspace/dashboard is initialized and before the pipeline starts (pre-hooks are skipped on `--resume`); `hooks.post` runs at the end according to `when`. Top-level hooks apply to every pipeline, and `hooks.pipelines.<name>` entries are appended for that pipeline. Hooks run via `$SHELL -lc` from the target repo by default, receive `CONVOY_RUN_ID`, `CONVOY_RUN_DIR`, `CONVOY_TARGET_DIR`, `CONVOY_PIPELINE`, `CONVOY_PROMPT_FILE`, and post-hooks also receive `CONVOY_RUN_STATUS`, plus `CONVOY_RUN_SCORE` on a scored pipeline and `CONVOY_GOAL_REACHED`/`CONVOY_GOAL_SCORE`/`CONVOY_GOAL_TARGET` when a [goal loop](#goal-mode) ran (in which case post-hooks run once, after the loop, not once per iteration). Success post-hooks run **after** automatic compaction and receive its outcome as `CONVOY_FINALIZATION_STATE` (plus `CONVOY_FINALIZATION_SHA`, `CONVOY_FINALIZATION_SUBJECT`, and `CONVOY_FINALIZATION_REASON` when set), so a publish hook can gate on a completed compaction. A failing hook fails the run unless `continueOnError: true` is set. Each hook is also a row in the dashboard pipeline — pre-hooks ahead of the steps and post-hooks after the `Compact run` row — with live running/✓/✗/skipped status, and the tail of its output lands in that row's `logs` tab; the rows and their captured output are recorded in the run metadata, so re-opened runs show them too.
- **Hooks are trusted local shell commands**: `hooks.pre` runs after the run workspace/dashboard is initialized and before the pipeline starts (pre-hooks are skipped on `--resume`); `hooks.post` runs at the end according to `when`. Top-level hooks apply to every pipeline, and `hooks.pipelines.<name>` entries are appended for that pipeline. Hooks run via `$SHELL -lc` from the target repo by default, receive `CONVOY_RUN_ID`, `CONVOY_RUN_DIR`, `CONVOY_TARGET_DIR`, `CONVOY_PIPELINE`, `CONVOY_PROMPT_FILE`, and post-hooks also receive `CONVOY_RUN_STATUS`, plus `CONVOY_RUN_SCORE` on a scored pipeline, `CONVOY_GOAL_REACHED`/`CONVOY_GOAL_SCORE`/`CONVOY_GOAL_TARGET` when a [goal loop](#goal-mode) ran (in which case post-hooks run once, after the loop, not once per iteration), and recorded run usage (`CONVOY_RUN_COST`, `CONVOY_RUN_ADVISOR_COST`, `CONVOY_RUN_TOKENS_*`, and `CONVOY_RUN_DURATION_MS`; formats and presence rules are in [Goal mode](#goal-mode)) — pre-hooks never receive usage variables. Success post-hooks run **after** automatic compaction and receive its outcome as `CONVOY_FINALIZATION_STATE` (plus `CONVOY_FINALIZATION_SHA`, `CONVOY_FINALIZATION_SUBJECT`, and `CONVOY_FINALIZATION_REASON` when set), so a publish hook can gate on a completed compaction. A failing hook fails the run unless `continueOnError: true` is set. Each hook is also a row in the dashboard pipeline — pre-hooks ahead of the steps and post-hooks after the `Compact run` row — with live running/✓/✗/skipped status, and the tail of its output lands in that row's `logs` tab; the rows and their captured output are recorded in the run metadata, so re-opened runs show them too.

## Global configuration

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-09-11
83 changes: 83 additions & 0 deletions openspec/changes/archive/2026-09-11-hook-run-usage/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
## Context

`runHookCommand` (`src/hooks.ts`) builds the hook environment from a `RunHookContext` the runner assembles at two post-hook call sites: the success path after the summary is written and the failure path in the runner's catch block. The context carries `status`, an optional `score` and an optional `goal` outcome; each optional field becomes a conditional spread into `env`, so a hook sees a variable only when Convoy actually knows the value.

The numbers a hook would want already exist in memory at both call sites. `openRunMetadata` returns a `RunMetadataStore` whose `data.phases` holds, per phase, the executor usage (`cost`, `tokens`, written together by `recalculate` from a `PhaseUsage` accumulator), the advisor aggregate (`advisor.cost`, from `phaseAdvisorEvent`) and `durationMs` (`endedAt - startedAt`, set by `phaseEnded` before its first `await`, so a failed phase already carries it when the failure hooks run). Hook rows are phases too: pre-hooks get a duration, never a cost. The store exposes phases only by name (`snapshot(name)`), and no code path today sums them for a whole run in memory: `convoy runs` (`totalCost` in `src/runs.ts`) and `SUMMARY.md` (`readAdvisorSplit` in `src/advisor-report.ts`) both aggregate from disk, mixing `metadata.json` with attempt logs, and the dashboard sums its own `PhaseState` list.

The vocabulary is already settled by the code: *usage* is cost plus tokens (`ProgressUsage`, `PhaseUsage`, `phaseStepUsage`, `phaseUsageTotal`); the executor's usage and the advisor's are recorded apart and only `convoy runs` adds them up.

## Goals / Non-Goals

**Goals**

- Post-hooks — success and failure — receive the run's aggregated cost, advisor cost, tokens and duration from the in-memory metadata store.
- The aggregate follows the store's facts: nothing recorded means no variable, never a fabricated zero.
- One aggregation, unit-testable without a store, reused by the store.
- The README documents every variable and its format, and shows the `gh pr comment` and budget-guard use cases.

**Non-Goals**

- Pre-hooks (nothing has been recorded when they run).
- Per-step hooks or per-phase variables (there are no per-step hooks).
- Changing how `convoy runs`, the dashboard or `SUMMARY.md` compute or display cost and duration, including the `--resume` duration caveat below.
- Any remote publication by Convoy itself; the hook owns whatever happens with the numbers.

## Decisions

### D1: `sumRunUsage` in `src/usage.ts`, a pure fold over recorded phases

```ts
export type RunUsage = {
/** Executor plus advisor cost in USD; present when at least one phase recorded either. */
cost?: number
/** Advisor cost in USD; present only when the summed advisor spend is above zero. */
advisorCost?: number
/** Summed executor tokens; present when at least one phase recorded executor usage. */
tokens?: ProgressTokens
/** Sum of the recorded phases' durations; present when at least one phase recorded one. */
durationMs?: number
}

export function sumRunUsage(phases: Iterable<RunUsagePhase>): RunUsage | undefined
```

`RunUsagePhase` is a structural type — `{ cost?: number; tokens?: ProgressTokens; durationMs?: number; advisor?: { cost: number } }` — so `usage.ts` keeps depending only on `progress.ts` and does not import `PhaseMetadata` (which would close a `usage → metadata → usage` cycle). The fold reuses `emptyTokens`, `addTokens` and `safeCost`. `tokens` is set when any phase has a numeric `cost` (the same marker `totalCost` uses, and the invariant `recalculate` guarantees that tokens travel with it); `cost` when any phase has a numeric executor or advisor cost, so advisor-only spend still yields a total, as in `convoy runs`; `advisorCost` when the advisor sum is above zero; `durationMs` when any phase has one. The function returns `undefined` when none of the groups has data, so callers can spread it like `goal`.

Alternative considered: a store method `phases()` and the fold at the call site. Rejected: it exposes the store's internal record shape for a single consumer.

### D2: `RunMetadataStore.runUsage(): RunUsage | undefined`

The store delegates to `sumRunUsage(Object.values(data.phases))`. No I/O, no cache: `data` is the live record, and the call happens once per post-hook stage. Its name matches the glossary term (run usage) rather than an implementation-flavoured `usageTotals`.

Alternative considered: re-reading `metadata.json` (or attempt logs, as `readAdvisorSplit` does) from `hooks.ts`. Rejected: it adds I/O and a second source of truth inside the process that already holds the data, and the attempt logs would double count phases the metadata has totals for.

### D3: `RunHookContext.usage?: RunUsage` and the environment variables

`hooks.ts` adds the optional field and a `usageEnv(usage)` helper that returns the variables, spread into `env` after `CONVOY_GOAL_*`:

| Variable | Present when | Format |
|---|---|---|
| `CONVOY_RUN_COST` | `usage.cost !== undefined` | `toFixed(4)` |
| `CONVOY_RUN_TOKENS_INPUT/OUTPUT/REASONING/CACHE_READ/CACHE_WRITE/TOTAL` | `usage.tokens !== undefined` | integer |
| `CONVOY_RUN_ADVISOR_COST` | `usage.advisorCost !== undefined` | `toFixed(4)` |
| `CONVOY_RUN_DURATION_MS` | `usage.durationMs !== undefined` | integer (`Math.round`) |

Four fractional digits match the advisor split in `SUMMARY.md`; the dashboard's two are a display concession, and a hook that aggregates runs would lose the cheap ones. No `CONVOY_RUN_EXECUTOR_COST`: two numbers are enough and the third is a subtraction. Formatting lives in `hooks.ts` so `test/hooks.test.ts` pins the textual contract the README documents.

### D4: Both post-hook call sites pass the aggregate

The success and the failure call sites in `src/runner.ts` add `...(usage ? { usage } : {})` with `usage = metadata?.runUsage()`. The failure path uses optional access because the run may fail before `openRunMetadata` returned. The pre-hook call site is untouched. The post-hook row itself is `running` at that moment and carries no duration, so it does not count itself.

### D5: README

The hooks paragraph lists the new variables after `CONVOY_GOAL_*`, states the formats and that the duration is a sum of phase durations, and the goal-mode example gains a sibling: a post-hook that comments cost and duration on the PR with `gh pr comment`, and a one-line `when: always` budget guard using `awk` (a shell `[ -gt ]` cannot compare a decimal), with the note that a failing post-hook fails the run.

## Risks / Trade-offs

- A phase interrupted and resumed keeps its original `startedAt` (`phaseStarted` uses `??=`), so its `durationMs` — and therefore `CONVOY_RUN_DURATION_MS` — includes the time the run sat between the two processes. This is pre-existing `durationMs` behavior shared with `convoy runs` and the dashboard; changing it belongs to a separate change.
- A Claude Code phase reports cost and tokens like an OpenCode one (`src/claude-code.ts`), so the cost group does not go missing on mixed pipelines; a phase whose session never reported usage simply contributes nothing.
- Advisor consultations billed at zero (subscription models) leave `CONVOY_RUN_ADVISOR_COST` unset by design: the variable reports spend, not activity.

## Migration

None. Additive environment variables; no CLI, config, protocol or persisted-state change.
29 changes: 29 additions & 0 deletions openspec/changes/archive/2026-09-11-hook-run-usage/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
## Why

Post-hooks receive `CONVOY_RUN_STATUS`, `CONVOY_RUN_SCORE` and the `CONVOY_GOAL_*` outcome (`src/hooks.ts`, `runHookCommand`), but nothing about what the run spent or how long it worked. The runner invokes them while it still holds the run's metadata store in memory, and that store already carries every phase's executor cost, tokens, advisor cost and duration (`PhaseMetadata` in `src/metadata.ts`). A hook that wants to comment "this run cost $3.20 in 14 min" on the pull request, post the figure to a webhook, or fail a CI job above a budget has to re-open `$CONVOY_RUN_DIR/metadata.json` and reimplement the aggregation `convoy runs` performs — and every such hook does it slightly differently.

## What Changes

- Add a run-level usage aggregate to the metadata store: total cost (executor plus advisor), advisor cost, tokens and run duration, summed over every recorded phase in memory, with the same "absent when nothing was recorded" rule `CONVOY_RUN_SCORE` follows.
- Pass that aggregate to post-hooks — on success and on failure — as `CONVOY_RUN_COST`, `CONVOY_RUN_ADVISOR_COST`, `CONVOY_RUN_TOKENS_{INPUT,OUTPUT,REASONING,CACHE_READ,CACHE_WRITE,TOTAL}` and `CONVOY_RUN_DURATION_MS`, with a fixed, documented format.
- Document the variables in the README hooks paragraph with a post-hook that comments cost and duration on the PR and a one-line budget guard.
- Pre-hooks, the dashboard, `convoy runs`, `SUMMARY.md` and the metadata file format are untouched; Convoy still never publishes anything itself — the hook decides what to do with the numbers.

## Capabilities

### New Capabilities

- `hook-run-usage`: post-hooks receive the run's aggregated usage — cost, advisor cost, tokens and duration — from the run's own metadata, so a project can report or gate on spend without parsing run files.

### Modified Capabilities

<!-- None: pre-hooks and every existing hook variable keep their behavior. -->

## Impact

- `src/usage.ts` (run-level sum over recorded phases).
- `src/metadata.ts` (`RunMetadataStore.runUsage()`).
- `src/hooks.ts` (`RunHookContext.usage` and the new environment variables).
- `src/runner.ts` (both post-hook call sites pass the aggregate).
- `README.md` (hooks paragraph and example).
- No CLI surface, harness protocol, control protocol, config schema or persisted-state change.
Loading