retries_exhausted reports only the last attempt's failure and hides the first

docs/CLOUD.md

@@ -240,6 +240,17 @@ frame this vocabulary has no opinion about is reported as one line naming its
 type and size, and a line that is not JSON is printed as written, so `--raw`
 is never the only way to find out that something ran.
 
+A retried step that failed differently each time shows every attempt's error
+here, because the runner log is the `flows` process's own captured output and
+the step-failure diagnostic it writes lists every failed attempt (see
+[Reading a failed step](SURFACE.md#reading-a-failed-step)). Nothing is elided
+on the way out: the runner log is printed line for line. The limit is that the
+log is a *recording* — there is no journal-export endpoint, so `flows logs`
+can only show what the `flows` binary that ran the workflow printed at the
+time. A run executed by a build predating this diagnostic carries only the
+terminal attempt in its log, and no later reader can recover the earlier ones
+from Cloud.
+
 `flows status --cloud <run-id>` is the local `flows status` view, sourced from
 the run record and the step list instead of a journal: the `RUN` header with
 status, completion reason and summed spend, a `steps N` count, and one

docs/SURFACE.md

@@ -1016,7 +1016,10 @@ uses the opening shown here; an authored child failure opens with
 ```text
 FAILED [step_failed] Run "<run-id>" failed with completionReason: step_failed.
  Step "<step-id>" (<type>) completionReason: <reason> attempt=<n>/<budget> exit=<code>.
-Detail: <the worker's own account, when it left one>
+Attempts: <count> failed; <whether their recorded evidence agrees>
+  attempt 1: <reason> exit=<code> — stderr: <excerpt>
+  attempt 2: <reason> exit=<code> — stderr: <excerpt>
+Detail: <the gate's verdict, or the worker's own account>
 Stdout (captured excerpt):
 <excerpt>
 Stderr (captured excerpt):
@@ -1058,9 +1061,60 @@ journal to be excerpted.
 Each clause is present only when the journal holds the fact behind it; nothing
 is defaulted. The same fields appear as named keys on the `--json` diagnostic
 (`stepId`, `stepType`, `completionReason`, `attempt`, `maxIterations`,
-`exitCode`, `stdoutTail`, `stderrTail`, `detail`, `transcriptPath`, `hint`,
-`journalPath`), so the rendered line and the machine-readable record carry the
-same facts rather than the message being the only copy.
+`exitCode`, `stdoutTail`, `stderrTail`, `detail`, `transcriptPath`, `attempts`,
+`attemptEvidence`, `hint`, `journalPath`), so the rendered line and the
+machine-readable record carry the same facts rather than the message being the
+only copy.
+
+The scalar clauses always describe the **terminal** attempt. `Attempts:` is
+additive and appears only when the journal held more than one failed attempt of
+that step, so a step that failed once renders exactly as it did before. Every
+failed attempt is listed, oldest first, from its own `step.completed` entry —
+the kernel appends one per attempt, so a retry that failed differently from the
+first attempt is a journaled fact rather than something the last attempt's
+output has to be read for. Each entry carries the attempt's reason, its exit
+code when it had one, and up to 256 bytes of whichever of `detail`, `stderr`
+and `stdout` that attempt actually recorded; `(excerpt truncated)` marks an
+account that was cut, and an attempt that recorded nothing says so rather than
+printing empty fields. Attempt excerpts are redacted before they are bounded.
+The corresponding `attempts` entries in `--json` use the keys `attempt`,
+`completionReason`, `disposition`, `exitCode`, `stdoutTail`, `stderrTail`,
+`detail` and `truncated`.
+
+`detail` — on the terminal clause and on each attempt — is the verification
+record's account: the gate's verdict for a deterministic step (`output did not
+contain "READY"`), or what the worker reported for an agent or llm step. A gate
+can refuse output that a command produced happily, so an attempt can report
+exit 0, empty tails and a verdict that is the entire reason it failed. It is
+withheld only when it would repeat bytes already printed beside it: the
+daemon's `{exit_code, stdout_tail, stderr_tail}` render of a worker failure,
+and the `exit_code` gate's bare `exit code was <code>` next to the exit code
+it restates.
+
+`attemptEvidence` says whether the attempts failed for the same reason, and is
+one of:
+
+- `differs` — two attempts that each recorded evidence recorded different
+  evidence. A retry that fails differently usually means an earlier attempt
+  already had a side effect, so the message adds *An earlier attempt may have
+  had side effects.*
+- `unchanged` — every attempt recorded evidence and all of it agrees.
+- `unknown` — at least one attempt journaled nothing about why it failed, or
+  its evidence reached the journal already truncated by the daemon, so whether
+  the causes differ cannot be decided.
+
+Only attempts that recorded something are compared: an attempt that journaled
+no account of itself is unequal to every other one on paper while establishing
+nothing, and would otherwise put the side-effect warning on a crash that is
+evidence of nothing. A render the daemon truncated is compared, because
+truncation can only make two accounts look more alike than they were.
+
+The comparison runs on the journal record before any display bound, and reads
+the verification gate, verdict and detail as well as the exit codes and output
+tails, so two attempts whose visible excerpts are identical are still reported
+as differing when the records behind them are not. `verification_failed` on a
+retry and `retries_exhausted` at the budget limit are the kernel's label for
+the same fallback, so that pair alone is not counted as a changed cause.
 
 `attempt=<n>/<budget>` is read from the journal, not from the spec: `n` is the
 `step.attempt.started` envelope's attempt number and `budget` is the

kernel/relayflowd-core/src/machine/tests.rs

@@ -691,3 +691,180 @@ fn deterministic_lease_rejects_invalid_and_foreign_fields() {
         );
     }
 }
+
+/// Pin the journal contract the `retries_exhausted` diagnostic reads.
+///
+/// A retried step appends one `step.completed` PER ATTEMPT, each carrying its
+/// own attempt envelope and — for a deterministic step — its own preserved
+/// `{exit_code, stdout_tail, stderr_tail}`. Without that, the first attempt's
+/// error would not exist to report and the reader could only ever show the
+/// last one. Nothing about retry policy is asserted or changed here.
+///
+/// The two attempts below fail with the SAME exit code and DIFFERENT stderr,
+/// which is the shape that matters: the kernel's own verification record is
+/// byte-identical across them ("exit code was 1; output did not contain
+/// \"hello\""), so a reader comparing only that record would call two
+/// different failures the same failure. The distinguishing evidence lives in
+/// the preserved output.
+#[test]
+fn each_failed_attempt_journals_its_own_output_beside_an_identical_verdict() {
+    let spec = retrying_spec();
+    let mut step = spec.steps[0].clone();
+    step.max_iterations = 2;
+
+    let first_output = json!({
+        "exit_code": 1,
+        "stdout_tail": "",
+        "stderr_tail": "remote: GitLab: You cannot push commits for 'factory@example.com'",
+    });
+    let first = completion_actions(
+        "run",
+        &step,
+        1,
+        0,
+        AttemptResult::successful(first_output.clone(), "kernel"),
+        1_000,
+    );
+    let Action::Append(first_entry) = &first[0] else {
+        panic!("a failed attempt must append its own completion");
+    };
+    let first_payload: StepCompletedPayload =
+        serde_json::from_value(first_entry.payload.clone()).unwrap();
+    assert_eq!(first_entry.attempt, Some(1));
+    assert_eq!(first_payload.disposition, Disposition::Retry);
+    assert_eq!(
+        first_payload.completion_reason,
+        CompletionReason::VerificationFailed
+    );
+    assert_eq!(first_payload.output, first_output);
+
+    let second_output = json!({
+        "exit_code": 1,
+        "stdout_tail": "",
+        "stderr_tail": "nothing staged inside the declared scope",
+    });
+    let second = completion_actions(
+        "run",
+        &step,
+        2,
+        1,
+        AttemptResult::successful(second_output.clone(), "kernel"),
+        2_000,
+    );
+    let Action::Append(second_entry) = &second[0] else {
+        panic!("the terminal attempt must append its own completion");
+    };
+    let second_payload: StepCompletedPayload =
+        serde_json::from_value(second_entry.payload.clone()).unwrap();
+    assert_eq!(second_entry.attempt, Some(2));
+    assert_eq!(second_payload.disposition, Disposition::StepDone);
+    assert_eq!(
+        second_payload.completion_reason,
+        CompletionReason::RetriesExhausted
+    );
+    assert_eq!(second_payload.output, second_output);
+
+    // Both attempts survive, and only the output tells them apart.
+    assert_ne!(first_payload.output, second_payload.output);
+    assert_eq!(first_payload.verification, second_payload.verification);
+    let verification = first_payload.verification.expect("a failure names itself");
+    assert_eq!(verification.gate, "exit_code+output_contains");
+    assert_eq!(verification.verdict, crate::entry::VerificationVerdict::Fail);
+}
+
+/// The label change at the budget limit is POLICY, not a different cause.
+///
+/// Identical deterministic failures are labelled `verification_failed` while
+/// an iteration remains and `retries_exhausted` once it does not, because
+/// `completion_actions` picks the fallback reason from the branch it took. A
+/// reader that compared the raw labels would report "the retry failed for a
+/// different reason" on every ordinary repeated failure, so this pins that
+/// the two labels can sit over byte-identical evidence.
+#[test]
+fn the_exhaustion_label_replaces_verification_failed_over_identical_evidence() {
+    let spec = retrying_spec();
+    let mut step = spec.steps[0].clone();
+    step.max_iterations = 2;
+    let output = json!({"exit_code": 1, "stdout_tail": "", "stderr_tail": "same failure"});
+
+    let mut payloads = Vec::new();
+    for (attempt, semantic_executions, now_ms) in [(1, 0, 1_000), (2, 1, 2_000)] {
+        let actions = completion_actions(
+            "run",
+            &step,
+            attempt,
+            semantic_executions,
+            AttemptResult::successful(output.clone(), "kernel"),
+            now_ms,
+        );
+        let Action::Append(entry) = &actions[0] else {
+            panic!("every attempt appends a completion");
+        };
+        assert_eq!(entry.attempt, Some(attempt));
+        payloads.push(
+            serde_json::from_value::<StepCompletedPayload>(entry.payload.clone()).unwrap(),
+        );
+    }
+
+    assert_eq!(
+        payloads[0].completion_reason,
+        CompletionReason::VerificationFailed
+    );
+    assert_eq!(
+        payloads[1].completion_reason,
+        CompletionReason::RetriesExhausted
+    );
+    assert_eq!(payloads[0].output, payloads[1].output);
+    assert_eq!(payloads[0].verification, payloads[1].verification);
+}
+
+/// A worker-reported failure keeps its OWN label on both attempts.
+///
+/// `failure_reason` is supplied, so neither fallback applies: the retry and
+/// the terminal completion both read `worker_error`, and each carries the
+/// worker's own detail. Only the `verification_failed` → `retries_exhausted`
+/// pair is a policy-only transition; every other label difference across
+/// attempts is a real difference in what the worker reported.
+#[test]
+fn a_worker_reported_reason_is_not_rewritten_by_the_retry_branch() {
+    let spec = retrying_spec();
+    let mut step = spec.steps[0].clone();
+    step.max_iterations = 2;
+    let result = |detail: &str| AttemptResult {
+        failure_reason: Some(CompletionReason::WorkerError),
+        failure_detail: Some(detail.to_owned()),
+        ..AttemptResult::successful(Value::Null, "kernel")
+    };
+
+    let retry = completion_actions("run", &step, 1, 0, result("push rejected"), 1_000);
+    let Action::Append(retry_entry) = &retry[0] else {
+        panic!("a worker failure appends a completion");
+    };
+    let retry_payload: StepCompletedPayload =
+        serde_json::from_value(retry_entry.payload.clone()).unwrap();
+    assert_eq!(retry_payload.completion_reason, CompletionReason::WorkerError);
+    assert_eq!(retry_payload.disposition, Disposition::Retry);
+    assert_eq!(
+        retry_payload.verification.as_ref().map(|v| v.detail.as_str()),
+        Some("push rejected")
+    );
+
+    let terminal = completion_actions("run", &step, 2, 1, result("nothing staged"), 2_000);
+    let Action::Append(terminal_entry) = &terminal[0] else {
+        panic!("the terminal worker failure appends a completion");
+    };
+    let terminal_payload: StepCompletedPayload =
+        serde_json::from_value(terminal_entry.payload.clone()).unwrap();
+    assert_eq!(
+        terminal_payload.completion_reason,
+        CompletionReason::WorkerError
+    );
+    assert_eq!(terminal_payload.disposition, Disposition::StepDone);
+    assert_eq!(
+        terminal_payload
+            .verification
+            .as_ref()
+            .map(|v| v.detail.as_str()),
+        Some("nothing staged")
+    );
+}

packages/sdk/src/authored-node-runner.ts

@@ -13,7 +13,9 @@ import {
   AuthoredFlowExecutionError, AuthoredHumanParked,
   type AuthoredFlowExecutionErrorCode, type AuthoredHumanWait,
 } from './authored-flow-error.js';
-import type { ParkCause, StepFailedDetails } from './failure-kinds.js';
+import type {
+  AttemptEvidenceComparison, ParkCause, StepAttemptFailure, StepFailedDetails,
+} from './failure-kinds.js';
 import { HUMAN_WAIT_ID } from './authored-human.js';
 import { assertAuthoredPromiseHooks } from './authored-runtime-capability.js';
 
@@ -200,26 +202,76 @@ export function parkCauseFrame(value: unknown): ParkCause | undefined {
  * evidence frame would replace the answer with a worse one.
  */
 export function stepFailedFrame(value: unknown): StepFailedDetails | undefined {
-  if (typeof value !== 'object' || value === null || Array.isArray(value)) return undefined;
-  const frame = value as Record<string, unknown>;
-  const text = (key: string): string | undefined =>
-    typeof frame[key] === 'string' ? (frame[key] as string).slice(0, 8192) : undefined;
-  const count = (key: string): number | undefined =>
-    typeof frame[key] === 'number' && Number.isSafeInteger(frame[key]) ? frame[key] as number : undefined;
+  const frame = frameRecord(value);
+  if (frame === undefined) return undefined;
   const details: StepFailedDetails = {};
   for (const [key, parsed] of [
-    ['stepId', text('stepId')], ['stepType', text('stepType')],
-    ['completionReason', text('completionReason')], ['attempt', count('attempt')],
-    ['maxIterations', count('maxIterations')], ['exitCode', count('exitCode')],
-    ['stdoutTail', text('stdoutTail')], ['stderrTail', text('stderrTail')],
-    ['detail', text('detail')], ['transcriptPath', text('transcriptPath')],
-    ['hint', text('hint')], ['journalPath', text('journalPath')],
+    ['stepId', frameText(frame, 'stepId')], ['stepType', frameText(frame, 'stepType')],
+    ['completionReason', frameText(frame, 'completionReason')], ['attempt', frameCount(frame, 'attempt')],
+    ['maxIterations', frameCount(frame, 'maxIterations')], ['exitCode', frameCount(frame, 'exitCode')],
+    ['stdoutTail', frameText(frame, 'stdoutTail')], ['stderrTail', frameText(frame, 'stderrTail')],
+    ['detail', frameText(frame, 'detail')], ['transcriptPath', frameText(frame, 'transcriptPath')],
+    ['attempts', attemptFrames(frame['attempts'])],
+    ['attemptEvidence', comparisonFrame(frame['attemptEvidence'])],
+    ['hint', frameText(frame, 'hint')], ['journalPath', frameText(frame, 'journalPath')],
   ] as const) {
     if (parsed !== undefined) (details as Record<string, unknown>)[key] = parsed;
   }
   return Object.keys(details).length === 0 ? undefined : details;
 }
 
+/**
+ * The attempt history, reduced element by element on the same terms as the
+ * frame that carries it. An element that is not an object contributes nothing
+ * rather than voiding the whole history: a report naming three of four
+ * attempts is still the first attempt's error, which is the fact the terminal
+ * scalars cannot supply.
+ */
+function attemptFrames(value: unknown): StepAttemptFailure[] | undefined {
+  if (!Array.isArray(value)) return undefined;
+  const attempts: StepAttemptFailure[] = [];
+  for (const element of value) {
+    const source = frameRecord(element);
+    if (source === undefined) continue;
+    const attempt: StepAttemptFailure = {};
+    for (const [key, parsed] of [
+      ['attempt', frameCount(source, 'attempt')],
+      ['completionReason', frameText(source, 'completionReason')],
+      ['disposition', frameText(source, 'disposition')],
+      ['exitCode', frameCount(source, 'exitCode')],
+      ['stdoutTail', frameText(source, 'stdoutTail')],
+      ['stderrTail', frameText(source, 'stderrTail')],
+      ['detail', frameText(source, 'detail')],
+      // Only a literal `true` claims truncation; anything else leaves the
+      // excerpt unlabelled rather than labelling a complete one as cut.
+      ['truncated', source['truncated'] === true ? true : undefined],
+    ] as const) {
+      if (parsed !== undefined) (attempt as Record<string, unknown>)[key] = parsed;
+    }
+    if (Object.keys(attempt).length > 0) attempts.push(attempt);
+  }
+  return attempts.length === 0 ? undefined : attempts;
+}
+
+/** An unrecognised verdict is dropped, and rendering then says `unknown`. */
+function comparisonFrame(value: unknown): AttemptEvidenceComparison | undefined {
+  return value === 'differs' || value === 'unchanged' || value === 'unknown' ? value : undefined;
+}
+
+function frameRecord(value: unknown): Record<string, unknown> | undefined {
+  return typeof value === 'object' && value !== null && !Array.isArray(value)
+    ? value as Record<string, unknown> : undefined;
+}
+
+function frameText(frame: Record<string, unknown>, key: string): string | undefined {
+  return typeof frame[key] === 'string' ? (frame[key] as string).slice(0, 8192) : undefined;
+}
+
+function frameCount(frame: Record<string, unknown>, key: string): number | undefined {
+  return typeof frame[key] === 'number' && Number.isSafeInteger(frame[key])
+    ? frame[key] as number : undefined;
+}
+
 /** The IPC frame is a claim, not a durable terminal fact or a sandbox boundary. */
 export async function verifyAuthoredNodeResult(
   result: AuthoredFlowExecutionResult, metadata: AuthoredRootMetadata,

packages/sdk/src/authored-worker-step.ts

@@ -330,6 +330,8 @@ function stepDetails(
     ...(found.stderrTail === undefined ? {} : { stderrTail: found.stderrTail }),
     ...(found.detail === undefined ? {} : { detail: found.detail }),
     ...(found.transcriptPath === undefined ? {} : { transcriptPath: found.transcriptPath }),
+    ...(found.attempts === undefined ? {} : { attempts: found.attempts }),
+    ...(found.attemptEvidence === undefined ? {} : { attemptEvidence: found.attemptEvidence }),
     ...(found.hint === undefined ? {} : { hint: found.hint }),
     ...(found.journalPath === undefined ? {} : { journalPath: found.journalPath }),
   };