Skip to content

Failure report shows only the last 1,024 bytes, which never contains the failing test #530

Description

@khaliqgant

Problem

When a deterministic step fails, the runner reports:

Stdout (last 1,024 bytes):

For any test runner that streams results and prints a summary last (node
--test TAP, cargo, vitest, pytest), the last 1,024 bytes are always the
tail of the run
— the totals and the final few passing cases. The failing
case is by construction earlier in the stream, so it is exactly the part that
gets cut.

Concrete case

A failing node --test step in a relay PR reported:

ok 13 - pty-exit: child process exit is detected
ok 14 - pty-exit: long-running process stays alive until released
1..14
# tests 14
# pass 13
# fail 1

Every line visible in the failure report is ok. The one not ok was in
tests 1–12 and never appeared. The report proves a failure happened and
withholds which one — the single fact the reader needs.

Recovering it meant compiling the suite and re-running all 14 tests locally
(~2 min, and it did not reproduce), because the journal SQLite is not
uploaded as a CI artifact.

Why the existing escape hatches don't cover it

  • flows replay <run> --at <step> needs --data-dir, which points inside
    .workflow-artifacts/ on the runner. Not uploaded, so it is gone once the
    job ends.
  • Re-running with more verbosity re-runs the whole matrix.

Proposals (any one would resolve this)

  1. Prefer the head+tail over the tail. Show first 512 + last 512 bytes
    with a … N bytes elided … marker. Cheap, no config, and catches runners
    that print failures inline.
  2. Make the byte budget configurableoutputTailBytes on the step, or
    RELAYFLOWS_OUTPUT_TAIL_BYTES. 1,024 is very small for a test step.
  3. Grep-forward on failure. Let a step declare
    failureHighlight: ["^not ok", "^FAILED", "^\\s+✖"]; on non-zero exit the
    runner also prints matching lines from anywhere in the captured stream.
    This is the most targeted fix — it surfaces the failing case regardless of
    where it sits in the stream.

(3) plus (1) would mean a red step names what failed without a local repro.

Impact

Any 80-to-100 campaign that runs a real test suite in a deterministic step.
The gate correctly goes red and then cannot say why, which pushes diagnosis
back to a local re-run — the exact loop the gate exists to avoid.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    garden-readyScoped and ready for an agent to pick up

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions