Skip to content

e2e job always reports success, so the required ci-success check passes even when e2e is broken #2358

Description

@kirilly

The Tests workflow has an e2e job that can't currently pass, but it shows green on every run, and ci-success (the check our main/dev ruleset requires) depends on it. So a broken e2e never blocks a merge, and nobody notices it's broken.

Evidence

Run https://github.com/peanutprotocol/peanut-ui/actions/runs/27425406519/job/81061724408. The "Run E2E tests" step logs TEST_HARNESS_SECRET is not set and exits 1, but the step and the job both report success.

Why it's green

  • In .github/workflows/tests.yml the step has continue-on-error: true, so the job passes no matter what happens.
  • The step never passes TEST_HARNESS_SECRET (or API_BASE_URL / UI_BASE_URL) into the env, and nothing starts the API or UI, so e2e/global-setup.ts throws at getHarnessSecret() before any test runs. It cannot pass as written.
  • e2e is listed in ci-success.needs, and the ruleset "Require ci-success on api + ui (main, dev)" makes ci-success the required check. So the required gate is satisfied by a job that is guaranteed green.
  • The report job lists e2e in needs but only parses the unit JUnit, so its "all green" summary never reflects e2e anyway.

Options

  • Wire the secret and stand up the API/UI, then remove continue-on-error so real failures block.
  • Or, until it's wired, skip the job cleanly with an if: guard and take it out of ci-success.needs (the way eslint is already handled), so we stop claiming a passing check that never actually ran.

Either way, stop counting a can't-fail job as a required check.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions