Add RSpec test reporter to CI workflows

[tophcodes]

Adds RSpec test result reporting to both parallel and non-parallel test workflows using dorny/test-reporter. After all spec jobs complete, the results are published as a GitHub Check Run called RSpec Tests on the PR or commit — failures are listed individually with expandable error output. No more scrolling through logs.

The reporter uses RSpec's built-in JSON formatter (rspec-json), so services do not need the rspec_junit_formatter gem.

For the non-parallel workflow, a --format json --out line in .rspec is enough — the file is written once and picked up by the publish step at the end of the job.

For the parallel workflow, the formatter must be registered in spec_helper.rb rather than .rspec, because parallel_tests' rake task invocation does not pass .rspec formatter config through to workers. spec_helper.rb is explicitly required by each worker subprocess. TEST_ENV_NUMBER (already used for SimpleCov) gives each worker a unique output filename so they don't overwrite each other.

Each specs matrix job uploads its JSON files as an artifact named rspec-json-N. The finalize job downloads all artifacts into subdirectories and publishes the consolidated report. finalize runs with if: always() so the report appears even when tests fail. The report steps use fail-on-error: false and continue-on-error: true so a missing JSON file does not break the build.

dorny/test-reporter also supports jest-junit and mocha-json, so the same approach can be applied to frontend services using Jest or Mocha — no additional tooling needed.

Note

Services need to opt in. For single-worker services: add --format json --out tmp/rspec-results.json to .rspec. For parallel services: register the formatter in spec_helper.rb using TEST_ENV_NUMBER. Without any output file the upload and report steps silently no-op.

Warning

Todo — Verify this works correctly for repos that do not use RSpec at all. A real CI run against a non-RSpec service (e.g. a pure frontend) is needed to confirm nothing silently breaks.

Example PRs with enabled test reporter

• hausgold/payment-api#216 — test-nonparallel
• hausgold/maklerportal-api#1636 — test-parallel

[Jack12816]

First off: I like the idea and the result. Just some thoughts:

No more scrolling through logs.

That can be an anoying thing, but it also shows deprecations and an eagle eyed
dev will be informed afterwards. This makes the not-convenient way worth the
hassle, at least for me.

using dorny/test-reporter

I'm not a fan of external GHA workflows and alike. This stuff is run within the
same execution context where all our secrets and code are accessible. Maybe we
can implement this ourself with a hausgold/potpourri helper and reuse the
coverage logic for parallel handling (coverage part, coverage merge).
Generating markdown content from a JSON file shouldn't be this hard, I guess.

Example rspec json (from your Payment API pr):
rspec-results.json

Integration:

--format json --out tmp/rspec-results.json

vs

RSpec.configuration.add_formatter(
  RSpec::Core::Formatters::JsonFormatter,
  "tmp/rspec-results#{ENV['TEST_ENV_NUMBER']}.json"
)

Could be just added to painless/rspec
(or a painless/rspec/support/rspec_formatter.rb file), this way its distributed
automatically and enabled by default. And on a shared place for future changes.

This would save us from various config changes on all applications, and this
should be done for simplecov, too in the future.

Job naming:

RSpec Tests does not work well for frontends, and the root namespace is not ideal. Renaming to Test / Summary would be nice.

[Jack12816]

See: https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28#create-a-check-run