Skip to content

feat: compare results with an earlier moment, and self-contained HTML reports - #152

Merged
berkantepli merged 5 commits into
mainfrom
feat/compare-and-html-report
Oct 14, 2026
Merged

berkantepli merged 5 commits into
mainfrom
feat/compare-and-html-report

Conversation

@berkantepli

Copy link
Copy Markdown
Owner

Summary

  • What changed: POST /api/v1/runs/compare groups each check as newly failing, fixed, still failing (noting a changed reason), slower (50% and 50 ms, as on Trends), or only on one side.
    • Coverage & Risk: Compare with… (before the latest run, 24 hours ago, 7 days ago); rows open their run. Bulk runs get a batchId so a whole bulk run counts as one run.
    • Run details: Compare with the previous run of the same endpoint or flow.
    • CLI: api-qa run --baseline last.json prints and records the changes; --fail-on new fails only on checks that passed in the baseline.
  • HTML reports: POST /api/v1/reports/html renders one self-contained file (inline styles, no scripts or external resources, strict CSP meta, every value escaped). Evidence for failures and errors only unless asked.
    • Run details: Download HTML report (with the comparison to the previous run). Coverage & Risk: Export HTML (with the open comparison). CLI: --html report.html (header values masked).
  • README, docs/GUIDE.md, docs/API.md, and CHANGELOG (Unreleased) updated. Additive only: two new routes, new CLI options, an optional comparison key in the --json report.

Test plan

  • pytest (333 passed): comparison rules, HTML escaping/CSP/evidence, routes, CLI --baseline/--fail-on new/--html
  • npm test (246 passed), npm run build
  • npm run test:e2e (28 passed), including a demo-API test that creates a real new failure, compares, downloads both reports, and opens a run from a row
  • Screenshots of the panel and the downloaded report in light and dark

🤖 Generated with Claude Code

berkantepli and others added 5 commits October 14, 2026 16:28
…L report

- POST /api/v1/runs/compare groups each check as newly failing, fixed, still failing,
  slower (50% and 50 ms, as on Trends), or only in one side.
- POST /api/v1/reports/html renders a run or coverage report: one file, styles inline,
  no scripts, every value escaped, evidence for failures only unless asked.
- api-qa run: --html FILE, --baseline FILE (an earlier --json report), and
  --fail-on new to fail only on checks that passed in the baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eports

- Coverage & Risk: Compare with… (before the latest run, 24 hours ago, 7 days ago) lists
  what is newly failing, fixed, still failing, or slower; each row opens its run.
- Run details: Compare with the previous run, and Download HTML report (evidence of
  passed checks on request; the comparison with the previous run is included).
- Coverage & Risk: Export HTML, with the open comparison.
- Bulk runs get a batch id, so a whole bulk run counts as the latest run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	backend/app/cli.py
…I reference

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	CHANGELOG.md
#	README.md
#	frontend/src/RunDetail.jsx
#	frontend/src/styles.css
@berkantepli
berkantepli merged commit f42195a into main Oct 14, 2026
4 checks passed
@berkantepli
berkantepli deleted the feat/compare-and-html-report branch October 14, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant