fix: render platforms without paired baselines - #9
Conversation
setup-benchmark-go-action
Action bundle sizes
Parser and validation operations
Compared only with the latest matching platform in the main series. |
There was a problem hiding this comment.
Review summary
This change cleanly relaxes the same-runner baseline rule from a global "baseline required for every result artifact" to a per-platform, all-or-nothing rule: a platform may omit its baseline entirely, but presence must be consistent across all shards of the same platform. The implementation is solid:
src/artifact.jsuses aplatformHasBaselinemap with a symmetric, order-independent consistency check (previous === hasBaseline), and the specific per-platform assert message matches the updated test.- Any baseline that is present is still fully validated (schema,
shardId,platform, and sample pairing), so the relaxation drops only a coverage/completeness guarantee, not an integrity one. baselines.length === 0 ? [] : mergeShards(baselines)correctly guardsmergeShardsagainst an empty set.- Untrusted fields rendered in the report note (
baseline.source.sha,baseline.source.url) remain validated (40-hex SHA, HTTPS-only URL wrapped in<...>autolink) — no injection surface introduced. - Test coverage is good: rejects inconsistent shards within a platform, allows a whole platform without a baseline, and verifies the
newmarking plus the note text.
One documentation follow-up below. No blocking issues.
Findings
[P2] Stale README: baseline behavior no longer matches the per-platform logic
README.md (lines 424–430) still documents the old all-or-nothing model:
When every platform artifact includes
baseline-benchmark-file, each metric is compared with that paired baseline measured by the same runner job... Otherwise, each metric is compared with the newest matching platform inmain.
After this PR that wording is inaccurate. With mixed platforms (some with a same-runner baseline, some without), the paired platforms render vs base while the unpaired platforms are marked new (metricComparison returns an undefined baseline) — they do not fall back to the main series. The comparison decision is now per platform, not a single global switch. Consider updating this paragraph to describe the per-platform behavior and note that platforms lacking a paired baseline are marked new, matching the new report note in src/report.js.
A pull request may introduce a platform that its base cannot build yet. In that case, requiring every result artifact to contain a baseline either hides the new platform or makes publication fail.
This change keeps same-runner pairing strict within each platform while allowing a complete platform to have no baseline. Reports retain paired deltas for platforms with baselines and render metrics from newly supported platforms as
new, with an explicit note.Validation:
npm run check(39 tests)npm run buildnew