Skip to content

Benchmark publishing: validated results PRs for docs and website #162

Description

@SarthakWade

Problem

Benchmark results currently feed repository documentation, but a future external benchmark lab needs a controlled publication path. Directly mutating production docs from benchmark runs would allow noisy, incomplete, or compromised results to become public claims.

Proposed workflow

A manual, scheduled, or release-triggered lab run produces a versioned result bundle with provenance. After validation, automation opens a reviewable PR in this repository updating the canonical result artifact. Existing website provenance checks consume that artifact.

Do not deploy directly from the benchmark repository and do not create a second website deployment workflow.

Acceptance criteria

  • Result schema, tool versions, environment, run ID, commit IDs, sample counts, and timestamps are validated.
  • Publication rejects incomplete suites, schema drift, missing raw evidence, implausible values, and unsigned or unauthorized sources.
  • Generated prose clearly states methodology and limitations.
  • Repeated runs do not create duplicate PRs.
  • Failed or statistically unstable runs do not replace the last accepted baseline.
  • CI verifies docs and website output before merge.
  • Rollback is documented and requires no benchmark rerun.

Blocked by #161.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:qa-evidenceQA evidence, benchmarks, E2E suitesarea:webapps/web marketing and docs sitepriority:mediumScheduled, not blockingstatus:blockedWaiting on another item or an owner decisiontype:ciBuild, CI, automation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions