feat: production polish — package layout, CI, viewer, docs

[JacobPEvans-personal]

Summary

Take mlx-benchmarks from early-stage scaffolding to a polished,
senior-reviewer-ready project in four logical commits.

• Publisher — migrate scripts/publish_run.py into a proper
  src/mlx_benchmarks/ package (typed envelope, runtime system
  detection, converter protocol, strict jsonschema validation before
  every publish, structured JSON-lines logging). Expose as
  mlx-bench-publish. Keep the old script as a back-compat shim.
• Schema v1.1 — add $id, optional reproducibility fields
  (python_version, mlx_version, mlx_lm_version, lm_eval_version,
  kernel, seed, gen_kwargs, model_revision, quantization,
  duration_seconds), and ship canonical valid + invalid envelope
  examples. Non-breaking.
• Quality gates — ruff + mypy strict + pytest + pre-commit.
  18-test pytest suite with schema round-trip, publisher dry-run,
  runtime system smoke. Zero findings, zero warnings.
• Viewer as a first-class artifact — move app.py to space/ with
  HF Spaces front-matter, add viewer chart tests (5 cases) covering
  empty-data path, bar/trend/pivot rendering.
• CI — six new workflows (test, dry-run-publish, codeql,
  dependency-review, deploy-space, release-please) + rewrite of
  validate-schema to call committed script instead of heredoc. All
  user-controllable inputs flow through env: blocks; trusted-org
  actions pinned to major-version tags.
• Docs — full README rewrite with hero + real badges + correct
  quickstart, plus CONTRIBUTING, SECURITY, architecture, schema prose,
  CODEOWNERS, .editorconfig, bug/benchmark-request issue templates, PR
  template. Session notes moved to docs/journal/.

Out of scope (follow-ups)

• Converter coverage beyond lm-eval (vllm throughput, framework-eval)
  — architecture is in place (get_converter registry), implementation
  deferred.
• Viewer bells-and-whistles (heatmap, leaderboard, sample drill-down)
  — the current three tabs ship first; enhancements deferred.
• F8-sweep historical rescue — already published as
  data/run-rescue-*.parquet shards; verified in Phase 0.

Test plan

• .venv/bin/ruff check . — all checks passed
• .venv/bin/ruff format --check . — 25 files formatted
• .venv/bin/mypy src/mlx_benchmarks — strict, 0 errors
• .venv/bin/pytest tests space/tests — 23 pass
• .venv/bin/python scripts/validate_schema.py — schema + 3 TOMLs OK
• mlx-bench-publish tests/fixtures/lm_eval_results_sample.json --kind lm-eval --suite reasoning --dry-run — green
• Live MLX smoke — BLOCKED on wedged llama-swap backend (see below)

Live-smoke status

Pipeline fully validated via fixture round-trip (the full code path a live
benchmark would take: raw JSON → LmEvalConverter → typed envelope →
runtime system detection → schema validation → Parquet → deterministic
HF filename).

Live MLX smoke could not complete this session: llama-swap on :11434 is
wedged after the pinned default Qwen3.5-27B-4bit got SIGKILL'd. The
vllm-mlx log shows 30+ minutes of 502s on POST /v1/chat/completions.
Any lm_eval run hangs until the service is restarted
(launchctl kickstart -k gui/$(id -u)/dev.vllm-mlx.server). I chose not
to take that shared-system action without approval.

Once the backend is back, the 3-model sweep drafted in
~/.claude/plans/i-never-know-vivid-sonnet.md Phase 8 is ready to run
as-is — pointing at Qwen3.5-9B-MLX-4bit, gemma-4-e4b-it-4bit, and
DeepSeek-R1-0528-Qwen3-8B-4bit for gsm8k_cot_zeroshot --limit 3.

Migration notes

• scripts/publish_run.py keeps its old CLI shape via shim; no
  breaking change for existing runbooks.
• uv sync picks up the new required deps (jsonschema, psutil,
  huggingface-hub, pyarrow) automatically. pyproject.toml no
  longer declares [tool.uv] package = false.
• Schema additions are all optional → existing consumers unaffected.

Release-please

Merge will bump to 0.3.0 (minor; feat + breaking-package-layout is
arguably major, but the public contract holds — published envelopes
remain valid, old CLI invocation still works via shim).

[Copilot]

Pull request overview

This PR polishes mlx-benchmarks into a production-ready project by packaging the publisher as mlx_benchmarks, extending/validating the envelope schema, adding a tested Gradio viewer artifact, and introducing CI + repo hygiene docs/templates.

Changes:

• Introduces a typed mlx_benchmarks package with converter(s), system detection, schema validation, CLI entrypoint, and HF publish path.
• Extends schema.json (non-breaking optional fields) and ships canonical valid/invalid envelope examples + schema tests.
• Adds CI workflows + pre-commit/mypy/ruff/pytest gates and promotes the dataset viewer under space/ with tests and deploy tooling.

Reviewed changes

Copilot reviewed 55 out of 58 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_system.py	Adds smoke tests for runtime system detection shape/content.
tests/test_schema.py	Adds Draft-07 schema self-validation + example fixture validation tests.
tests/test_publish.py	Adds unit tests for publish helpers (slugify/path/Parquet/dry-run).
tests/test_lm_eval_converter.py	Adds end-to-end lm-eval fixture → envelope → schema validation test.
tests/test_cli.py	Adds CLI argparse/dispatch smoke tests (dry-run, tag validation, JSON errors).
tests/fixtures/lm_eval_results_sample.json	Adds canonical lm-eval raw fixture used by converter/CLI tests.
tests/conftest.py	Adds shared fixtures for lm-eval + valid/invalid envelope examples.
src/mlx_benchmarks/system.py	Implements best-effort system metadata detection (os/chip/memory/versions).
src/mlx_benchmarks/publish.py	Implements envelope→rows→Parquet and HF dataset publish with deterministic shard paths.
src/mlx_benchmarks/logging_config.py	Adds console logging configuration with optional JSON-lines formatter.
src/mlx_benchmarks/envelope.py	Adds typed envelope definitions + cached jsonschema validator and schema lookup.
src/mlx_benchmarks/converters/lm_eval.py	Adds lm-eval results.json → envelope converter.
src/mlx_benchmarks/converters/base.py	Defines converter protocol + ConverterContext dataclass.
src/mlx_benchmarks/converters/init.py	Adds converter registry + get_converter() with explicit unknown-kind error.
src/mlx_benchmarks/cli.py	Adds mlx-bench-publish CLI for conversion + validation + publish/dry-run.
src/mlx_benchmarks/init.py	Exposes public API symbols and package version.
space/tests/test_charts.py	Adds Plotly chart-builder tests and empty-data behavior tests.
space/requirements.txt	Declares Space runtime dependencies for viewer deployment.
space/app.py	Viewer polish/formatting adjustments and small refactors.
space/README.md	Adds HF Spaces front-matter + viewer usage/deploy documentation.
scripts/validate_schema.py	Adds standalone schema + TOML validation script for CI/local use.
scripts/publish_run.py	Replaces legacy script with back-compat shim to the new CLI.
scripts/deploy_space.py	Adds script to atomically sync space/ contents to HF Space.
schema.json	Adds $id, optional reproducibility fields, and embedded examples.
pyproject.toml	Converts repo into an installable package, adds scripts/extras, and configures ruff/mypy/pytest/coverage.
harness/framework-eval/eval_smolagents.py	Minor hardening: switch to Path(...).open() for fixture reads.
harness/framework-eval/eval_qwen_agent.py	Minor hardening/typing: Path(...).open(), ClassVar, kwargs naming, formatting.
harness/framework-eval/eval_openai_tool_calling.py	Minor hardening: switch to Path(...).open() for fixture reads.
harness/framework-eval/eval_google_adk.py	Minor hardening: switch to Path(...).open() for fixture reads.
harness/framework-eval/README.md	Updates status text and links to follow-up benchmark request template.
examples/envelope.valid.json	Adds canonical valid envelope example fixture.
examples/envelope.invalid.json	Adds canonical invalid envelope example fixture.
docs/schema.md	Adds prose schema walkthrough and versioning guidance.
docs/journal/2026-04-19-qwen36-benchmark-session.md	Adds archived benchmark session notes.
docs/journal/2026-04-10-bifrost-benchmark-session.md	Adds archived benchmark session notes.
docs/architecture.md	Adds architecture diagram and component/CI overview.
configs/lm-eval/qwen3-tasks/utils.py	Improves task utils exports and validation (e.g., strict zip, parity checks).
configs/LAYOUT.md	Updates current vs planned config layout and links to request template.
SECURITY.md	Adds security policy, token handling guidance, and unsafe-code warnings.
README.md	Major rewrite: badges, quickstart, layout, schema/publisher usage, viewer instructions.
CONTRIBUTING.md	Adds developer workflow, quality gates, and contribution conventions.
CODEOWNERS	Adds repo-wide code owner.
CLAUDE.md	Updates agent-facing conventions and common tasks.
.release-please-manifest.json	Adds release-please manifest with current version.
.release-please-config.json	Adds release-please configuration for Python releases/changelog sections.
.pre-commit-config.yaml	Adds pre-commit hooks for formatting/lint/type/actionlint.
.github/workflows/validate-schema.yml	Reworks schema validation workflow to call committed validator script.
.github/workflows/test.yml	Adds CI workflow for ruff/mypy/pytest across Python 3.11/3.12.
.github/workflows/release-please.yml	Adds automated release PR workflow via release-please.
.github/workflows/dry-run-publish.yml	Adds CI workflow to dry-run publisher on canonical fixture.
.github/workflows/deploy-space.yml	Adds workflow to deploy space/ to HF Space using scripts/deploy_space.py.
.github/workflows/dependency-review.yml	Adds dependency review workflow gate.
.github/workflows/codeql.yml	Adds CodeQL workflow for Python security-and-quality queries.
.github/PULL_REQUEST_TEMPLATE.md	Adds PR template with required local test checklist.
.github/ISSUE_TEMPLATE/config.yml	Adds issue template config and contact links.
.github/ISSUE_TEMPLATE/bug.yml	Adds structured bug report template.
.github/ISSUE_TEMPLATE/benchmark-request.yml	Adds structured benchmark/converter request template.
.editorconfig	Adds repository-wide editor defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive project structure for the mlx-benchmarks harness, including configuration files, issue templates, a pre-commit configuration, and documentation. The changes establish a standardized envelope schema for benchmark results and a publisher CLI to upload these results to a HuggingFace dataset. I have identified a few issues: the pre-commit configuration uses invalid revision tags for hooks, there is a typo in the repository URL within the contribution guide, and the security policy references a non-existent configuration file.

[JacobPEvans-personal]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}