Repository navigation
Benchmark implementations the harness doesn't list - #10
lucasmeijer wants to merge 2 commits into
Conversation
An app outside the built-in list runs under its own name once its image is named (NAME_IMAGE), so a misspelt known app still fails. It gets no implementation-specific preparation; NAME_TOPOLOGY (a JSON object) describes it in the results, and NAME_SOURCE points any app at a checkout outside the workspace. The app resolution moves into BenchmarkSupport#implementations, with tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Custom topology validation accepts JSON null despite the documented object-only requirement.
1 open issue
What changed in this PR
Enables the shared benchmark harness to measure implementations outside its built-in list.
Changes:
- Accepts custom apps with explicit images and optional topology metadata.
- Centralizes implementation resolution and adds source-checkout overrides.
- Adds resolver tests to
bin/check.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
| README.md | Documents custom implementations and overrides. |
| bin/check | Runs implementation resolver tests. |
| bench/test_implementations.rb | Tests resolution and input validation. |
| bench/comparison_support.rb | Adds centralized implementation resolution. |
| bench/compare.rb | Uses resolved implementations and custom topology. |
Review effort: Balanced
| raise "#{app} isn't a known implementation; set #{name}_IMAGE to benchmark it" unless known || env["#{name}_IMAGE"] | ||
| kind = app == "express-bun" ? "express" : app | ||
| topology = known ? nil : JSON.parse(env.fetch("#{name}_TOPOLOGY", "{}")) | ||
| raise "#{name}_TOPOLOGY must be a JSON object" unless topology.nil? || topology.is_a?(Hash) |
There was a problem hiding this comment.
Fixed in 9513e95: nil is now allowed only for known apps, so an unlisted app's NAME_TOPOLOGY='null' raises "must be a JSON object". Added the null case beside the array one in bench/test_implementations.rb (it fails without the fix); bench/test_*.rb all pass.
ZIG_TOPOLOGY='null' parsed to nil, which the check let through as if the app were known, so the results recorded a null topology for it. Only known apps (whose topology compare.rb supplies) may have none; an unlisted app's NAME_TOPOLOGY must be a JSON object. Regression case beside the array one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Filed pull 11, making this no longer required. |

Hi, human Lucas here: I've been playing around with a campfire port to a language/compiler that's not public yet, and I'd love to be able to the the verifier infra here for my rewrite. i'll publish my rewrite as well, but it will be a bit weak sauce because nobody can replicate the benchmark numbers.
Lets
bin/benchmarkmeasure an implementation that isn't in its built-in list, so a new port can be compared under the same contracts and write audits before it is added here.--apps NAMEaccepts an app outside the built-in list only when its image is named (NAME_IMAGE), so a misspelt built-in app still fails with a clear message.NAME_TOPOLOGY(an optional JSON object) is recorded as its topology in the results, and its source revision is read from<workspace>/once-campfire-NAMElike any other app.NAME_SOURCEpoints any app, built-in or not, at a checkout outside the workspace.BenchmarkSupport#implementations. Built-in apps resolve exactly as before. New tests (bench/test_implementations.rb) are run bybin/check.