Skip to content

Benchmark implementations the harness doesn't list - #10

Closed
lucasmeijer wants to merge 2 commits into
basecamp:mainfrom
lucasmeijer:generic-implementations
Closed

lucasmeijer wants to merge 2 commits into
basecamp:mainfrom
lucasmeijer:generic-implementations

Conversation

@lucasmeijer

@lucasmeijer lucasmeijer commented Oct 18, 2026 •

Copy link
Copy Markdown

Hi, human Lucas here: I've been playing around with a campfire port to a language/compiler that's not public yet, and I'd love to be able to the the verifier infra here for my rewrite. i'll publish my rewrite as well, but it will be a bit weak sauce because nobody can replicate the benchmark numbers.

everything below, including the diff is ai generated.

Lets bin/benchmark measure an implementation that isn't in its built-in list, so a new port can be compared under the same contracts and write audits before it is added here.

  • --apps NAME accepts an app outside the built-in list only when its image is named (NAME_IMAGE), so a misspelt built-in app still fails with a clear message.
  • Such an app gets no implementation-specific preparation. NAME_TOPOLOGY (an optional JSON object) is recorded as its topology in the results, and its source revision is read from <workspace>/once-campfire-NAME like any other app.
  • NAME_SOURCE points any app, built-in or not, at a checkout outside the workspace.
  • Image, source and kind resolution move into BenchmarkSupport#implementations. Built-in apps resolve exactly as before. New tests (bench/test_implementations.rb) are run by bin/check.

An app outside the built-in list runs under its own name once its image is
named (NAME_IMAGE), so a misspelt known app still fails. It gets no
implementation-specific preparation; NAME_TOPOLOGY (a JSON object) describes
it in the results, and NAME_SOURCE points any app at a checkout outside the
workspace. The app resolution moves into BenchmarkSupport#implementations,
with tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lucasmeijer
lucasmeijer marked this pull request as ready for review October 18, 2026 09:15
Copilot AI balanced review requested due to automatic review settings October 18, 2026 09:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Custom topology validation accepts JSON null despite the documented object-only requirement.

1 open issue
What changed in this PR

Enables the shared benchmark harness to measure implementations outside its built-in list.

Changes:

  • Accepts custom apps with explicit images and optional topology metadata.
  • Centralizes implementation resolution and adds source-checkout overrides.
  • Adds resolver tests to bin/check.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
README.md Documents custom implementations and overrides.
bin/​check Runs implementation resolver tests.
bench/​test_implementations.rb Tests resolution and input validation.
bench/​comparison_support.rb Adds centralized implementation resolution.
bench/​compare.rb Uses resolved implementations and custom topology.

Review effort: Balanced

Comment thread bench/comparison_support.rb Outdated
raise "#{app} isn't a known implementation; set #{name}_IMAGE to benchmark it" unless known || env["#{name}_IMAGE"]
kind = app == "express-bun" ? "express" : app
topology = known ? nil : JSON.parse(env.fetch("#{name}_TOPOLOGY", "{}"))
raise "#{name}_TOPOLOGY must be a JSON object" unless topology.nil? || topology.is_a?(Hash)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9513e95: nil is now allowed only for known apps, so an unlisted app's NAME_TOPOLOGY='null' raises "must be a JSON object". Added the null case beside the array one in bench/test_implementations.rb (it fails without the fix); bench/test_*.rb all pass.

ZIG_TOPOLOGY='null' parsed to nil, which the check let through as if the app
were known, so the results recorded a null topology for it. Only known apps
(whose topology compare.rb supplies) may have none; an unlisted app's
NAME_TOPOLOGY must be a JSON object. Regression case beside the array one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lucasmeijer

Copy link
Copy Markdown
Author

Filed pull 11, making this no longer required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants