revert: remove the minimal profile and ship one shape - #313
Merged
Merged
Conversation
This reverts commit 6bfa9a4 (#312). The profile existed so the harness could measure itself in a bare shape comparable to the baselines benchmark frameworks already ship. That premise does not hold. Leaderboards score (model, agent) pairs, so a harness competes as exactly the shape it ships, and the bare baselines are separate pinned agents inside the frameworks themselves. A self-authored bare shape produces a number nobody reads as the model's and nobody wants as the product's. Its prefix was byte-stable, but the run was not neutral either: permission rules, approval mode, and every settings field still varied per install. What it cost while it existed: a second prompt with its own budget test, a second registry assembly, two extra ledger-attribution branches in the refreeze path, a hooks suppression flag threaded through re-discovery, and a four-way CLI validation matrix. One shape remains, the harness as shipped, which is the thing any published number should describe.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…ape refreeze A manifest written by the one-day build with a second session shape parsed as a full one after the revert, while its transcript kept the bare one-line prompt: a session resumed as seven tools under a prompt written for four, neither shape, and with the fingerprint matching disk nothing would ever repair it. The manifest version gate is the designed retirement path for a persisted shape: reading v4 as absent puts the session on built-ins until the next turn start, which refreezes prompt and manifest from disk together. The regression test forges the retired record and shows the first turn-start check rebuilding the full prompt beside the full tool set and writing a current-version manifest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#312 added
--profile minimal, a second shape of the harness (four tools, a bare prompt, no hooks) so it could measure itself in a form comparable to bare benchmark baselines. Checked against the benchmark frameworks, the premise does not hold: leaderboards score (model, agent) pairs, so a harness competes as exactly the shape it ships, and the bare baselines are separate, pinned agents inside the frameworks themselves. A self-authored bare shape yields a number nobody reads as the model's and nobody wants as the product's. Nor was the run neutral: with byte-identical prompt bytes, permission rules, approval mode, and every settings field still varied per install.Summary
git revertof feat: add a minimal profile for measurement runs #312: one shape remains, the harness as shipped.Profileenum and manifest field, the second registry assembly and second prompt with its own budget test, the hooks suppression flag threaded through re-discovery, the two extra ledger-attribution branches in the refreeze path, the fail-closed manifest read-back, the--profileflag and its four-way validation, the/statusrow, and the docs.Test Plan
cargo test --workspace --locked: 943 passed, 0 failed, 11 ignored (the count before feat: add a minimal profile for measurement runs #312 plus the new regression test).cargo +1.97.0 clippy --workspace --all-targets --locked -- -D warnings: clean.git grepfor every identifier the profile introduced returns nothing.Greptile Summary
This change retires manifest version 4 so sessions created with the removed minimal profile are rebuilt into the single supported session shape. It also adds regression coverage for replacing the retired bare prompt and persisting the current manifest version.
Confidence Score: 5/5
Safe to merge.
The previous session-migration concern was resolved by greptile-apps[bot] without explanation. The current change retires version-4 manifests and covers the legacy-session refreeze path; no outstanding findings remain.
Reviews (2): Last reviewed commit: "fix(core): retire manifest v4 so session..." | Re-trigger Greptile