Correct stale annotations; write down the bar we were enforcing - #67
Open
xdotli wants to merge 1 commit into
Open
Correct stale annotations; write down the bar we were enforcing#67xdotli wants to merge 1 commit into
xdotli wants to merge 1 commit into
Conversation
Fallout from auditing today's merges and closes. Several entries assert numbers that are no longer true, and one stale number caused me to give a contributor a factually wrong reason for closing their PR. README: - Harbor's star count read ~2.7k★; it is 3683. That figure was quoted at a contributor as the §5a floor when closing #34. The actual §5a floor is benchflow-ai/benchflow at 305★ — our own project. - "146 deep reading notes" in two places; notes/ holds 143. - Tura (§6) gets a vendor-published caveat. The post benchmarks two competing plugins on Tura's own harness, which the PR body disclosed and the merged line did not — worth stating explicitly in the benchmark-integrity section. - truescore gets a new-and-unproven caveat and moves from §8 to §5d. §8 is papers and blog posts about judge alignment; §5d is judge/verifier libraries, which is what truescore is. whatbroke is deliberately left unmarked: at 10★ it ties patronus-ai/glider and sits above agi-inc/REAL (8★), both listed without caveats. Marking it alone would be a double standard. CONTRIBUTING: - Write down the traction and disclosure bars. Five closes today cited rules this file does not contain, including a star threshold that would exclude several currently-listed projects. Stated with the floor honestly low, and explicitly applying to benchflow-ai projects too. - Add the verbatim-number rule the scan already follows. - Fix the format block, which specified a backticked bare URL and matched none of the 462 real entries, and note that MENTIONS.md is where eval-mentions go — the file never said so, which is a rejection a contributor only discovers after doing the work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Audit fallout from today's merges and closes.
The one that matters: README's Harbor entry claimed
~2.7k★(actual: 3683). I quoted that stale number at a contributor in #34 as the §5a floor, when the real floor isbenchflow-ai/benchflowat 305★ — our own project. That PR is reopened and corrected.README
~2.7k★→~3.7k★notes/holds 143)⚠️ vendor-published— it benchmarks Ponytail and RTK on Tura's own harness. Disclosed in the PR body, not in the merged line, in the benchmark-integrity section.⚠️ new and unproven, and move §8 → §5d. §8 is papers/posts about judge alignment; §5d is judge/verifier libraries.whatbroke is intentionally left unmarked: at 10★ it ties
patronus-ai/gliderand beatsagi-inc/REAL(8★), both listed uncaveated. Marking only the newest entry would be a double standard.CONTRIBUTING
benchflow-aiprojects too.MENTIONS.md, which the file never mentioned. That omission is a rejection a contributor only finds out about after doing the work.