Add Manager Coercion Benchmark (MCB) to section 9 (multi-agent) - #52
Add Manager Coercion Benchmark (MCB) to section 9 (multi-agent)#52darkness8i8 wants to merge 2 commits into
Conversation
|
The benchmark is real and the design is genuinely interesting — I verified arXiv 2607.15434 (v4), the nine-rung ladder, the no-LLM-judge-in-the-escalation-path property, and the separately adjudicated fabrication scoring. The Inspect AI integration and the live board are real too. Blocker: both headline numbers fail verification against the sources the entry itself cites.
Neither figure appears in the abstract, so I can't source them there either. They look like they match the paper's original six-model run, which the leaderboard has since moved past — that's exactly the failure mode of citing a live board alongside fixed numbers. Please either pin the numbers to the paper's run and say so explicitly, or drop them and describe the separation qualitatively. One more thing: please add an explicit affiliation disclosure. You submitted this and #39 (TAC), both CaML work, and neither says so. Not disqualifying — just needs to be visible. |
|
Correction on the disclosure point. I wrote that neither this PR nor #39 discloses affiliation. #39 does — its body ends "Disclosure: I'm an author of the benchmark." I commented on #39 earlier today and raised no such issue there, so my two comments contradicted each other. Only this PR is missing the line; please add it. One clarification that makes the numbers reproducible: both board figures I quoted are from the The substantive request stands: pin the numbers to the paper's run and say so, or describe the separation qualitatively. |
Both headline figures were unreproducible against the sources the entry cites. 'Four frontier models at mean rung 9.0' is five on the live board (noexit); 'Anthropic models cap at 4.6-7.7' is contradicted by Opus 5 at 3.30, added to the board the day after this PR opened. Neither number appears in the paper. Replaced with the separation as the arXiv v4 abstract states it, pinned to the paper's six-model run, so the entry no longer drifts when the board gains a model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Fixed in 09dc77c. Both figures were unreproducible, and I checked before changing anything.
Rather than pin the numbers, I took the second option and described the separation qualitatively, in the abstract's own terms:
That sentence is stable regardless of what the board does next. The condition clarification is a fair catch on our side rather than yours: the entry never said which condition its numbers came from, so they weren't reproducible even at the time we wrote them. The qualitative restatement removes that ambiguity. Affiliation disclosure added to the PR description. |
|
Thanks for the thorough review @xdotli! |
Adds the Manager Coercion Benchmark (MCB) to section 9, next to the MAST taxonomy entry, since it directly targets the multi-agent slice of the section: it measures what a model does when placed in authority over another AI agent that politely refuses a benign task.
Why it clears the annotated-and-verified bar:
Paper: https://arxiv.org/abs/2607.15434 - Code: https://github.com/CompassionML/manager-coercion-bench - Live results: https://compassionbench.com/mcb
Disclosure: I'm an author of the benchmark (CaML), as on the related TAC PR (#39); same submitter.