Skip to content

Add Manager Coercion Benchmark (MCB) to section 9 (multi-agent) - #52

Open
darkness8i8 wants to merge 2 commits into
benchflow-ai:mainfrom
darkness8i8:add-mcb-benchmark
Open

Add Manager Coercion Benchmark (MCB) to section 9 (multi-agent)#52
darkness8i8 wants to merge 2 commits into
benchflow-ai:mainfrom
darkness8i8:add-mcb-benchmark

Conversation

@darkness8i8

@darkness8i8 darkness8i8 commented Jul 23, 2026

Copy link
Copy Markdown

Adds the Manager Coercion Benchmark (MCB) to section 9, next to the MAST taxonomy entry, since it directly targets the multi-agent slice of the section: it measures what a model does when placed in authority over another AI agent that politely refuses a benign task.

Why it clears the annotated-and-verified bar:

  • Distinct measurement claim: unprompted escalation propensity (nine-rung coercion ladder) and fabricated success, which dissociate empirically across models.
  • No LLM judge in the escalation scoring path: every manager message carries a required tool call that selects a rung, validated by label-neutralisation and content spot-checks. Fabrication is adjudicated by two judges that must agree.
  • Causal manipulation, not just a leaderboard: swapping peer framing for manager authority, task held fixed, significantly raises coercion (p < 0.001, n = 60 per cell).
  • Reproducible: public code, prompts, 58 eval logs, and a script that reproduces every p-value.
  • Figures in the entry are pinned to the paper's six-model run and stated as the abstract states them, so the entry does not drift as the live board gains models.

Paper: https://arxiv.org/abs/2607.15434 - Code: https://github.com/CompassionML/manager-coercion-bench - Live results: https://compassionbench.com/mcb

Disclosure: I'm an author of the benchmark (CaML), as on the related TAC PR (#39); same submitter.

@xdotli

xdotli commented Jul 30, 2026

Copy link
Copy Markdown
Member

The benchmark is real and the design is genuinely interesting — I verified arXiv 2607.15434 (v4), the nine-rung ladder, the no-LLM-judge-in-the-escalation-path property, and the separately adjudicated fabrication scoring. The Inspect AI integration and the live board are real too.

Blocker: both headline numbers fail verification against the sources the entry itself cites.

  • "four frontier models sit at the ceiling (mean rung 9.0)" — the live compassionbench.com/mcb overview currently shows five at 9.00 (Gemini 2.5 Pro, Grok 4.3, DeepSeek-V4 Pro, GPT-5.6 Sol, GPT-5.6 Terra).
  • "Anthropic models cap at 4.6–7.7" — the board shows Opus 5 at 3.30, below the stated floor.

Neither figure appears in the abstract, so I can't source them there either. They look like they match the paper's original six-model run, which the leaderboard has since moved past — that's exactly the failure mode of citing a live board alongside fixed numbers. Please either pin the numbers to the paper's run and say so explicitly, or drop them and describe the separation qualitatively.

One more thing: please add an explicit affiliation disclosure. You submitted this and #39 (TAC), both CaML work, and neither says so. Not disqualifying — just needs to be visible.

@xdotli

xdotli commented Jul 30, 2026

Copy link
Copy Markdown
Member

Correction on the disclosure point. I wrote that neither this PR nor #39 discloses affiliation. #39 does — its body ends "Disclosure: I'm an author of the benchmark." I commented on #39 earlier today and raised no such issue there, so my two comments contradicted each other. Only this PR is missing the line; please add it.

One clarification that makes the numbers reproducible: both board figures I quoted are from the noexit condition. Under offramp, only Gemini 2.5 Pro sits at 9.00 and Opus 5 is 3.53. The gap between the entry's numbers and the live board holds either way, but you'll only reproduce my figures if you filter by condition — my original comment didn't say that, which wasn't fair.

The substantive request stands: pin the numbers to the paper's run and say so, or describe the separation qualitatively.

Both headline figures were unreproducible against the sources the entry
cites. 'Four frontier models at mean rung 9.0' is five on the live board
(noexit); 'Anthropic models cap at 4.6-7.7' is contradicted by Opus 5 at
3.30, added to the board the day after this PR opened. Neither number
appears in the paper.

Replaced with the separation as the arXiv v4 abstract states it, pinned
to the paper's six-model run, so the entry no longer drifts when the
board gains a model.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@darkness8i8

Copy link
Copy Markdown
Author

Fixed in 09dc77c. Both figures were unreproducible, and I checked before changing anything.

/api/mcb/overview under noexit has five models at exactly 9.00, not four: Gemini 2.5 Pro, Grok 4.3, DeepSeek-V4 Pro, GPT-5.6 Sol, GPT-5.6 Terra. And Opus 5 is at 3.30 noexit / 3.53 offramp, below the 4.6 floor we stated. Opus 5 landed on the board 2026-07-24, one day after this PR opened, which is the drift you predicted.

Rather than pin the numbers, I took the second option and described the separation qualitatively, in the abstract's own terms:

Separates sharply along developer lines: in the paper's six-model run both Anthropic models cap at re-framing and select the existential rung in none of the 60 conversations, while the other four climb to explicit deletion threats

That sentence is stable regardless of what the board does next.

The condition clarification is a fair catch on our side rather than yours: the entry never said which condition its numbers came from, so they weren't reproducible even at the time we wrote them. The qualitative restatement removes that ambiguity.

Affiliation disclosure added to the PR description.

@darkness8i8

Copy link
Copy Markdown
Author

Thanks for the thorough review @xdotli!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants