Discrepancy between Paix2-router.json's MiniMax-M3 results and reproduction via OpenRouter's minimax provider

[nilam-divyam]

We attempted to reproduce MiniMax-M3's per-query results from Paix2-router.json (PR #164) for the 760 queries Paix2 routed to MiniMax-M3, and observed a large, consistent discrepancy in both accuracy and input token counts.

Setup: All 760 prompts were verified byte-for-byte identical between Paix2-router.json and our own collection. We called minimax/minimax-m3 via OpenRouter, provider-pinned to minimax (not MiniMax's native Anthropic-compatible API, which Paix2's submitted llm_inference/model_inference.py._call_minimax uses instead) across three settings, isolating temperature and reasoning-effort as variables:

variant	temperature	reasoning	max_tokens	n	accuracy	input tokens (mean)	output tokens (mean)
Paix2 (native MiniMax API)	Not set	Not set	2048	760	84.12%	268.9	132.4
OpenRouter, provider = Minimax	0.0	"none"	2048	760	65.11%	390.5	141.7
OpenRouter, provider = Minimax	1.0	"none"	2048	760	62.82%	390.5	130.3
OpenRouter, provider = Minimax	Not set	Not set	2048 → 8192**	691*	69.52%	387.4	735.5

* fewer than 760 because of empty responses where unconstrained reasoning consumed the entire completion budget before an answer was produced.

** the queries that came back empty at max_tokens=2048 were retried at max_tokens=8192.

Findings

1. Input token count: despite byte-identical prompts, our OpenRouter-routed requests consistently report ~35–45% more input tokens than Paix2's. A linear fit across all 760 gives ours_input ≈ 0.995 × paix2_input + 123, R²=0.98 → a near-constant ~123-token additive gap, not a per-token tokenizer-efficiency difference. We could not attribute this to any documented OpenRouter request parameter (temperature, reasoning, transforms/plugins all ruled out).
2. Accuracy: none of our three OpenRouter-routed configurations get within 12 points of Paix2's reported 84.12%, even after isolating and testing every parameter we control (temperature: no effect; reasoning effort: some effect, but higher token count; token budget: recovers coverage, not accuracy).
3. Temperature is conclusively ruled out as a factor, with no improvement observed in accuracy.

We can't rule out that MiniMax's native Anthropic-compatible endpoint (which Paix2 uses) is genuinely more capable/better-configured than OpenRouter's minimax provider route for this model — that would be a legitimate, reproducible reason for the gap, and we did not test the native endpoint ourselves. We're raising this because the size and consistency of the gap (~19pp accuracy, ~123-token systematic input-token offset) seemed worth documenting for anyone else trying to reproduce or audit MiniMax-M3 results on the leaderboard, and to ask: is there guidance on which endpoint/config submissions should standardize on for reproducibility?