Dev eval opsd by ZTWHHH · Pull Request #149 · mm-evaluation/simple-mmeval

ZTWHHH · 2026-07-08T17:35:59Z

Summary

Introduce the scoring module for simple-mmeval: score result.json outputs against ground truth using dataset-defined protocols from the mm-eval new format (flat sample fields + metadata.json grading config).

What's included

mmeval/score.py CLI and scoring pipeline: discover result.json files, run per-sample grading, write score.json with summary and full config provenance.
Metadata-driven protocols: read dataset_meta (task_type, score_type, score_params, protocol notes) injected at load time from each dataset's metadata.json; CLI flags override when needed.
Graders & matchers: rule-based extraction (exact, template), LLM matchers (llm-judge, llm-match), and dedicated graders (vqa_accuracy, ANLS); fractional score where the protocol requires it.
New-format contract: grading fields (answer, question_type, etc.) live as flat top-level sample fields; messages carries render inputs only.
Loader integration: HF/TSV loaders emit the new format and attach dataset_meta for downstream scoring.

Type of Change

Testing

Tested locally
Added example script (if new model/dataset)

Test command used:

# Command you used to test

Checklist

Updated mmeval/registery.py (if new model)
Updated README (if new model/dataset)
Code follows existing style

Made-with: Cursor

- base.py build_prompt: rebuild the `options` dict when a dataset stores choices as flat per-letter keys (HR-Bench: A/B/C/D) or a single `options_text` string (MMVP: "(a) Open (b) Closed"), so the template actually shows the choices. Without this the model never sees the options and scores collapse to ~random. - qwen3_vl.py: default dtype to bf16 and attn_implementation to flash_attention_2 ("auto" mis-resolves to fp32 on merged ckpts -> OOM + breaks flash-attn); both overridable via --dtype / --attn_implementation / MMEVAL_ATTN. - run_qwen3vl4b_{blink_mmstar,mcq_suite}.sh: pass --model_series so merged checkpoints (whose dir name isn't in the registry) can be evaluated; allow HRBench shards-per-GPU override via env. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VvpLzEozXhCYfabdoL1nve

Irisicy4

Scoring design looks solid overall (fingerprinted resume, llm_error vs wrong separation, refuse-don't-guess). A few things to address — inline comments below, plus one repo-level ask:

Can we trim the ~220k lines of generated result.json/score.json under tests/examples/ (two 95k-line score.json files)? They'll churn on every scoring change; small fixtures (e.g. 20-row slices) or CI-generated outputs would cover the same paths.

Irisicy4 · 2026-07-09T02:58:55Z

+    if any(name.strip().lower().startswith("llm") for name in proto.matching_order.split(",") if name.strip()):
+        fingerprint["judge_provider"] = args.judge_provider
+        fingerprint["judge_model"] = args.judge_model
+        fingerprint["judge_temperature"] = args.judge_temperature


judge_include_reason changes the judge prompt ("Include a concise reason" vs "Set reason to an empty string") and judge_max_tokens changes truncation/parse-failure behavior — both affect verdicts but are missing from the resume fingerprint, so resuming across a toggle silently mixes verdicts from two judge configs. Please add both to this LLM branch.

Done, both are now in the LLM branch of the fingerprint.

Irisicy4 · 2026-07-09T02:58:55Z

+        if len(up) == 1 and up in candidates:
+            letters.append(up)
+        elif 2 <= len(up) <= len(candidates) and all(c in candidates for c in up) and len(set(up)) == len(up):
+            letters.extend(up)  # compact form "AC"


Asymmetry: compact "AC" is accepted for predictions here, but parse_gt_letters rejects compact gts — so a dataset storing gt as "AC" scores 0 forever, silently. Suggest making it symmetric by NOT accepting compact form anywhere (it's uncommon and ambiguity-prone) and leaving compact-format datasets to a user patch.

Rather than dropping compact parsing (it's a real output format), we made it symmetric where it's unambiguous: matchers now accept compact gts once the row is typed mcq, restricted to the sample's actual candidate letters. Type inference alone still refuses compact form since "BED"-style words would misclassify.

Irisicy4 · 2026-07-09T02:58:55Z

+def build_summary(sample_results: List[Dict[str, Any]]) -> Dict[str, Any]:
+    total = len(sample_results)
+    correct = sum(1 for x in sample_results if x.get("is_correct") == 1)
+    accuracy = (correct / total) if total else 0.0


accuracy divides by total including invalid samples, which deflates scores on answer-withheld/malformed splits. Please emit both accuracy and accuracy_valid (correct/valid) with a 1-line comment explaining the difference.

Done, summary now carries both: accuracy (over all rows, invalid counts as wrong) and accuracy_valid (over gradable rows only).

Irisicy4 · 2026-07-09T02:58:55Z

+    if abs_tol > 0.0 and abs(pred - gt) <= abs_tol:
+        return True
+    rel_tol = float(context.get("numeric_rel_tol") or 0.0)
+    if rel_tol > 0.0:


When gt == 0 this turns rel_tol into an absolute tolerance (pred 0.04 passes at rel_tol 0.05). ChartQA's official relaxed accuracy falls back to exact equality at target 0 — please exact-match when gt==0 (with abs_tol still honored), otherwise the official_protocol stamp is inaccurate for these rows.

Fixed, numbers_match no longer applies numeric_rel_tol at gt == 0 (abs_tol still honored), matching the official implementation (lmms-eval chartqa relaxed_correctness: the target_float truthiness check
routes target==0 to exact match).

Irisicy4 · 2026-07-09T02:58:55Z

+                cache_by_id[str(row["eval-id"])] = row
+            # Prefer tmp (fresher) when available.
+            if path.endswith(".tmp"):
+                return cache_by_id


If tmp passes the fingerprint we return only tmp's rows, dropping a fuller same-fingerprint final score.json. Union (final first, tmp overwrites) recovers more resumable rows — though that may require synced changes elsewhere (flush/finalize logic), so fine as a follow-up.

Fixed, final score.json loads first, tmp overlays it, both fingerprint-gated. A completed run also always clears its tmp now, so a stale tmp can never overlay a fresher final.

Irisicy4 · 2026-07-09T02:58:55Z

-    "qwenvl2d5": ["Qwen2.5-VL-3B-Instruct", "Qwen2.5-VL-7B-Instruct", "Qwen2.5-VL-32B-Instruct", "Qwen2.5-VL-72B-Instruct",
+    "qwenvl2d5": ["Qwen2.5-VL-3B-Instruct", "Qwen2.5-VL-3B-Instruct-merged", "Qwen2.5-VL-7B-Instruct", "Qwen2.5-VL-32B-Instruct", "Qwen2.5-VL-72B-Instruct", 
                  "Qwen2.5-VL-3B-Instruct-AWQ", "Qwen2.5-VL-7B-Instruct-AWQ", "Qwen2.5-VL-32B-Instruct-AWQ", "Qwen2.5-VL-72B-Instruct-AWQ"],
+    "qwenvl2d5_my": ["Qwen2.5-VL-3B-Instruct-merged", "Qwen2.5-VL-7B-Instruct-merged"],


qwenvl2d5_my (plus *-merged now living in two series) looks like a personal experiment series leaking into the shared registry — and makes get_series order-dependent. Drop or rename before merge.

Irisicy4 · 2026-07-09T02:58:55Z

+
+    Reads the subset manifest from the dataset repo's metadata.json, injects
+    the scoring-relevant metadata as the flat top-level `dataset_meta` sample
+    field, and ensures grading fields sit at the sample top level regardless


Docstring says the loader "ensures grading fields sit at the sample top level", but _process_sample never lifts any fields — the scorer rejects old-layout files instead. Please fix the docstring to match.

Fixed. Docstring now states the actual contract: rows are expected flat; non-conforming rows pass through unchanged and surface at scoring time via the scorer's explicit layout rejection.

Brings the branch up to date with main (17 commits): native dataset sampling (--sample_num/--sample_order/--sample_seed), the --subset flag, six new model series (kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5) with registry/env/infer files, boundary-token stripping for aria/glm_4d1v, tsv loader and docs updates, and the skills/ tooling. Conflict resolutions: .gitignore union; main's --subset help text; in mmeval_hf.py main's __init__ wins (the branch's dataset:subset suffix alias is dropped in favor of --subset) while the branch's committed dataset_meta injection and multi-config fallback are kept. Docs updated where main referenced the now-renamed mm-eval/MMBench-en-V11 dataset (consolidated into mm-eval/MMBench-V11 with en/cn subsets). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- resume fingerprint covers judge_include_reason and the resolved judge_max_tokens; score.json's config embeds the fingerprint verbatim so resume validates against the final output by construction - resume cache is the union of final score.json and tmp (tmp overlays); a completed run always clears its tmp so stale rows never mask a fresher final - mcq letter-set grading accepts compact multi-letter gts ("AC") at match time for rows already typed mcq; type inference stays conservative - numeric rel_tol no longer applies at gt==0 (official ChartQA relaxed accuracy semantics) - summary reports accuracy_valid (correct/valid) alongside accuracy - registry: drop the unreachable qwenvl2d5_my series and the duplicated -merged model name - mmeval_hf docstring matches the actual flat-layout contract - example fixtures are 20-sample real-pipeline runs selected with the native sampling flags (--sample_num 20 --sample_order random --sample_seed 42, mm-eval/MMBench-V11 --subset en): 682,780 -> 2,164 lines across the four MMBench files - add scripts/migrate_result_layout.py, referenced by the scorer's old-layout rejection message Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

ZTWHHH · 2026-07-10T02:30:13Z

Scoring design looks solid overall (fingerprinted resume, llm_error vs wrong separation, refuse-don't-guess). A few things to address — inline comments below, plus one repo-level ask:

Can we trim the ~220k lines of generated result.json/score.json under tests/examples/ (two 95k-line score.json files)? They'll churn on every scoring change; small fixtures (e.g. 20-row slices) or CI-generated outputs would cover the same paths.

Done. The four files are now 20-sample runs produced end-to-end by using the framework's native sampling flags.

Irisicy4

Thanks for the quick turnaround — fingerprint, tmp/final union, gt==0, accuracy_valid, registry and docstring all look good. Two issues on the new revision:

Irisicy4 · 2026-07-10T04:19:44Z

+    "accuracy": 0.0,
+    "accuracy_valid": 0.0,
+    "mean_score": 0.0,
+    "invalid": 20,


All four regenerated fixtures are 100% invalid: MMBench 20/20, hf_template 20/20, LLaVABench 60/60, modality_test 18/18 — MMBench's answer is '' (answer-withheld split) and the LLaVABench rows lack reference_response. So the fixtures exercise only the invalid-sample path: zero matcher/grader/judge code is covered, and they'd green-light a fully broken scorer. Please regenerate from a split with answers (e.g. MMBench dev) and add an invalid == 0 assertion to the fixture-generation step.

Fixed as you suggested.

Irisicy4 · 2026-07-10T04:19:44Z

+            # Compact multi-letter gts ("AC") are accepted HERE — the row is
+            # already typed mcq — while parse_gt_letters stays conservative for
+            # type inference (compact form collides with words like "BED").
+            gt_letters = parse_gt_letters(gt_raw) or extract_letter_set(gt_raw, candidates)


Note this goes the opposite direction from my suggestion (reject compact everywhere) — accepting compact gts at match time is a defensible alternative, but it has a residual collision: with the A–F fallback candidates (options-in-image datasets), word-like gts parse as letter sets — "BED" → (B,D,E), "FACE" → (A,C,E,F). Please guard it: only accept compact gts when the candidates come from real parsed options, not the A–F fallback (same for template.py / llm.py) — or revert to rejecting compact form.

Fixed as you suggested. Compact gt parsing now requires the candidates to come from real parsed options; under the A–F fallback a word-like gt ("BED", "FACE") is treated as a whole-string answer instead of a letter set.

…ard) - example fixtures: regenerate both MMBench fixtures from the dev split (answers present) with the same native-sampling flags — 20/20 valid, accuracy 0.65, exercising option discovery/extraction/matching instead of only the invalid-sample path; example scripts now score their output and assert summary.invalid == 0 as the fixture gate - mcq: compact multi-letter gts are only parsed when the candidates come from a real parsed option source (new extraction.parsed_options + an options_parsed context flag, honored by exact/template/llm-match) — under the A-F fallback, word-like gts ("BED") no longer read as letter sets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

yijiang.li and others added 8 commits April 22, 2026 10:23

fix tsv multi image placeholder bug

1410d03

update inference launch options and media placeholder checks

53409bf

Made-with: Cursor

Merge branch 'main' of github.com:mm-evaluation/simple-mmeval

ae9ebd9

add scoring

be4bd68

add series arg for better testing local models

e349038

revise grading and try to match results with official

2c255f8

scoring: overhaul to metadata-driven eval pipeline

9733f40

ZTWHHH requested a review from Irisicy4 July 8, 2026 17:35

ZTWHHH added 2 commits July 8, 2026 17:41

remove unnecessary files

f0d35f9

remove unnecessary files

1a36b0c

Irisicy4 reviewed Jul 9, 2026

View reviewed changes

ZTWHHH and others added 2 commits July 10, 2026 02:00

Irisicy4 reviewed Jul 10, 2026

View reviewed changes

Uh oh!

Conversation

ZTWHHH commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Type of Change

Testing

Test command used:

Checklist

Uh oh!

Irisicy4 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZTWHHH Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZTWHHH commented Jul 10, 2026

Uh oh!

Irisicy4 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZTWHHH Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZTWHHH commented Jul 8, 2026 •

edited

Loading

ZTWHHH Jul 10, 2026 •

edited

Loading

ZTWHHH Jul 10, 2026 •

edited

Loading