diff --git a/.gitignore b/.gitignore index 94a66840..63adecaf 100644 --- a/.gitignore +++ b/.gitignore @@ -173,4 +173,7 @@ cython_debug/ # PyPI configuration file .pypirc mydata -.tmp/ \ No newline at end of file +datasets +.claude +work_dirs +.tmp/ diff --git a/docs/en/SCORING.md b/docs/en/SCORING.md new file mode 100644 index 00000000..c95c3bba --- /dev/null +++ b/docs/en/SCORING.md @@ -0,0 +1,141 @@ +# Scoring + +`mmeval/score.py` grades the `result.json` files produced by inference into +`score.json` files: + +```bash +PYTHONPATH=. python3 mmeval/score.py --score_out_dir work_dirs/my_run +``` + +For how well the framework reproduces official published results, see +[VALIDATION.md](VALIDATION.md). + +## The pipeline model + +All scoring is expressed as a **pipeline: an ordered list of atomic stages**. +There are no composite scorer types — a protocol like "rules first, LLM judge +for the rest" is written as the list `exact-match,rule-match,llm-judge`. +The naming scheme: `X-match` = candidate extraction via X (exact form / +rule heuristics / an LLM) followed by rule comparison; `llm-judge` = the LLM +decides correctness directly; `vqa-accuracy` / `anls` = named official +metrics (grader stages). + +| Stage | Mechanism | What it does | +|---|---|---| +| `exact-match` | rules, free | Strict-form comparison: bare option letter / clean letter set, pure yes/no, whole-string number or text. Near-zero false positives; passes anything ambiguous. | +| `rule-match` | rules, free | Robust answer extraction from verbose / chain-of-thought output (option-letter inference, answer cues, `\boxed{}`, option-text restatement), then comparison. | +| `llm-match` | LLM | The LLM extracts the chosen option letter(s); rule-based set compare (VLMEvalKit protocol). MCQ only; passes non-MCQ samples. | +| `llm-judge` | LLM | The LLM judges correctness directly from question + response + ground truth. Both verdicts decide the sample (yes → 1.0, no → 0.0). | +| `vqa-accuracy` | grader, fractional | Official VQA consensus accuracy (leave-one-out `min(1, matches/3)` over the multi-annotator answer list). | +| `anls` | grader, fractional | ANLS: `1 − min` normalized Levenshtein distance over the references, zeroed below the threshold. | + +The two LLM stages run their judge through a configurable provider +(`--judge_provider`): `local` (an in-process open-weight model loaded with the +framework's own conventions — no external service), `openai`, or +`azure_openai`. The stage logic, prompt, retry, and `llm_error` accounting are +identical across providers. + +### Execution semantics + +Stages run in order; each returns one of three outcomes: + +- **pass** — the stage cannot decide the sample; the next stage runs. If every + stage passes, the sample scores `0.0` with `decided_by: null` (undecided, + counted wrong). Rule stages never decide a miss — they pass instead, so a + later stage can still grade the sample. +- **score** — the stage decided; the pipeline short-circuits. Each sample row + records `score` (0..1), `is_correct` (`1` only at full credit) and + `decided_by` (the stage name). Fractional protocols read + `summary.mean_score` as the headline; `summary.accuracy` is the strict + full-credit rate. +- **invalid** — the sample cannot be graded under the stage's protocol (e.g. + `vqa-accuracy` without the multi-annotator reference list); recorded as + `status: invalid` with the stage's reason. + +The grader stages (`vqa-accuracy`, `anls`) always decide, so they are only +valid as the **last** stage, at most one per pipeline (validated with a clear +error). Compositions like +`exact-match,anls` (exact tier first, fractional similarity for the rest) are +legal pipelines. + +LLM/API failures never become verdicts: the failing stage passes with an +`llm_error: ...` reason that survives into the sample row and +`summary.llm_errors`, and such rows are re-scored on resume — never guessed. + +### CLI + +```bash +--score_pipeline exact-match,rule-match # default +--score_pipeline rule-match,llm-match # MCQ letter-extraction protocol +--score_pipeline llm-judge # pure LLM judging +--score_pipeline vqa-accuracy # official VQA accuracy +``` + +Stage parameters are pipeline-level flags, consumed by the stages that use +them (per-knob precedence: explicit CLI flag > dataset metadata > default; +`score.json config.knob_sources` records who decided each knob): + +| Flag | Consumed by | +|---|---| +| `--score_numeric_rel_tol`, `--score_numeric_abs_tol` | `exact-match`, `rule-match` (numeric answers) | +| `--score_string_match` (`exact`/`contains`/`anls`) | `exact-match`, `rule-match` (open answers) | +| `--score_anls_threshold` | `anls` and the `anls` string-match mode of the rule-based stages | +| `--judge_provider/model/temperature/max_tokens/...` | the LLM stages (`llm-match`, `llm-judge`) | + +## Dataset metadata contract + +mm-eval datasets declare their official protocol per subset in the repo-root +`metadata.json`. The scorer reads it from the `dataset_meta` block the loader +injects into every sample (so it travels inside `result.json`): + +```jsonc +"subsets": { + "en": { + "task_type": "multiple_choice_qa", + "score_pipeline": ["rule-match", "llm-match"], // the official protocol + "score_params": {"numeric_rel_tol": 0.05}, // optional stage params + "score_protocol": {"note": "..."} // verbatim protocol note + } +} +``` + +`score_pipeline` takes one of three forms: + +1. **A stage list** — the official protocol; running it stamps + `config.official_protocol: true`. An explicit `--score_pipeline` that differs + overrides it, stamps the output `official_protocol: false`, and appends a + protocol note saying so. +2. **`[]`** — the dataset explicitly declares that **no official protocol + exists**; the scorer runs the default pipeline and stamps + `official_protocol: false` with a note. +3. **`{"unsupported": "", "reason": "..."}`** — the official + protocol cannot be executed by this scorer (e.g. corpus-level caption + metrics, code execution). Scoring is **refused** with the reason; an + explicit `--score_pipeline` forces approximate scoring, stamped non-official. + +Datasets with no `score_pipeline` declaration score under the default +pipeline with `official_protocol: null` (unknown). + +### Published metadata + +Every dataset on the mm-eval org declares `score_pipeline` natively +(org-wide migration completed 2026-07-19; the loader's temporary +`score_type`→`score_pipeline` translation boundary is deleted). The retired +composite `score_type` vocabulary is rejected everywhere with an actionable +error: the loader refuses a metadata.json that still carries it (re-push the +dataset), and the scorer refuses a `result.json` whose `dataset_meta` +contains it (produced by an older loader — re-run inference or upgrade the +file). + +The audited per-subset pipeline for all published datasets is listed in +[SCORING_COVERAGE.md](SCORING_COVERAGE.md). + +## Resume + +Scoring resumes from `score.json` / `score.json.tmp` caches keyed by a config +fingerprint that captures everything verdict-determining: the resolved +pipeline, stage params, and — only when an LLM stage is present — the +judge identity (provider/model/temperature/include_reason/resolved +max_tokens). Any change invalidates the cache; rule-only pipelines survive +judge-config edits. Rows that failed with `llm_error` are always re-scored. +Reruns of an unchanged config are byte-identical. diff --git a/docs/en/SCORING_COVERAGE.md b/docs/en/SCORING_COVERAGE.md new file mode 100644 index 00000000..a835915b --- /dev/null +++ b/docs/en/SCORING_COVERAGE.md @@ -0,0 +1,224 @@ +# Scoring pipeline coverage — mm-eval org audit + +Survey of every dataset repo on https://huggingface.co/mm-eval: per subset, +the pre-migration composite `score_type` (audited 2026-07-15, retired) and +the atomic `score_pipeline` **now published natively in each repo's +metadata.json** (org-wide migration completed 2026-07-19; the loader's +translation boundary is deleted — see `docs/en/SCORING.md`, "Published +metadata"). Every supported pipeline below was validated against the stage +registry; REFUSED rows raise an explicit, actionable error instead of +silently mis-grading. + +**169 subsets across 128 dataset repos: 158 supported, 11 refused.** (`mm-eval/VLMEvalKit` carries no metadata.json — it is a raw TSV mirror, not an mm-eval-format dataset — and is excluded.) + +## Stage usage census + +Official-protocol demand per atomic stage (datasets / subsets): + +- `exact-match`: 61 datasets / 80 subsets +- `rule-match`: 91 datasets / 127 subsets +- `llm-match`: 30 datasets / 47 subsets +- `llm-judge`: 30 datasets / 39 subsets +- `vqa-accuracy`: 4 datasets / 4 subsets +- `anls`: 4 datasets / 4 subsets + +Narrowest stages: `vqa-accuracy` (VQAv2, OK-VQA, TextVQA, VizWiz-VQA) and +`anls` (DocVQA, InfographicVQA, ST-VQA, ChartNet; the ANLS comparator also +backs `string_match=anls`, e.g. ChartQAPro) — 4 datasets each, above the +1–2-dataset prune threshold, and both are the ecosystem's canonical +fractional metrics. + +| Dataset | Subset | Retired score_type (pre-2026-07-19) | Published score_pipeline | Status | Notes | +|---|---|---|---|---|---| +| 3DSRBench | main | llm_extract | rule-match,llm-match | supported | | +| A-OKVQA | main | rule | exact-match,rule-match | supported | | +| AI2D | main | llm_extract | rule-match,llm-match | supported | | +| AlgoPuzzleVQA | default | rule | exact-match,rule-match | supported | | +| ArxivQA | main | llm_extract | rule-match,llm-match | supported | | +| BLINK | Art_Style | llm_extract | rule-match,llm-match | supported | | +| BLINK | Counting | llm_extract | rule-match,llm-match | supported | | +| BLINK | Forensic_Detection | llm_extract | rule-match,llm-match | supported | | +| BLINK | Functional_Correspondence | llm_extract | rule-match,llm-match | supported | | +| BLINK | IQ_Test | llm_extract | rule-match,llm-match | supported | | +| BLINK | Jigsaw | llm_extract | rule-match,llm-match | supported | | +| BLINK | Multi_view_Reasoning | llm_extract | rule-match,llm-match | supported | | +| BLINK | Object_Localization | llm_extract | rule-match,llm-match | supported | | +| BLINK | Relative_Depth | llm_extract | rule-match,llm-match | supported | | +| BLINK | Relative_Reflectance | llm_extract | rule-match,llm-match | supported | | +| BLINK | Semantic_Correspondence | llm_extract | rule-match,llm-match | supported | | +| BLINK | Spatial_Relation | llm_extract | rule-match,llm-match | supported | | +| BLINK | Visual_Correspondence | llm_extract | rule-match,llm-match | supported | | +| BLINK | Visual_Similarity | llm_extract | rule-match,llm-match | supported | | +| BenchLMM | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 1], 'official_judge_model': 'gpt-4-0613'} | +| CMMMU | art_and_design | rule | exact-match,rule-match | supported | | +| CMMMU | business | rule | exact-match,rule-match | supported | | +| CMMMU | health_and_medicine | rule | exact-match,rule-match | supported | | +| CMMMU | humanities_and_social_sciences | rule | exact-match,rule-match | supported | | +| CMMMU | science | rule | exact-match,rule-match | supported | | +| CMMMU | technology_and_engineering | rule | exact-match,rule-match | supported | | +| CRPE | exist | rule | exact-match,rule-match | supported | | +| CRPE | relation | rule | exact-match,rule-match | supported | | +| CV-Bench | main | rule | exact-match,rule-match | supported | | +| CVQA | main | llm_extract | rule-match,llm-match | supported | | +| CharXiv | reasoning | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| ChartNet | main | anls | anls | supported | score_params: {'threshold': 0.0} | +| ChartQA | main | rule | exact-match,rule-match | supported | score_params: {'numeric_rel_tol': 0.05} | +| ChartQAPro | main | rule | exact-match,rule-match | supported | score_params: {'numeric_rel_tol': 0.05, 'string_match': 'anls', 'anls_threshold': 0.5} | +| DocVQA | main | anls | anls | supported | score_params: {'threshold': 0.5} | +| DynaMath | main | llm_extract | rule-match,llm-match | supported | score_params: {'numeric_abs_tol': 0.001} | +| EMMA | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| ENEM | main | rule | exact-match,rule-match | supported | | +| EXAMS-V | default | rule | exact-match,rule-match | supported | | +| EndoBench | main | llm_extract | rule-match,llm-match | supported | | +| Ferret-Bench | default | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-0314'} | +| Flickr8k | main | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| GMAI-MMBench | default | llm_extract | rule-match,llm-match | supported | | +| GQA | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| GQA-Spatial | main | rule | exact-match,rule-match | supported | | +| Geometry3K | main | none | [] (no official protocol) | supported | default pipeline, stamped non-official | +| HRBench4K | main | llm_extract | rule-match,llm-match | supported | | +| HRBench8K | main | llm_extract | rule-match,llm-match | supported | | +| HallusionBench | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | score_params: {'judge_inputs': ['gt_answer_details']} | +| HallusionBench | non_image | rule_llm_judge | exact-match,rule-match,llm-judge | supported | score_params: {'judge_inputs': ['gt_answer_details']} | +| HumanEval-V | main | execution | unsupported: code-execution | REFUSED | requires official test-case execution (pass@1); use the benchmark's own evaluator (score_params: {'metric': 'pass@1'}) | +| IconQA | choose_img | rule | exact-match,rule-match | supported | | +| IconQA | choose_txt | rule | exact-match,rule-match | supported | | +| IconQA | fill_in_blank | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| InfographicVQA | main | anls | anls | supported | score_params: {'threshold': 0.5} | +| LEGO-Puzzles | default | llm_extract | rule-match,llm-match | supported | | +| LLaVA-Bench-COCO | default | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-0314'} | +| LLaVA-Bench-Wilder | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-vision-preview'} | +| LLaVA-Bench-in-the-Wild | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 10], 'official_judge_model': 'gpt-4-0314', 'judge_inputs': ['caption']} | +| LiveBench | 2024-05 | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o'} | +| LiveBench | 2024-06 | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LiveBench | 2024-07 | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LiveBench | 2024-08 | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LiveBench | 2024-09 | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 10], 'official_judge_model': 'gpt-4o', 'judge_inputs': ['criteria']} | +| LogicVista | main | llm_extract | rule-match,llm-match | supported | | +| M3CoT | main | rule | exact-match,rule-match | supported | | +| MEGA-Bench | main | per_task_metric | unsupported: per-task-metric | REFUSED | requires the benchmark's own evaluator (per-row metric dispatch) (score_params: {'metric_field': 'task_name'}) | +| MM-IFEval | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'judge_inputs': ['constraints']} | +| MM-Vet-v2 | default | llm_judge | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'scale': [0.0, 1.0], 'official_judge_model': 'gpt-4-0613', 'reference_syntax': 'and_or_composition'} | +| MMBench | cc | llm_extract | rule-match,llm-match | supported | | +| MMBench | cn | llm_extract | rule-match,llm-match | supported | | +| MMBench | en | llm_extract | rule-match,llm-match | supported | | +| MMBench-V11 | cn | llm_extract | rule-match,llm-match | supported | | +| MMBench-V11 | en | llm_extract | rule-match,llm-match | supported | | +| MME | main | rule | exact-match,rule-match | supported | | +| MME-RealWorld-Lite | main | rule | exact-match,rule-match | supported | | +| MMEvalPro | main | rule | exact-match,rule-match | supported | | +| MMHal-Bench | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [0, 6], 'official_judge_model': 'gpt-4', 'judge_inputs': ['image_content']} | +| MMIU | default | llm_extract | rule-match,llm-match | supported | | +| MMMU | main | rule | exact-match,rule-match | supported | | +| MMMU-Pro | standard_10_options | rule | exact-match,rule-match | supported | | +| MMMU-Pro | standard_4_options | rule | exact-match,rule-match | supported | | +| MMMU-Pro | vision | rule | exact-match,rule-match | supported | | +| MMStar | main | llm_extract | rule-match,llm-match | supported | | +| MMT-Bench | main | llm_extract | rule-match,llm-match | supported | | +| MMVP | main | llm_extract | rule-match,llm-match | supported | | +| MMVU | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| MMVet | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'scale': [0.0, 1.0], 'official_judge_model': 'gpt-4-0613', 'reference_syntax': 'and_or_composition'} | +| MMVet-v2 | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'partial_credit', 'scale': [0.0, 1.0], 'official_judge_model': 'gpt-4-0613', 'reference_syntax': 'and_or_composition'} | +| MS-COCO-Captions | main | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| MTVQA | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'contains'} | +| MaRVL | main | rule | exact-match,rule-match | supported | | +| Mantis-Eval | main | rule | exact-match,rule-match | supported | | +| MathVerse | testmini | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| MathVerse | testmini_text_only | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| MathVision | main | rule | exact-match,rule-match | supported | | +| MathVista | main | llm_extract | rule-match,llm-match | supported | | +| MedXpertQA | MM | rule | exact-match,rule-match | supported | | +| Mementos | default | llm_extract | unsupported: keyword-set-f1 | REFUSED | requires keyword extraction + set-F1 grading (Mementos); use the benchmark's own evaluator (score_params: {'compare': 'keyword_set_f1'}) | +| MuirBench | main | rule | exact-match,rule-match | supported | | +| NExT-QA | mc | rule | exact-match,rule-match | supported | | +| NLVR2 | main | rule | exact-match,rule-match | supported | | +| NaturalBench | default | rule | exact-match,rule-match | supported | | +| NoCaps | main | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| OCR-VQA | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| OCRBench | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'contains'} | +| OCRBench-v2 | default | per_task_metric | unsupported: per-task-metric | REFUSED | requires the benchmark's own evaluator (per-row metric dispatch) (score_params: {'metric_field': 'type'}) | +| OK-VQA | main | vqa_accuracy | vqa-accuracy | supported | | +| OlympiadBench | main | rule | exact-match,rule-match | supported | | +| PMC-VQA | main | rule | exact-match,rule-match | supported | | +| POPE | main | rule | exact-match,rule-match | supported | | +| PathVQA | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| PhyX | mc | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| PuzzleVQA | default | rule | exact-match,rule-match | supported | | +| Q-Bench-Plus | default | llm_extract | rule-match,llm-match | supported | | +| R-Bench-V | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'binary', 'official_judge_model': 'gpt-4o'} | +| RealWorldQA | main | llm_extract | rule-match,llm-match | supported | | +| SAT | real | rule | exact-match,rule-match | supported | | +| SAT | synthetic | rule | exact-match,rule-match | supported | | +| SEED-Bench-2-Plus | main | llm_extract | rule-match,llm-match | supported | | +| SEED-Bench-H | main | llm_extract | rule-match,llm-match | supported | | +| SEEDBench | main | llm_extract | rule-match,llm-match | supported | | +| SLAKE | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| ST-VQA | main | anls | anls | supported | score_params: {'threshold': 0.5} | +| SciFIBench | caption2figure | llm_extract | rule-match,llm-match | supported | | +| SciFIBench | figure2caption | llm_extract | rule-match,llm-match | supported | | +| SciVQA | default | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['rouge1', 'rouge_l', 'bertscore']}) | +| ScienceQA | main | rule | exact-match,rule-match | supported | | +| ScienceQA-IMG | default | rule | exact-match,rule-match | supported | | +| ScreenQA | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| SeePhys | default | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| SpatialEval | vqa | rule | exact-match,rule-match | supported | | +| SpatialEval | vtqa | rule | exact-match,rule-match | supported | | +| TableVQA-Bench | fintabnetqa | rule | exact-match,rule-match | supported | | +| TableVQA-Bench | vtabfact | rule | exact-match,rule-match | supported | | +| TableVQA-Bench | vwtq | rule | exact-match,rule-match | supported | | +| TableVQA-Bench | vwtq_syn | rule | exact-match,rule-match | supported | | +| TallyQA | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| TempCompass | caption_matching | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| TempCompass | captioning | llm_judge | llm-judge | supported | score_params: {'rubric': 'binary', 'official_judge_model': 'gpt-3.5-turbo-1106'} | +| TempCompass | multi_choice | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| TempCompass | yes_no | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| TextCaps | main | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l']}) | +| TextVQA | main | vqa_accuracy | vqa-accuracy | supported | | +| TheoremQA | image | rule | exact-match,rule-match | supported | score_params: {'numeric_rel_tol': 0.04} | +| VLMsAreBlind | default | rule | exact-match,rule-match | supported | | +| VQA-RAD | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| VQAv2 | main | vqa_accuracy | vqa-accuracy | supported | | +| VSI-Bench | mc | rule | exact-match,rule-match | supported | | +| VSR | zeroshot | llm_extract | rule-match,llm-match | supported | | +| VStarBench | main | llm_extract | rule-match,llm-match | supported | | +| VibeEval | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [1, 5], 'official_judge_model': 'reka-core-20240501'} | +| VisIT-Bench | default | llm_judge | llm-judge | supported | score_params: {'rubric': 'binary', 'official_judge_model': 'gpt-4', 'judge_inputs': ['instruction_conditioned_caption']} | +| VisOnlyQA | main | llm_extract | rule-match,llm-match | supported | | +| Visual7W | main | llm_extract | rule-match,llm-match | supported | | +| VisualMRC | main | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['rouge_l', 'bleu4', 'meteor', 'cider']}) | +| VisualSimpleQA | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'three_way', 'official_judge_model': 'SimpleQA-style ChatGPT grader (GPT-4o class)'} | +| VizWiz-Captions | main | caption_metrics | unsupported: caption-metrics | REFUSED | requires corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator (score_params: {'metrics': ['cider', 'bleu4', 'meteor', 'rouge_l', 'spice']}) | +| VizWiz-VQA | main | vqa_accuracy | vqa-accuracy | supported | | +| WHOOPS | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| We-Math | main | rule | exact-match,rule-match | supported | | +| WebSRC | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | +| WildVision-Bench | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'scale', 'scale': [-2, 2], 'official_judge_model': 'gpt-4o'} | +| Winoground | main | rule | exact-match,rule-match | supported | | +| WorldMedQA-V | default | llm_extract | rule-match,llm-match | supported | | +| WorldVQA | main | llm_judge | llm-judge | supported | score_params: {'rubric': 'three_way', 'official_judge_model': 'gpt-4o-1120'} | +| ZeroBench | default | rule | exact-match,rule-match | supported | | +| ZoomBench | main | rule_llm_judge | exact-match,rule-match,llm-judge | supported | | +| xGQA | main | rule | exact-match,rule-match | supported | score_params: {'string_match': 'exact'} | + +## Retired score_type distribution (pre-migration, historical) + +- `rule`: 63 +- `llm_extract`: 48 +- `llm_judge`: 22 +- `rule_llm_judge`: 17 +- `caption_metrics`: 7 +- `anls`: 4 +- `vqa_accuracy`: 4 +- `per_task_metric`: 2 +- `none`: 1 +- `execution`: 1 + +## Refused protocols + +- `caption_metrics` (7 subsets) → corpus-level caption metrics (CIDEr/BLEU/ROUGE/BERTScore); use the benchmark's own evaluator. +- `per_task_metric` (2) → per-row metric dispatch; use the benchmark's own evaluator. +- `execution` (1) → official test-case execution (pass@1). +- `llm_extract` + `compare=keyword_set_f1` (1, Mementos) → keyword extraction + set-F1 grading. + +An explicit `--score_pipeline` can force approximate scoring for these; the +output is then stamped `official_protocol: false` with a protocol note. diff --git a/docs/en/SUPPORTED.md b/docs/en/SUPPORTED.md index ddbdedd6..bdf57db0 100644 --- a/docs/en/SUPPORTED.md +++ b/docs/en/SUPPORTED.md @@ -163,7 +163,7 @@ Currently available mm-eval HuggingFace datasets: | Dataset | Splits | Usage | |---------|--------|-------| | [MMBench-en](https://huggingface.co/datasets/mm-eval/MMBench-en) | dev, test | `mmeval_hf@mm-eval/MMBench-en` | -| [MMBench-en-V11](https://huggingface.co/datasets/mm-eval/MMBench-en-V11) | dev, test | `mmeval_hf@mm-eval/MMBench-en-V11` | +| [MMBench-V11](https://huggingface.co/datasets/mm-eval/MMBench-V11) | dev, test | `mmeval_hf@mm-eval/MMBench-V11 --subset en` (subsets: en, cn) | ### 3. VLMEvalKit Datasets (`evalkit@`) diff --git a/docs/en/USAGE.md b/docs/en/USAGE.md index eb1874d0..be0fc429 100644 --- a/docs/en/USAGE.md +++ b/docs/en/USAGE.md @@ -42,7 +42,8 @@ Run evaluation on a HuggingFace-hosted dataset with built-in prompt templates: ```bash python mmeval/run.py \ --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ --split test \ --out_dir work_dirs/mmbench_test \ --gpu_per_parallel 1 \ @@ -98,7 +99,8 @@ Distribute inference across multiple GPUs with automatic data sharding: ```bash python mmeval/run.py \ --model_name_or_path Qwen/Qwen2.5-VL-72B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ --split test \ --out_dir work_dirs/mmbench_72b \ --gpu_per_parallel 4 \ @@ -114,7 +116,8 @@ Run a quick deterministic smoke test on 20 randomly selected samples: ```bash python mmeval/run.py \ --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ --split test \ --out_dir work_dirs/mmbench_smoke \ --gpu_per_parallel 1 \ diff --git a/docs/en/VALIDATION.md b/docs/en/VALIDATION.md new file mode 100644 index 00000000..dbcf8725 --- /dev/null +++ b/docs/en/VALIDATION.md @@ -0,0 +1,54 @@ +# Validation: reproduced official results + +Has this framework been validated against official evaluation results? Yes. +Each check below runs a real model end to end — the framework's own inference +and scoring — using the dataset's **official scoring pipeline unchanged**, and +compares the result against an official reference: a model card, a benchmark's +technical report, the [OpenVLM leaderboard](https://huggingface.co/spaces/opencompass/open_vlm_leaderboard), +or the upstream [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) +evaluator run on the same machine. + +## How to read the table + +- **`eval_score`** is this framework's result; **`reference_score`** is the + official number; **`delta` = `eval_score` − `reference_score`** (percentage + points). +- Scores are each benchmark's official headline metric, as a percentage: + accuracy for the rule-based pipelines (ChartQA, OCRBench) and mean + ANLS / VQA-accuracy for the fractional metrics (DocVQA, InfographicVQA, + TextVQA). +- **`score_pipeline`** is the ordered list of scoring stages that ran (see + [SCORING.md](SCORING.md)); it is the dataset's declared official protocol. +- **`custom_template`** = `yes` where the model was prompted with the official + benchmark's own prompt template rather than the dataset's shipped default + (see notes below). + +**Agreement.** All ten checks agree with their official reference to within +**±1.0 percentage point** on the inference side, and the scoring side is +**exact**: on identical model predictions the framework's per-sample verdicts +match the official comparators with **zero mismatches across 33,300 samples** +(ANLS and VQA-accuracy agree to 1e-9). The small delta that remains is +inference-side variance — prompt construction, image handling, generation +settings — not scoring. + +## Reproductions + +| model | model_revision | dataset | split | score_pipeline | custom_template | samples | eval_score | reference_score | delta | reference_source | notes | +|---|---|---|---|---|---|---|---|---|---|---|---| +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | ChartQA | test | exact-match, rule-match (numeric_rel_tol 0.05) | no | 2500 | 74.24 | 73.50 | +0.74 | Qwen2-VL-2B-Instruct model card (ChartQA_test) | | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | OCRBench | test | exact-match, rule-match (string_match contains) | no | 1000 | 79.10 | 79.70 | -0.60 | OpenVLM leaderboard (797/1000); model card 79.4 | | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | DocVQA | validation | anls (threshold 0.5) | no | 5349 | 88.48 | 89.24 | -0.76 | Upstream VLMEvalKit run, same machine | | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | InfographicVQA | validation | anls (threshold 0.5) | yes | 2801 | 64.31 | 64.20 | +0.11 | Upstream VLMEvalKit run, same machine | matched official inference config | +| Qwen2-VL-2B-Instruct | 895c3a49bc3f | TextVQA | validation | vqa-accuracy | no | 5000 | 79.16 | 79.70 | -0.54 | Qwen2-VL-2B-Instruct model card (TextVQA_val) | | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | ChartQA | test | exact-match, rule-match (numeric_rel_tol 0.05) | no | 2500 | 83.64 | 84.00 | -0.36 | Qwen2.5-VL Technical Report Table 5 (arXiv:2502.13923) | | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | OCRBench | test | exact-match, rule-match (string_match contains) | no | 1000 | 82.70 | 82.80 | -0.10 | OpenVLM leaderboard (828/1000); same-machine upstream 82.6 | matched official inference config | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | DocVQA | validation | anls (threshold 0.5) | no | 5349 | 92.46 | 93.01 | -0.55 | Upstream VLMEvalKit run, same machine | | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | InfographicVQA | validation | anls (threshold 0.5) | no | 2801 | 75.15 | 76.03 | -0.88 | Upstream VLMEvalKit run, same machine | matched official inference config | +| Qwen2.5-VL-3B-Instruct | 66285546d2b8 | TextVQA | validation | vqa-accuracy | no | 5000 | 79.82 | 79.30 | +0.52 | Qwen2.5-VL Technical Report Table 5 + model card | | + +**Matched official inference config.** Three checks needed the official +inference configuration — image pixel window, prompt construction, and +generation length — reproduced before the model's outputs agreed with the +reference to within tolerance. Where the official prompt template itself was +reproduced, `custom_template` is `yes` (here, 2B InfographicVQA). This is +inference-side alignment only; no scoring behaviour changes. diff --git a/mmeval/data/base.py b/mmeval/data/base.py index faf627aa..2a4bbd85 100644 --- a/mmeval/data/base.py +++ b/mmeval/data/base.py @@ -9,7 +9,6 @@ from jinja2 import Environment - class BaseDataset(ABC): """Dataset base class for loading and processing datasets. @@ -209,7 +208,7 @@ def build_prompt(self, prompt_template: str, sample: Dict[str, Any]) -> str: Rendered template string """ env = Environment() - env.globals.update({'zip': zip, 'enumerate': enumerate, 'len': len, 'range': range, 'list': list, + env.globals.update({'zip': zip, 'enumerate': enumerate, 'len': len, 'range': range, 'list': list, 'dict': dict, 'str': str, 'int': int, 'float': float, 'bool': bool, 'sum': sum, 'max': max, 'min': min}) template = env.from_string(prompt_template) return template.render(**sample) diff --git a/mmeval/data/mmeval_hf.py b/mmeval/data/mmeval_hf.py index cfdd74b9..40761f1b 100644 --- a/mmeval/data/mmeval_hf.py +++ b/mmeval/data/mmeval_hf.py @@ -7,7 +7,15 @@ class MMEvalHFDataset(BaseDataset): - """Dataset loader for HuggingFace datasets in mm-eval format.""" + """Dataset loader for HuggingFace datasets in mm-eval format. + + Reads the subset manifest from the dataset repo's metadata.json and injects + the scoring-relevant metadata as the flat top-level `dataset_meta` sample + field. Rows are expected in the flat layout (grading fields as top-level + columns; messages[0] carries render inputs only) — non-conforming rows are + passed through unchanged and surface at scoring time via the scorer's + explicit layout rejection. + """ def __init__(self, args): self.dataset_name = args.dataset.split("@", 1)[1] @@ -40,9 +48,43 @@ def _load_raw_data(self, args) -> tuple: raise ValueError( f"{self.dataset_name}: subset {self.subset!r} not in {list(subsets)}" ) - dataset_template = subsets[self.subset].get("prompt_template") + subset_block = subsets[self.subset] + dataset_template = subset_block.get("prompt_template") + + # Scoring data-flow contract: inject the subset's scoring-relevant + # metadata into every sample as the flat top-level `dataset_meta` + # field, so it lands in result.json and the scorer can resolve the + # official grading protocol without access to the HF manifest. CLI + # flags on the scorer always override these values. + if "score_type" in subset_block: + raise ValueError( + f"{self.dataset_name} subset {self.subset!r}: metadata.json " + f"declares the retired composite `score_type` vocabulary " + f"(retired org-wide 2026-07-19) — re-push the dataset with the " + f"`score_pipeline` schema (docs/en/SCORING.md, 'Dataset " + f"metadata contract')." + ) + meta_block = {} + for key in ("task_type", "score_params"): + value = subset_block.get(key) + if value: + meta_block[key] = value + if "score_pipeline" in subset_block: + # Declared protocol passes through verbatim; [] (explicitly no + # official protocol) is a meaningful value, so test presence. + meta_block["score_pipeline"] = subset_block["score_pipeline"] + note = ((subset_block.get("score_protocol") or {}).get("note") or "").strip() + if note: + meta_block["score_note"] = note + self._dataset_meta = meta_block or None - ds = load_dataset(self.dataset_name, name="default", split=self.split) + # Prefer multi-config layout: HF config name == mm-eval subset name. + # Fall back to the single-`default` config layout, where one config + # holds all subsets' splits (named `_`). + try: + ds = load_dataset(self.dataset_name, name=self.subset, split=self.split) + except (ValueError, FileNotFoundError): + ds = load_dataset(self.dataset_name, name="default", split=self.split) return ds, dataset_template def convert_circular(self, **kwargs) -> any: @@ -52,14 +94,12 @@ def convert_circular(self, **kwargs) -> any: def _process_sample(self, idx: int): sample = dict(self._raw_dataset[idx]) - # Add eval-id if not present if "eval-id" not in sample: sample["eval-id"] = idx messages = sample["messages"] message_list = json.loads(messages) if isinstance(messages, str) else messages - # Get media from sample level (HF datasets may store as single image or list) media = sample.get("media") if media is None: media_list = [] @@ -68,10 +108,10 @@ def _process_sample(self, idx: int): else: media_list = [media] - # Ensure sample["media"] is always a list sample["media"] = media_list - - # Process all messages with sample-level media indexed by placeholder order sample["messages"] = self._process_messages(message_list, media_list) + if self._dataset_meta: + sample.setdefault("dataset_meta", dict(self._dataset_meta)) + return sample diff --git a/mmeval/data/tsv.py b/mmeval/data/tsv.py index ec88c41c..933bb9a2 100644 --- a/mmeval/data/tsv.py +++ b/mmeval/data/tsv.py @@ -104,7 +104,6 @@ def _extract_media_paths(self, sample: Dict[str, Any]) -> list: """ media_paths = [] - # Priority 1: Check for image_url if 'image_url' in sample and pd.notna(sample['image_url']): image_url = sample['image_url'] if image_url.startswith('[') and image_url.endswith(']'): @@ -112,7 +111,6 @@ def _extract_media_paths(self, sample: Dict[str, Any]) -> list: else: media_paths = [image_url] - # Priority 2: Check for base64 image data elif 'image' in sample and pd.notna(sample['image']): image = sample['image'] if image.startswith('[') and image.endswith(']'): @@ -145,7 +143,6 @@ def _process_sample(self, index: int) -> Dict[str, Any]: # If placeholder/media counts mismatch, all placeholders are moved to prefix. question = normalize_question_with_media(question, len(media_list)) - # Build options dict and choices list options = { choice_index: sample[choice_index] for choice_index in string.ascii_uppercase if choice_index in sample and not pd.isna(sample[choice_index]) @@ -156,7 +153,7 @@ def _process_sample(self, index: int) -> Dict[str, Any]: if options: message["options"] = options message["choices"] = list(options.keys()) - + hint = sample.get("hint", None) if hint and pd.notna(hint): message["hint"] = hint diff --git a/mmeval/infer/qwen3_vl.py b/mmeval/infer/qwen3_vl.py index 0d3ddecb..d73e40c9 100644 --- a/mmeval/infer/qwen3_vl.py +++ b/mmeval/infer/qwen3_vl.py @@ -1,3 +1,4 @@ +import os import re import copy @@ -14,8 +15,16 @@ class TaskRunner(Task): def __init__(self, args): self.args = args self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu") - self.dtype = getattr(args, "dtype") or "auto" - self.default_model_kwargs = {"device_map": "auto"} + # Qwen3-VL is a bf16-native model; "auto" mis-resolves to fp32 on this merged ckpt, + # which both doubles attention memory AND breaks flash_attention_2 (needs fp16/bf16). + # Default to bf16 unless the user explicitly passes --dtype. + _dt = getattr(args, "dtype", None) + self.dtype = _dt if _dt and _dt != "auto" else torch.bfloat16 + # flash_attention_2 is EXACT attention (same math as sdpa, just memory-efficient): + # avoids materializing the O(N^2) attention matrix that OOMs on no-cap full-res images. + # Overridable via --attn_implementation; default on since flash-attn is installed. + _attn = getattr(args, "attn_implementation", None) or os.environ.get("MMEVAL_ATTN", "flash_attention_2") + self.default_model_kwargs = {"device_map": "auto", "attn_implementation": _attn} self.default_gen_kwargs = {"max_new_tokens": 128} self.model_kwargs = parse_model_kwargs(args, self.default_model_kwargs) self.gen_kwargs = parse_gen_kwargs(args, self.default_gen_kwargs) @@ -53,10 +62,13 @@ def parse_input(self, message): { "type": "image", "image": media, - "min_pixels": 4 * 32 * 32, - "max_pixels": 256 * 32 * 32, + # Aligned with VLMEvalKit: no hardcoded min/max_pixels, so the + # processor's defaults apply (shortest_edge=65536, + # longest_edge=16777216). The previous cap of max_pixels=256*32*32 + # (262144 ≈ 512x512) downscaled ~17% of MMStar images and hurt + # fine-perception accuracy. } - ) + ) elif chunk == constants.video: media = media_list[media_idx] media_idx += 1 @@ -94,9 +106,9 @@ def _generate_response(self, inputs): def run_sample(self, sample: dict): ori_sample = copy.deepcopy(sample) message = sample["messages"][0] - + user_message = self.parse_input(message) - + text = self.processor.apply_chat_template( user_message, tokenize=False, add_generation_prompt=True ) @@ -112,12 +124,12 @@ def run_sample(self, sample: dict): video_metadatas = None inputs = self.processor( - text=text, - images=images, - videos=videos, + text=text, + images=images, + videos=videos, video_metadata=video_metadatas, - return_tensors="pt", - do_resize=False, + return_tensors="pt", + do_resize=False, **video_kwargs ) inputs = inputs.to(self.model.device) diff --git a/mmeval/infer/qwenvl2.py b/mmeval/infer/qwenvl2.py index 824b9fe0..e8a0ed84 100644 --- a/mmeval/infer/qwenvl2.py +++ b/mmeval/infer/qwenvl2.py @@ -27,9 +27,12 @@ def __init__(self, args): def load_model(self, args): self.model = Qwen2VLForConditionalGeneration.from_pretrained(args.model_name_or_path, torch_dtype=self.dtype, **self.model_kwargs) self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path) - min_pixels = 256 * 28 * 28 - max_pixels = 1280 * 28 * 28 - self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, min_pixels=min_pixels, max_pixels=max_pixels) + processor_kwargs = {"min_pixels": 256 * 28 * 28, "max_pixels": 1280 * 28 * 28} + if getattr(args, "min_pixels", None) is not None: + processor_kwargs["min_pixels"] = args.min_pixels + if getattr(args, "max_pixels", None) is not None: + processor_kwargs["max_pixels"] = args.max_pixels + self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, **processor_kwargs) def parse_input(self, message:dict): question = message["prompt"] diff --git a/mmeval/infer/qwenvl2d5.py b/mmeval/infer/qwenvl2d5.py index 0c6ee5ff..b337ea35 100644 --- a/mmeval/infer/qwenvl2d5.py +++ b/mmeval/infer/qwenvl2d5.py @@ -32,9 +32,16 @@ def __init__(self, args): def load_model(self, args): self.model = Qwen2_5_VLForConditionalGeneration.from_pretrained(args.model_name_or_path, torch_dtype=self.dtype, **self.model_kwargs) self.tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path) - min_pixels = 256 * 28 * 28 - max_pixels = 1280 * 28 * 28 - self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, min_pixels=min_pixels, max_pixels=max_pixels) + # Aligned with VLMEvalKit: do NOT pass min/max_pixels by default, so the + # processor's defaults apply (min_pixels=3136, max_pixels=12845056 ≈ 12.8M). + # The previous cap of max_pixels=1280*28*28 (~1MP) was ~12.8x lower and + # downscaled images. --min_pixels/--max_pixels override per run. + processor_kwargs = {} + if getattr(args, "min_pixels", None) is not None: + processor_kwargs["min_pixels"] = args.min_pixels + if getattr(args, "max_pixels", None) is not None: + processor_kwargs["max_pixels"] = args.max_pixels + self.processor = AutoProcessor.from_pretrained(args.model_name_or_path, **processor_kwargs) def parse_input(self, message): question = message["prompt"] @@ -145,4 +152,4 @@ def run_sample(self, sample: dict): if __name__ == "__main__": args = parse_args() model_evaluator = TaskRunner(args) - model_evaluator.inference_dataset() + model_evaluator.inference_dataset() \ No newline at end of file diff --git a/mmeval/run.py b/mmeval/run.py index 5eb809b8..0c211219 100644 --- a/mmeval/run.py +++ b/mmeval/run.py @@ -19,7 +19,7 @@ def get_series(model_name: str): args = parse_args() model_name_or_path = args.model_name_or_path - series = get_series(model_name_or_path.split("/")[-1]) + series = args.model_series or get_series(model_name_or_path.split("/")[-1]) infer_file = series_infer_env_mapping[series]["infer_file"] infer_env = series_infer_env_mapping[series]["env"] diff --git a/mmeval/score.py b/mmeval/score.py new file mode 100644 index 00000000..f2e8700d --- /dev/null +++ b/mmeval/score.py @@ -0,0 +1,31 @@ +from mmeval.scoring.pipeline import discover_result_files, score_result_file +from mmeval.utils.argparser import parse_args + + +def run_score(args): + files = discover_result_files(out_dir=args.score_out_dir, pattern=args.score_result_glob) + if not files: + raise RuntimeError(f"No result files found under {args.score_out_dir} with pattern {args.score_result_glob}") + + done = [] + failed = [] + for result_file in files: + try: + done.append(score_result_file(result_file, args)) + except Exception as exc: + failed.append({"result_file": result_file, "error": str(exc)}) + + for item in done: + print( + f"✅ [score] {item['result_file']} -> {item['score_file']} " + f"(acc={item['summary']['accuracy']:.4f}, resumed={item.get('resumed_count', 0)})" + ) + if failed: + for item in failed: + print(f"❌ [score] {item['result_file']}: {item['error']}") + raise SystemExit(1) + + +if __name__ == "__main__": + args = parse_args() + run_score(args) diff --git a/mmeval/scoring/__init__.py b/mmeval/scoring/__init__.py new file mode 100644 index 00000000..0a22a81b --- /dev/null +++ b/mmeval/scoring/__init__.py @@ -0,0 +1,6 @@ +from mmeval.scoring.pipeline import score_result_file, discover_result_files + +__all__ = [ + "score_result_file", + "discover_result_files", +] diff --git a/mmeval/scoring/extraction.py b/mmeval/scoring/extraction.py new file mode 100644 index 00000000..bcc6b8db --- /dev/null +++ b/mmeval/scoring/extraction.py @@ -0,0 +1,607 @@ +import re +import unicodedata +from typing import Any, Dict, List, Optional, Tuple + + +OPTION_LETTERS = list("ABCDEFGHIJKLMNOPQRSTUVWXYZ") + +# Question-type taxonomy. Every type is justified by at least one mm-eval org +# dataset (per-subset audit: docs/en/SCORING_COVERAGE.md): mcq (ScienceQA-IMG, MMStar, ...; +# multi-letter answers like LogicVista's "A, C" are mcq with |letters|>1), +# yes_no (VSR, GQA-Spatial verify rows), numeric (CharXiv, VLMsAreBlind, SeePhys), +# open (MM-Vet-v2, CharXiv descriptive). +QUESTION_TYPES = ("mcq", "yes_no", "numeric", "open") + + + +def _to_text(value: Any) -> str: + if isinstance(value, list): + if not value: + return "" + value = value[0] + if value is None: + return "" + return str(value) + + +def _extract_boxed_content(text: str) -> str: + marker = "\\boxed{" + start = text.find(marker) + if start < 0: + return text + i = start + len(marker) + depth = 1 + content = [] + while i < len(text): + ch = text[i] + if ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + return "".join(content).strip() + content.append(ch) + i += 1 + return text + + +# normalize_text machinery, compiled once (hot path: called several times per sample). +_COT_TAG_RE = re.compile(r"", re.IGNORECASE) +_LATEX_CMD_RE = re.compile(r"\\(?:left|right|displaystyle|mathrm|textbf|textit|text)\b") +_LATEX_SPACING_RE = re.compile(r"\\(?:,|;|!|\:)") +_WHITESPACE_RE = re.compile(r"\s+") +# Inference artifacts and unicode operator variants folded via one translate/replace pass. +_ARTIFACT_REPLACEMENTS = ( + ("```", " "), ("<|end_of_sentence|>", ""), ("<|end▁of▁sentence|>", ""), + ("", ""), ("", ""), ("", ""), ("Falcon: ", ""), +) +_OPERATOR_TRANSLATION = str.maketrans({"−": "-", "–": "-", "—": "-", "×": "*", "·": "*", "÷": "/"}) + + +def normalize_text( + value: Any, + lower: bool = False, + extract_boxed: bool = True, + strip_latex_commands: bool = True, +) -> str: + """Syntax-level, semantics-preserving normalization of a model answer: + NFKC folding, \boxed{} content extraction, / tag removal, + inference-artifact stripping, light LaTeX cleanup, unicode operator folding, + whitespace collapse (and optional lowercasing).""" + text = _to_text(value) + + text = unicodedata.normalize("NFKC", text) + if extract_boxed: + text = _extract_boxed_content(text) + + text = _COT_TAG_RE.sub(" ", text) + for old_str, new_str in _ARTIFACT_REPLACEMENTS: + text = text.replace(old_str, new_str) + + if strip_latex_commands: + text = _LATEX_CMD_RE.sub(" ", text) + text = _LATEX_SPACING_RE.sub(" ", text) + text = text.replace("\\%", "%") + + text = text.translate(_OPERATOR_TRANSLATION) + + text = _WHITESPACE_RE.sub(" ", text.strip()) + if lower: + text = text.lower() + return text + + +def normalize_for_exact(value: Any) -> str: + return normalize_text(value, lower=True).strip() + + +_WRAPPING_QUOTES_RE = re.compile(r"^[\"'`]+|[\"'`]+$") +_TRAILING_SENTENCE_PUNCT_RE = re.compile(r"[\.。!！?？]+$") + + +def normalize_open_answer(value: Any) -> str: + """normalize_for_exact plus open-answer specifics: strip wrapping quotes and + trailing sentence punctuation ("... Bridge." vs "bridge").""" + text = normalize_for_exact(value) + text = _WRAPPING_QUOTES_RE.sub("", text) + text = _TRAILING_SENTENCE_PUNCT_RE.sub("", text) + return _WHITESPACE_RE.sub(" ", text).strip() + + +# Render inputs — the fields a dataset's prompt template can reference; they are +# the ONLY fields that live inside messages[0] (dataset-standard boundary rule). +_RENDER_INPUT_FIELDS = frozenset({"question", "options", "choices", "hint", "prompt"}) + + +def get_field(sample: Dict[str, Any], field: str, default: Any = None) -> Any: + """Read a field from the ONE canonical result.json layout: the prediction + at messages[-1].response; render inputs (question/options/choices/hint/ + prompt) inside messages[0]; every grading field (answer, question_type, + reference_response, and any --score_gt_field) at the sample TOP level. + All backends emit this layout; files with grading fields embedded in + messages[0] are rejected by the scorer.""" + if not field: + return default + if field == "messages[-1].response": + messages = sample.get("messages") or [] + last = messages[-1] if messages else None + return last.get("response", default) if isinstance(last, dict) else default + if field in _RENDER_INPUT_FIELDS: + messages = sample.get("messages") or [] + first = messages[0] if messages else None + return first.get(field, default) if isinstance(first, dict) else default + return sample.get(field, default) + + +def get_question(sample: Dict[str, Any]) -> str: + q = get_field(sample, "question") + return "" if q is None else str(q) + + +_PAREN_OPTION_RE = re.compile(r"\(([A-H])\)") +_LABELLED_OPTION_RE = re.compile(r"(?:^|[\n\s])([A-H])[\.\):]") + + +def _option_letters_from_question(question: str) -> List[str]: + """Parse option labels embedded in the question text. Handles both the + "A: ... B: ..." (MMStar) and "(A) ... (B) ..." (BLINK) inline formats. Returns + only the leading contiguous run from A (A,B,C,D,...) to avoid stray letters.""" + if not isinstance(question, str): + return [] + found = set() + # "(A)" / "(A) " style and "A." / "A:" / "A)" at a token boundary. + for m in _PAREN_OPTION_RE.findall(question): + found.add(m.upper()) + for m in _LABELLED_OPTION_RE.findall(question): + found.add(m.upper()) + letters = [] + for ch in OPTION_LETTERS[:8]: + if ch in found: + letters.append(ch) + else: + break # keep only the contiguous A,B,C,... prefix + return letters + + +def _choices_to_letters(choices: List[Any]) -> List[str]: + """A `choices` list is either option letters (['A','B',...]) or option values + (['0','3',...]). Letters are used directly; values map positionally to A,B,C,...""" + items = [str(c).strip() for c in choices] + if items and all(len(c) == 1 and c.upper() in OPTION_LETTERS for c in items): + return [c.upper() for c in items] + return OPTION_LETTERS[: len(items)] + + +_LETTER_PREFIX_RE = re.compile(r"\s*([A-Z])\s*[\.\):]\s*(.+)$") + + +def _strip_letter_prefix(text: str, letter: str) -> str: + """VSI-Bench stores choices as letter-prefixed strings ('A. back-right'); + strip the positional letter prefix so option-text matching sees the bare text.""" + m = _LETTER_PREFIX_RE.match(text) + return m.group(2).strip() if m and m.group(1) == letter else text + + +def _flat_letter_options(src: Dict[str, Any]) -> Dict[str, str]: + """Options stored as per-letter keys with non-empty values (GMAI-MMBench, + WorldMedQA-V, HRBench store msg['A']='...' with empty options/choices; an + empty value, e.g. GMAI's E='', means the option does not exist).""" + out = {} + for letter in OPTION_LETTERS[:8]: + value = src.get(letter) + if value is None or str(value).strip() == "": + break # contiguous prefix only, like the inline-question parser + out[letter] = str(value) + return out + + +def parsed_options(sample: Dict[str, Any]) -> Optional[List[str]]: + """Candidate option letters from an explicit option source, or None when the + sample carries none (options-in-image datasets). All option sources — the + `options` dict, the `choices` list, inline-question text and flat per-letter + keys — are render inputs and therefore live inside messages[0] + (dataset-standard boundary rule); only grading fields moved to the sample + top level.""" + messages = sample.get("messages") or [] + msg = messages[0] if messages and isinstance(messages[0], dict) else None + if not isinstance(msg, dict): + return None + + options = msg.get("options") + if isinstance(options, dict) and options: + return [str(k).strip().upper() for k in options.keys()] + choices = msg.get("choices") + if isinstance(choices, list) and choices: + return _choices_to_letters(choices) + + # Flat per-letter keys inside the user message (GMAI-MMBench msg["A"].."E", + # WorldMedQA-V, HRBench) — empty values excluded. + flat = _flat_letter_options(msg) + if flat: + return list(flat.keys()) + + # Inline-options datasets (MMStar/BLINK keep options in the question text and + # ship empty `options`/`choices`): derive the real letters from the question so + # we don't fall back to A-F and mis-count stray letters (e.g. "F = ma"). + letters = _option_letters_from_question(msg.get("question") or msg.get("prompt") or "") + if letters: + return letters + return None + + +def extract_options(sample: Dict[str, Any]) -> List[str]: + """Candidate option letters: the sample's parsed option source, else A-F + (some MCQ datasets, e.g. LogicVista, keep the choices in the image and don't + list them in the question text). Whether the item is actually MCQ vs open is + decided by infer_question_type from the ground-truth shape, not by this + default.""" + return parsed_options(sample) or OPTION_LETTERS[:6] + + +def extract_option_texts(sample: Dict[str, Any], letters: List[str]) -> Dict[str, str]: + """Map option letter -> option text, from whichever source the sample uses + (all render inputs inside messages[0] — see extract_options).""" + messages = sample.get("messages") or [] + msg = messages[0] if messages and isinstance(messages[0], dict) else None + if isinstance(msg, dict): + msg_options = msg.get("options") + if isinstance(msg_options, dict) and msg_options: + return {str(k).strip().upper(): str(v) for k, v in msg_options.items()} + # BLINK-style: `choices` holds the option *values* positionally (A,B,C,...). + msg_choices = msg.get("choices") + if isinstance(msg_choices, list) and msg_choices: + vals = [str(c) for c in msg_choices] + if not all(len(v.strip()) == 1 and v.strip().upper() in OPTION_LETTERS for v in vals): + # VSI-Bench stores letter-prefixed values ("A. back-right"); strip + # the positional letter so text matching sees the bare option. + return { + OPTION_LETTERS[i]: _strip_letter_prefix(vals[i], OPTION_LETTERS[i]) + for i in range(min(len(vals), len(OPTION_LETTERS))) + } + + # Flat per-letter option keys inside the user message (HRBench msg["A"]="27B", + # GMAI-MMBench, WorldMedQA-V) with empty options/choices. + option_texts = {} + for letter in letters: + if isinstance(msg, dict) and letter in msg: + option_texts[letter] = str(msg[letter]) + return option_texts + + +_GT_LABEL_STRIP_RE = re.compile(r"[()\[\].:;\s\*]") +_LETTER_SEPARATOR_RE = re.compile(r"[,;/&\s]+|\band\b", re.IGNORECASE) + + +def parse_gt_letters(gt: Any) -> Optional[Tuple[str, ...]]: + """MCQ ground truth -> sorted letter tuple. A single bare letter ("B", "(D)") + gives a 1-tuple; multi-letter answers must be SEPARATED letters like "A, C" / + "A and C" (LogicVista format), >=2 unique letters within A-H. Compact "AC" is + deliberately NOT accepted — as a *type signal* it would misread open answers + that happen to be A-H-only words ("BED").""" + g = _GT_LABEL_STRIP_RE.sub("", str(gt)).upper() + if len(g) == 1 and g in OPTION_LETTERS: + return (g,) + tokens = [t for t in _LETTER_SEPARATOR_RE.split(str(gt).strip()) if t] + if len(tokens) < 2: + return None + letters = [] + for t in tokens: + if len(t) == 1 and t.upper() in OPTION_LETTERS[:8]: + letters.append(t.upper()) + else: + return None + if len(set(letters)) != len(letters): + return None + return tuple(sorted(letters)) + + +def parse_number(value: Any) -> Optional[float]: + """The whole (normalized) answer is a number, else None. A trailing + percent sign reads as /100 — the official numeric semantics (ChartQA + relaxed_correctness `_to_float`, lmms-eval chartqa/utils.py, taken from + Qwen-VL evaluate_vqa.py:L113): "24%" is 0.24, NOT 24, so a model + answering "24%" against gt "24" does not match.""" + text = normalize_open_answer(value).replace(",", "") + percent = text.endswith("%") + if percent: + text = text.rstrip("%").strip() + try: + number = float(text) + except ValueError: + return None + return number / 100.0 if percent else number + + +_WORD_RE = re.compile(r"[a-z]+") + + +def extract_yes_no(value: Any) -> Optional[str]: + """VLMEvalKit YOrN_Extraction parity (vlmeval/dataset/utils/yorn.py:254-261): + word-level; "yes" without "no" -> yes, "no" without "yes" -> no, both or + neither -> None (upstream 'Unknown').""" + words = set(_WORD_RE.findall(normalize_text(value, lower=True))) + if "yes" in words and "no" not in words: + return "yes" + if "no" in words and "yes" not in words: + return "no" + return None + + +def infer_question_type(sample: Dict[str, Any], question_type: str = "auto", gt: Any = None) -> str: + """Resolve the sample's question type. Order: forced CLI value -> the sample's + standardized `question_type` field (recognized vocabulary only — mm-eval org + datasets like VSI-Bench reuse that field for subtask names, which fall + through) -> gt shape -> options presence.""" + forced = (question_type or "auto").strip().lower() + if forced in QUESTION_TYPES: + return forced + + # Per-sample field contract: exactly the four grading values. A 2026-07-19 + # sweep of every mm-eval dataset/split (577k+ rows, datasets-server + # statistics API) found ONLY these; dataset-level task_type stays in + # metadata, source-benchmark labels stay in `source_question_type`. + field_value = str(get_field(sample, "question_type", "") or "").strip().lower() + if field_value in QUESTION_TYPES: + return field_value + + # The ground-truth shape is the most reliable signal and correctly splits + # mixed benchmarks (RealWorldQA interleaves letter, yes/no and numeric gts; + # GQA-Spatial mixes verify and query rows; LogicVista mixes single- and + # multi-letter answers — both are mcq). + if gt is not None: + norm = normalize_open_answer(gt) + if norm in {"yes", "no"}: + return "yes_no" + if parse_gt_letters(gt) is not None: + return "mcq" + if parse_number(gt) is not None: + return "numeric" + if _GT_LABEL_STRIP_RE.sub("", str(gt)) != "": + return "open" + + # No usable gt signal (missing/blank gt — the sample is marked invalid before + # any stage runs): default to mcq. NOTE extract_options never returns empty + # (A-F fallback), so an "options present?" check here would be dead logic. + return "mcq" + + +_STRICT_CUE_RES = ( + re.compile(r"^(?:answer|option|choice)\s*[:\-]?\s*(?-i:([A-Z]))(?:[\s\.\),:;]|$)", re.IGNORECASE), + re.compile(r"(?:final answer|the answer is|answer is|choice is|option is)\s*[:\-]?\s*(?-i:([A-Z]))(?:[\s\.\),:;]|$)", re.IGNORECASE), +) + + +def extract_option_strict(text: Any, candidates: List[str]) -> Optional[str]: + # Prefer the content of the LAST ... tag if the model emitted + # the reasoning-style "...X" format. Otherwise + # the buried letter gets diluted by surrounding reasoning text and the regexes + # below — which anchor on the start of the string — miss it. + raw_text = _to_text(text) + ans = _ANSWER_TAG_RE.findall(raw_text) + clean = normalize_text(ans[-1]) if ans else normalize_text(text) + if not clean: + return None + + stripped = clean.strip() + uppercase = stripped.upper() + # The whole answer is a single option letter (any case) — unambiguous. + if uppercase in candidates: + return uppercase + # Labelled answer-first form "B. ..." / "B) ..." / "B: ...". Punctuation (not a + # space) and an uppercase letter are required, so "A teddy bear" is NOT option A. + if len(stripped) >= 2 and uppercase[0] in candidates and stripped[0].isupper() and stripped[1] in {".", ")", ":"}: + return uppercase[0] + + # Cue phrases are case-insensitive but the letter is matched case-SENSITIVELY + # ((?-i:...)), so "the answer is a teddy bear" never reads as option A. + for pattern in _STRICT_CUE_RES: + match = pattern.search(stripped) + if match: + candidate = match.group(1) + if candidate in candidates: + return candidate + return None + + +# Punctuation/markdown replaced by whitespace so option letters become standalone +# tokens — VLMEvalKit's character set plus full-width CJK punctuation (without it +# "最终答案：A" never splits and the trailing letter is lost). +_CHOICE_PUNCT_TRANSLATION = str.maketrans({ + ch: " " for ch in ".()[],:;!*#{}" + "：；，。、！？「」『』（）【】《》" +}) + +# Explicit "final answer" style cues (English + CJK markers 答案/最终答案/选). +# Cues resolve answers even when several option letters appear in the reasoning +# (where pure token-position logic gives up and upstream defers to its GPT +# extractor). Cue phrases are case-insensitive but the option letter is matched +# case-SENSITIVELY via (?-i:...) so a lowercase article ("the answer is a teddy +# bear") is never mistaken for option "A". +_ANSWER_CUE = re.compile( + r"(?:final answer|the answer is|answer is|answer\s*:|option is|" + r"(?:the\s+)?correct\s+(?:option|answer|choice)(?:\s+is)?|" # "correct option: (C)" + r"最终答案|答案|选)\s*[:\-：]?\s*" + # The letter may be wrapped by any mix of '(' / '*' / full-width parens in any + # order: "A", "(A)", "**A**", "**(A)**", "(A) text", "**(A) 0.0"... + r"[\(（\*]{0,4}(?-i:([A-Z]))[\)）\*]{0,4}(?:[\s\.\),:;、，。]|$)", + re.IGNORECASE, +) + + +def _strip_choice_punct(text: str) -> str: + return text.translate(_CHOICE_PUNCT_TRANSLATION) + + +def can_infer_option(answer: str, choices: List[str]) -> Optional[str]: + """VLMEvalKit-style: strip punctuation -> tokens -> the unique option letter + appearing near the end of the answer. Case-sensitive like upstream: pass the + ORIGINAL-case answer, or lowercase articles ("a cute cat") count as option A.""" + answer_mod = _strip_choice_punct(answer) + splits = [x.strip() for x in answer_mod.split() if x.strip()] + present = [c for c in choices if c in splits] + if len(present) == 1: + ch = present[0] + # Near the end (the model's final pick), mirroring VLMEvalKit's last-5 + # window; additionally a leading letter in a short (<=5 token) answer + # such as "B is the correct answer" is accepted — upstream defers those + # to its GPT extractor, which resolves them the same way. + if splits.index(ch) > len(splits) - 5 or (splits.count(ch) == 1 and len(splits) <= 5): + return ch + return None + + +def can_infer_text(answer: str, option_texts: Dict[str, str]) -> Optional[str]: + """VLMEvalKit-style: the answer restates exactly one option's text. Upstream's + length guard applies: a long CoT that merely mentions one option somewhere is + not an answer restatement.""" + if not option_texts: + return None + low = answer.lower() + if len(low) > 2 * sum(len(str(v)) for v in option_texts.values()): + return None + cands = [k for k, v in option_texts.items() if v and str(v).strip().lower() in low] + if len(cands) == 1: + return cands[0] + return None + + +# extract_option_robust machinery, compiled once. Rule order matters and is +# documented on each step below. +_ANSWER_TAG_RE = re.compile(r"\s*(.*?)\s*", re.DOTALL | re.IGNORECASE) +_WRAPPING_BRACKETS_RE = re.compile(r"^[\(\[]|[\)\]\.]$") +_BOXED_LETTER_RE = re.compile(r"\\boxed\{\s*(?:\\[a-zA-Z]+\s*\{\s*)?[\(\*]*\s*([A-Za-z])") +_CUE_TAIL_RE = re.compile( + r"(?:final answer|the answer is|answer is|answer\s*[:\-：]|option is|choice is)" + r"\s*[\(（\*]{0,4}([A-Za-z])[\)）\*]{0,4}\s*[\.。!！]?\s*$", + re.IGNORECASE, +) +_HEAD_LABEL_RE = re.compile(r"\s*\*{0,2}\(?([A-Z])\)?\s*[:\.\)]\s") +_BOLD_LABEL_RE = re.compile(r"\*\*\s*\(?([A-Z])\)?\s*[:\.]") + + +def extract_option_robust( + text: Any, candidates: List[str], option_texts: Optional[Dict[str, str]] = None +) -> Optional[str]: + """Robust (tier-2) MCQ letter extraction for chain-of-thought output: + VLMEvalKit's can_infer core (token position + option-text restatement) + extended with -tag, \boxed{}, answer-cue (incl. CJK), and + answer-label rules. Returns the extracted candidate letter or None. + + Args: the raw prediction text (str or 1-element list), the candidate + letters, and the letter->option-text map for restatement matching. + """ + raw = _to_text(text) + if not raw: + return None + cand_set = [str(c).strip().upper() for c in (candidates or OPTION_LETTERS)] + + # 1. The whole (normalized) answer is one bare option letter, any case + # ("b", "(C)", "\boxed{D}") — unambiguous, accept directly. + whole = _WRAPPING_BRACKETS_RE.sub("", normalize_text(raw).strip()).strip() + if whole.upper() in cand_set: + return whole.upper() + + # 2. The LAST ... tag, when present, is the model's final + # pick — resolve inside it (bare letter, token match, text restatement). + tags = _ANSWER_TAG_RE.findall(raw) + if tags: + inner_raw = tags[-1].strip() + if inner_raw.upper() in cand_set: + return inner_raw.upper() + inner = can_infer_option(inner_raw, cand_set) or can_infer_text(inner_raw, option_texts or {}) + if inner: + return inner + + # 3. The LAST \boxed{...} letter, tolerating an inner \text{}/\mathrm{} + # wrap and a leading '(' / '*': "$\boxed{C}$", "$\boxed{\text{C}}$". + boxes = [b.upper() for b in _BOXED_LETTER_RE.findall(raw) if b.upper() in cand_set] + if boxes: + return boxes[-1] + + # 4. The LAST explicit answer cue (markdown/CJK tolerant, uppercase letter). + cues = [c.upper() for c in _ANSWER_CUE.findall(raw) if c.upper() in cand_set] + if cues: + return cues[-1] + + # 5. A lowercase letter after a cue counts only when it TERMINATES the + # response ("the answer is b") — there it cannot be an article. Mid-text + # lowercase letters stay ignored ("the answer is a cute cat" is not A). + tail = _CUE_TAIL_RE.search(raw) + if tail and tail.group(1).upper() in cand_set: + return tail.group(1).upper() + + # 6. VLMEvalKit core: the unique candidate letter near the end of the + # answer, ORIGINAL case (uppercasing first would make every article "a" + # a hit for option A). + opt = can_infer_option(raw, cand_set) + if opt: + return opt + + # 7. Answer-label formats the position/count test drops: + # 7a — answer-first label ("C: A man surfing") in SHORT responses only; + # long answers often reconsider the leading letter later, so they + # defer to the end-position/cue rules above. + if len(raw.strip()) <= 80: + head = _HEAD_LABEL_RE.match(raw) + if head and head.group(1).upper() in cand_set: + return head.group(1).upper() + # 7b — a uniquely bolded option label "**D:**" anywhere (the model's + # final restated pick). + bolds = {b.upper() for b in _BOLD_LABEL_RE.findall(raw) if b.upper() in cand_set} + if len(bolds) == 1: + return next(iter(bolds)) + + # 8. Fallback: the answer restates exactly one option's text. + return can_infer_text(raw, option_texts or {}) + + +def _letters_from_segment(segment: str, candidates: List[str]) -> Optional[Tuple[str, ...]]: + """A text segment that consists ONLY of candidate letters and separators + ("A, C" / "A and C" / "AC") -> sorted unique letters; anything else None.""" + text = normalize_text(segment).strip().rstrip(".。") + if not text: + return None + tokens = [t for t in _LETTER_SEPARATOR_RE.split(text) if t] + letters: List[str] = [] + for t in tokens: + up = t.upper() + if len(up) == 1 and up in candidates: + letters.append(up) + elif 2 <= len(up) <= len(candidates) and all(c in candidates for c in up) and len(set(up)) == len(up): + letters.extend(up) # compact form "AC" + else: + return None + if not letters or len(set(letters)) != len(letters): + return None + return tuple(sorted(letters)) + + +_MULTI_ANSWER_CUE_RE = re.compile( + r"(?:final answers?|the answers? (?:are|is)|answers? (?:are|is)|answers?\s*[:\-：]|最终答案|答案)", + re.IGNORECASE, +) + + +def extract_letter_set(text: Any, candidates: List[str]) -> Optional[Tuple[str, ...]]: + """Multi-select prediction -> sorted letter tuple. Tries, in order: the LAST + tag, the segment after the LAST answer cue, then the whole answer. + Only letters-and-separators segments are accepted — free text defers to the + LLM extractor (official LogicVista protocol: GPT extracts the letters, then + sorted set equality; vlmeval/dataset/utils/logicvista.py:50-68).""" + raw = _to_text(text) + if not raw: + return None + cand = [str(c).strip().upper() for c in (candidates or OPTION_LETTERS[:8])] + + tags = _ANSWER_TAG_RE.findall(raw) + if tags: + got = _letters_from_segment(tags[-1], cand) + if got: + return got + + cues = list(_MULTI_ANSWER_CUE_RE.finditer(raw)) + if cues: + got = _letters_from_segment(raw[cues[-1].end():], cand) + if got: + return got + + return _letters_from_segment(raw, cand) diff --git a/mmeval/scoring/pipeline.py b/mmeval/scoring/pipeline.py new file mode 100644 index 00000000..048692ff --- /dev/null +++ b/mmeval/scoring/pipeline.py @@ -0,0 +1,420 @@ +import glob +import json +import os +import platform +import subprocess +from collections import Counter +from concurrent.futures import ThreadPoolExecutor, as_completed +from typing import Any, Dict, List + +from mmeval.scoring.stages import INVALID, LLM_STAGE_NAMES, SCORE, STAGE_REGISTRY +from mmeval.scoring.stages.llm import _resolve_setting, local_judge_revision +from mmeval.scoring.protocol import ResolvedProtocol, extract_dataset_meta, resolve_protocol +from mmeval.scoring.report import build_summary +from mmeval.scoring.extraction import ( + OPTION_LETTERS, + extract_option_texts, + extract_options, + get_field, + get_question, + infer_question_type, + parse_gt_letters, + parsed_options, +) + +try: + from tqdm import tqdm +except Exception: # pragma: no cover - fallback when tqdm is unavailable + tqdm = None + + +def discover_result_files(out_dir: str, pattern: str = "**/result.json") -> List[str]: + if not out_dir: + return [] + search_pattern = os.path.join(out_dir, pattern) + return sorted(x for x in glob.glob(search_pattern, recursive=True) if os.path.isfile(x)) + + +def build_stages(args, pipeline): + """Instantiate the validated pipeline (list of atomic stage names). + Construction is uniform; LLM-backed stages read their client/config + from args, the rest ignore it.""" + return [STAGE_REGISTRY[name](args) for name in pipeline] + + +def _resume_fingerprint(args, proto: ResolvedProtocol) -> Dict[str, Any]: + """The RESOLVED config subset that determines a sample's verdict (post + CLI/dataset_meta/default precedence). Cached rows produced under a different + fingerprint must not be reused — resuming a rule-only run into a + `rule-match,llm-match` rerun would silently keep the old verdicts.""" + fingerprint = { + "score_pipeline": list(proto.pipeline), + "score_gt_field": args.score_gt_field, + "score_pred_field": args.score_pred_field, + "score_question_type": args.score_question_type, + "score_numeric_rel_tol": proto.numeric_rel_tol, + "score_numeric_abs_tol": proto.numeric_abs_tol, + "score_string_match": proto.string_match, + "score_anls_threshold": proto.anls_threshold, + } + # With an LLM stage in the pipeline, the judge's identity determines + # verdicts too — a gpt-4o-mini cache must not resume into a gpt-5 run. + # Judge-free pipelines deliberately exclude these so a judge-config edit + # doesn't invalidate them. + if any(name in LLM_STAGE_NAMES for name in proto.pipeline): + fingerprint["judge_provider"] = args.judge_provider + fingerprint["judge_model"] = args.judge_model + fingerprint["judge_temperature"] = args.judge_temperature + # Both change verdicts: include_reason alters the judge prompt, and + # max_tokens changes truncation/parse-failure behavior. Fingerprint the + # RESOLVED max_tokens (CLI > env > default), like the stage uses. + fingerprint["judge_include_reason"] = bool(args.judge_include_reason) + fingerprint["judge_max_tokens"] = _resolve_setting( + getattr(args, "judge_max_tokens", None), "JUDGE_MAX_TOKENS", 2048) + if (args.judge_provider or "").strip().lower() == "local": + # A local model name doesn't pin weights the way an API model + # string does — the resolved snapshot commit must invalidate too. + fingerprint["judge_model_revision"] = local_judge_revision(args.judge_model) + return fingerprint + + +def _provenance() -> Dict[str, Any]: + """Environment identity embedded in score.json so any run can be reproduced. + Deterministic on a given checkout (no timestamps — reruns stay byte-identical).""" + try: + commit = subprocess.run( + ["git", "rev-parse", "HEAD"], + capture_output=True, text=True, timeout=5, + cwd=os.path.dirname(os.path.abspath(__file__)), + ).stdout.strip() or None + except Exception: + commit = None + try: + from importlib.metadata import version + openai_version = version("openai") + except Exception: + openai_version = None + return { + "git_commit": commit, + "python": platform.python_version(), + "openai": openai_version, + } + + + +def _load_cached_results(out_path: str, resume_enabled: bool, fingerprint: Dict[str, Any]) -> Dict[str, Dict[str, Any]]: + if not resume_enabled: + return {} + + cache_by_id: Dict[str, Dict[str, Any]] = {} + tmp_path = f"{out_path}.tmp" + # Union of both caches: the final score.json first, then the tmp overlays + # it (fresher rows win). Both must pass the fingerprint check, so every + # retained row was produced under the current verdict-determining config. + for path in (out_path, tmp_path): + if not os.path.exists(path): + continue + try: + with open(path, "r") as f: + payload = json.load(f) + if not isinstance(payload, dict): + continue # a bare-list tmp carries no config fingerprint; don't trust it + config = payload.get("config") or {} + if any(config.get(k) != v for k, v in fingerprint.items()): + print(f"[score] ignoring resume cache {path}: scoring config changed", flush=True) + continue + rows = payload.get("samples") + if not isinstance(rows, list): + continue + for row in rows: + if not isinstance(row, dict): + continue + if "eval-id" not in row: + continue + # Rows that failed because the judge API failed are not verdicts; + # re-score them instead of freezing a transient outage into 0s. + if str(row.get("reason", "")).startswith("llm_error"): + continue + cache_by_id[str(row["eval-id"])] = row + except Exception: + continue + return cache_by_id + + +def _flush_resume_tmp(tmp_path: str, samples: List[Dict[str, Any]], fingerprint: Dict[str, Any]) -> None: + # Atomic write: a crash mid-flush must never leave a truncated tmp that a + # resume would then have to discard (losing all progress since the last flush). + part_path = f"{tmp_path}.part" + with open(part_path, "w") as f: + json.dump({"config": fingerprint, "samples": samples}, f, indent=2) + os.replace(part_path, tmp_path) + + +def _score_single_sample(sample: Dict[str, Any], args, stages, proto: ResolvedProtocol) -> Dict[str, Any]: + gt = get_field(sample, args.score_gt_field, None) + if (gt is None or (isinstance(gt, str) and not gt.strip())) and args.score_gt_field == "answer": + # Judge-graded datasets carry the reference in the reserved + # `reference_response` field instead of `answer` — generic fallback, + # never per-dataset. + gt = get_field(sample, "reference_response", None) + pred = get_field(sample, args.score_pred_field, None) + question = get_question(sample) + question_type = infer_question_type(sample, args.score_question_type, gt=gt) + options = extract_options(sample) if question_type == "mcq" else [] + if question_type == "mcq": + # The gt letters prove candidate membership: when option discovery + # provably missed some (malformed inline lists, e.g. a "B Only ..." + # line without label punctuation), widen to the contiguous A..max + # span instead of failing on an option the dataset says exists. + gt_letters = parse_gt_letters(gt) or () + missing = [l for l in gt_letters if l not in options] + if missing: + top = max(OPTION_LETTERS.index(l) for l in list(options) + list(gt_letters) + if l in OPTION_LETTERS) + options = OPTION_LETTERS[:top + 1] + # Whether the candidates are a real parsed option source (vs the A-F + # fallback) — gates compact multi-letter gt parsing in the rule stages. + options_parsed = question_type == "mcq" and parsed_options(sample) is not None + option_texts = extract_option_texts(sample, options) + + eval_id = sample.get("eval-id", sample.get("id")) + output = { + "eval-id": eval_id, + "question_type": question_type, + "gt": gt, + "pred": pred, + "decided_by": None, + "matched": None, + "score": 0.0, + "is_correct": 0, + "status": "ok", + } + + # Blank gt (answer-withheld test splits) must be invalid, not "confidently 0". + if gt is None or pred is None or (isinstance(gt, str) and not gt.strip()): + output["status"] = "invalid" + output["reason"] = "missing_gt_or_pred" + return output + + context = { + "question_type": question_type, + "question": question, + "pred": pred, + "gt": gt, + "options": options, + "options_parsed": options_parsed, + "option_texts": option_texts, + "numeric_rel_tol": proto.numeric_rel_tol, + "numeric_abs_tol": proto.numeric_abs_tol, + "string_match": proto.string_match, + "anls_threshold": proto.anls_threshold, + } + + trace = [] + for stage in stages: + result = stage.run(sample, context) + if args.score_debug: + trace.append( + { + "stage": stage.name, + "outcome": result.outcome, + "score": result.score, + "matched": result.matched, + "reason": result.reason, + "meta": result.meta, + } + ) + if result.meta and "judge_prob" in result.meta: + # Observability only (P of the judge's verdict token); never a verdict input. + output["judge_prob"] = result.meta["judge_prob"] + if result.outcome == INVALID: + # The sample cannot be graded under this stage's protocol. + output["status"] = "invalid" + output["reason"] = result.reason + break + if result.outcome == SCORE: + # Decided: short-circuit. is_correct = full credit only; the + # fractional per-sample `score` feeds summary.mean_score. + output["decided_by"] = stage.name + output["matched"] = result.matched + output["score"] = float(result.score) + output["is_correct"] = int(result.score >= 1.0) + if result.reason is not None and args.judge_include_reason: + output["reason"] = result.reason + break + if stage.uses_llm and result.reason is not None: + # LLM stage pass-reasons are error markers ("llm_error: ...") or — + # only when --judge_include_reason — the judge's explanation; both + # must survive into the output unless a later stage decides + # (errors especially: see summary.llm_errors). + output["reason"] = result.reason + + if args.score_debug: + output["trace"] = trace + return output + + +def _score_sample_at(idx: int, sample: Dict[str, Any], args, stages, proto: ResolvedProtocol): + """Worker wrapper: score one sample, keep its position, default a missing eval-id.""" + scored = _score_single_sample(sample, args, stages, proto) + if scored["eval-id"] is None: + scored["eval-id"] = sample.get("eval-id", sample.get("id", idx)) + return idx, scored + + +def _reject_nonconforming_layout(result_file: str, samples: List[Dict[str, Any]], gt_field: str) -> None: + """Grading fields live ONLY at the sample top level; fail fast instead of + scoring every sample as missing-gt.""" + for sample in samples: + if not isinstance(sample, dict): + continue + messages = sample.get("messages") or [] + msg = messages[0] if messages and isinstance(messages[0], dict) else {} + if gt_field in msg and gt_field not in sample: + raise ValueError( + f"{result_file}: non-conforming result format — `{gt_field}` found inside " + f"messages[0]. Grading fields (answer, question_type, reference_response, " + f"--score_gt_field) must be TOP-LEVEL sample fields; messages carry only " + f"render inputs and the model response." + ) + + +def score_result_file(result_file: str, args) -> Dict[str, Any]: + with open(result_file, "r") as f: + samples = json.load(f) + if not isinstance(samples, list): + raise ValueError(f"{result_file} must be a JSON list.") + _reject_nonconforming_layout(result_file, samples, args.score_gt_field) + + # Metadata-driven protocol resolution (docs/en/SCORING.md): per knob, + # explicit CLI flag > dataset_meta from result.json > built-in default. + # Conflicting dataset_meta across samples or unsupported protocols raise here. + dataset_meta = extract_dataset_meta(samples, result_file) + proto = resolve_protocol(args, dataset_meta, result_file) + + out_path = os.path.join(os.path.dirname(result_file), args.score_output_name) + tmp_path = f"{out_path}.tmp" + fingerprint = _resume_fingerprint(args, proto) + cached_by_id = _load_cached_results(out_path=out_path, resume_enabled=args.score_resume, fingerprint=fingerprint) + stages = build_stages(args, proto.pipeline) + + scored_samples: List[Dict[str, Any]] = [None] * len(samples) + resumed_count = 0 + pbar = None + if args.score_progress_bar and tqdm is not None: + pbar = tqdm( + total=len(samples), + desc=f"Scoring {os.path.basename(os.path.dirname(result_file))}", + leave=False, + ) + + # Samples whose normalized eval-id is not unique (true duplicates, or an int/str + # collision like 1 vs "1") cannot be safely resumed from cache: one cached row + # would stand in for several distinct samples. Always re-score those. + id_counts = Counter( + str(s.get("eval-id", s.get("id", i))) if isinstance(s, dict) else str(i) + for i, s in enumerate(samples) + ) + dup_ids = {k for k, v in id_counts.items() if v > 1} + if dup_ids and cached_by_id: + print(f"[score] {len(dup_ids)} non-unique eval-id(s) in {result_file}; those samples will be re-scored, not resumed", flush=True) + + uncached_samples = [] + for idx, sample in enumerate(samples): + eval_id = str(sample.get("eval-id", sample.get("id", idx))) + cached = cached_by_id.get(eval_id) if eval_id not in dup_ids else None + if cached is not None: + scored_samples[idx] = cached + resumed_count += 1 + if pbar is not None: + pbar.update(1) + continue + uncached_samples.append((idx, sample)) + + # Threads only help when samples block on API calls (LLM stages); for + # judge-free pipelines the GIL makes them a net slowdown (measured + # ~3x on 10k samples). Default to serial there; an explicit + # --parallel_per_task always wins. + uses_api = any(n in LLM_STAGE_NAMES for n in proto.pipeline) + if "score_parallel_per_task" in getattr(args, "_explicit_flags", frozenset()) or uses_api: + sample_workers = max(1, int(getattr(args, "score_parallel_per_task", 1))) + else: + sample_workers = 1 + processed_new = 0 + + if sample_workers <= 1 or len(uncached_samples) <= 1: + for idx, sample in uncached_samples: + i, scored = _score_sample_at(idx, sample, args, stages, proto) + scored_samples[i] = scored + processed_new += 1 + if pbar is not None: + pbar.update(1) + if args.score_resume and processed_new % max(1, int(args.score_save_freq)) == 0: + _flush_resume_tmp(tmp_path, [x for x in scored_samples if x is not None], fingerprint) + else: + with ThreadPoolExecutor(max_workers=sample_workers) as executor: + futures = [ + executor.submit(_score_sample_at, idx, sample, args, stages, proto) + for idx, sample in uncached_samples + ] + for future in as_completed(futures): + i, scored = future.result() + scored_samples[i] = scored + processed_new += 1 + if pbar is not None: + pbar.update(1) + if args.score_resume and processed_new % max(1, int(args.score_save_freq)) == 0: + _flush_resume_tmp(tmp_path, [x for x in scored_samples if x is not None], fingerprint) + + if pbar is not None: + pbar.close() + + if any(x is None for x in scored_samples): + raise RuntimeError(f"Incomplete scoring results found for {result_file}") + + summary = build_summary(scored_samples) + + payload = { + "summary": summary, + "config": { + # RESOLVED per-knob values (CLI > dataset_meta > default); which + # source decided each knob is recorded in knob_sources. The resume + # fingerprint is embedded verbatim: _load_cached_results validates + # cached rows against this block, so it must stay a superset of + # every fingerprint key. + **fingerprint, + "task_type": proto.task_type, + "official_protocol": proto.official_protocol, + "knob_sources": proto.knob_sources, + "judge_provider": args.judge_provider, + "judge_model": args.judge_model, + "judge_temperature": args.judge_temperature, + "judge_include_reason": args.judge_include_reason, + "score_progress_bar": args.score_progress_bar, + "score_resume": args.score_resume, + "score_save_freq": args.score_save_freq, + "score_parallel_per_task": sample_workers, + "resumed_count": resumed_count, + "provenance": _provenance(), + # List of notes: the scorer-wide caveat, rubric degradations, the + # none-stamp, and score_protocol.note verbatim (score_note). + "protocol_notes": proto.protocol_notes, + }, + "samples": scored_samples, + } + + with open(out_path, "w") as f: + json.dump(payload, f, indent=2) + # A finished score.json supersedes any resume tmp regardless of how this + # run was configured — a stale tmp left behind would overlay OLDER rows + # onto a fresher final in a later resume's cache union. + if os.path.exists(tmp_path): + os.remove(tmp_path) + + return { + "result_file": result_file, + "score_file": out_path, + "summary": summary, + "resumed_count": resumed_count, + } + diff --git a/mmeval/scoring/protocol.py b/mmeval/scoring/protocol.py new file mode 100644 index 00000000..455c0fd2 --- /dev/null +++ b/mmeval/scoring/protocol.py @@ -0,0 +1,240 @@ +"""score_pipeline -> execution resolution (the metadata-driven scoring contract). + +Every sample may carry a flat top-level `dataset_meta` block (injected at +inference time by the mmeval_hf loader, persisted in result.json). Its +`score_pipeline` declares the dataset's official scoring protocol as an +ORDERED LIST OF ATOMIC STAGE NAMES (see mmeval/scoring/stages/): + + "score_pipeline": ["exact-match", "rule-match", "llm-judge"] + +Two non-list forms are part of the contract: +- [] — the dataset explicitly declares that no + official protocol exists; the scorer runs the default pipeline and stamps + the output non-official. +- {"unsupported": "", — the official protocol cannot be executed + "reason": ""} by this scorer; scoring is REFUSED with + an actionable error (an explicit --score_pipeline forces approximate scoring, + stamped non-official). + +The scorer resolves each knob with the precedence + explicit CLI flag > dataset_meta > built-in default, +records which source decided every knob (knob_sources), and stamps +official-protocol status. +""" +import json +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + +from mmeval.scoring.stages import STAGE_REGISTRY, GRADER_STAGE_NAMES + +# The scorer-wide caveat recorded into every score.json (first protocol note). +GLOBAL_PROTOCOL_NOTE = ( + "Per-sample scoring; official aggregations noted per dataset in these notes " + "are not reproduced. LLM failures are marked wrong and counted in " + "summary.llm_errors (never randomly guessed, unlike VLMEvalKit)." +) + +NO_PROTOCOL_NOTE = ( + "score_pipeline=[]: the dataset declares that no official eval protocol " + "exists; numbers are mm-eval-convention, not official benchmark results." +) + +_STRING_MATCH_VALUES = ("exact", "contains", "anls") + + +def parse_pipeline(spec, where: str) -> List[str]: + """Validate a pipeline spec (comma-separated string from the CLI, or a JSON + array from dataset metadata) into an ordered list of atomic stage names.""" + if isinstance(spec, str): + stages = [x.strip() for x in spec.split(",") if x.strip()] + elif isinstance(spec, (list, tuple)): + stages = [str(x).strip() for x in spec if str(x).strip()] + else: + raise ValueError(f"{where}: score pipeline must be a list of stage names, got {type(spec).__name__}") + if not stages: + raise ValueError(f"{where}: empty pipeline; known stages: {sorted(STAGE_REGISTRY)}") + unknown = [s for s in stages if s not in STAGE_REGISTRY] + if unknown: + raise ValueError(f"{where}: unknown stage(s) {unknown}; known stages: {sorted(STAGE_REGISTRY)}") + if len(set(stages)) != len(stages): + raise ValueError(f"{where}: duplicate stages in pipeline {stages}") + for s in stages[:-1]: + if s in GRADER_STAGE_NAMES: + raise ValueError( + f"{where}: `{s}` is a grader stage — it always decides every " + f"sample, so stages after it can never run; a grader must be " + f"the pipeline's last stage (and at most one per pipeline)" + ) + return stages + + +@dataclass +class ResolvedProtocol: + """Effective per-file scoring configuration after precedence resolution.""" + pipeline: List[str] + numeric_rel_tol: float + numeric_abs_tol: float + string_match: str # exact | contains | anls + anls_threshold: float + task_type: Optional[str] + official_protocol: Optional[bool] # None = unknown (no declaration) + protocol_notes: List[str] = field(default_factory=list) + knob_sources: Dict[str, str] = field(default_factory=dict) # knob -> cli|dataset_meta|default + + +def extract_dataset_meta(samples: List[Dict[str, Any]], result_file: str) -> Optional[Dict[str, Any]]: + """The single dataset_meta block of a result file, or None. Conflicting + blocks across samples mean a hand-merged file — refuse, never pick one.""" + distinct: Dict[str, Dict[str, Any]] = {} + for sample in samples: + if not isinstance(sample, dict): + continue + meta = sample.get("dataset_meta") + if isinstance(meta, dict) and meta: + distinct[json.dumps(meta, sort_keys=True)] = meta + if len(distinct) > 1: + keys = "\n ".join(sorted(distinct)) + raise ValueError( + f"{result_file} carries {len(distinct)} conflicting dataset_meta blocks " + f"(hand-merged file?). Score each source file separately, or strip the " + f"blocks and pass explicit CLI flags.\n {keys}" + ) + return next(iter(distinct.values()), None) + + +def _explicit(args) -> frozenset: + return getattr(args, "_explicit_flags", frozenset()) + + +def resolve_protocol(args, dataset_meta: Optional[Dict[str, Any]], result_file: str) -> ResolvedProtocol: + """Apply the per-knob precedence CLI > dataset_meta > default. Raises + ValueError for unsupported protocols (unless the user explicitly forced a + pipeline via --score_pipeline) and for the retired composite metadata schema.""" + explicit = _explicit(args) + meta = dataset_meta or {} + if "score_type" in meta: + raise ValueError( + f"{result_file}: dataset_meta uses the retired composite `score_type` " + f"vocabulary — this result.json was produced by an older loader. " + f"Re-run inference with the current loader (it translates published " + f"metadata to `score_pipeline`), or re-push the dataset with the " + f"score_pipeline schema." + ) + declared = meta.get("score_pipeline") + params = meta.get("score_params") or {} + + notes: List[str] = [GLOBAL_PROTOCOL_NOTE] + sources: Dict[str, str] = {} + + # --- pipeline ------------------------------------------------------------ + declared_stages: Optional[List[str]] = None # a declared, runnable protocol + unsupported: Optional[Dict[str, Any]] = None + if isinstance(declared, dict): + if "unsupported" not in declared: + raise ValueError( + f"{result_file}: malformed score_pipeline object {declared!r}; " + f'expected {{"unsupported": "", "reason": "..."}} or a stage list' + ) + unsupported = declared + elif isinstance(declared, (list, tuple)): + declared_stages = parse_pipeline(declared, f"{result_file} dataset_meta.score_pipeline") if declared else [] + elif declared is not None: + raise ValueError( + f"{result_file}: score_pipeline must be a stage list or an " + f'{{"unsupported": ...}} object, got {type(declared).__name__}' + ) + + if "score_pipeline" in explicit: + pipeline = parse_pipeline(args.score_pipeline, "--score_pipeline") + sources["score_pipeline"] = "cli" + if unsupported is not None: + notes.append( + f"explicit --score_pipeline overrides the dataset's unsupported official " + f"protocol ({unsupported.get('unsupported')}): numbers are approximate, " + f"NOT comparable to official results." + ) + elif declared_stages and pipeline != declared_stages: + notes.append( + f"explicit --score_pipeline overrides the dataset's declared pipeline " + f"{declared_stages}: the official protocol is NOT being executed; " + f"numbers are not comparable to official results." + ) + elif unsupported is not None: + name = unsupported.get("unsupported", "unknown") + reason = unsupported.get("reason", "no reason recorded") + raise ValueError( + f"{result_file}: the dataset's official protocol ({name}) is unsupported: " + f"{reason}. Pass an explicit --score_pipeline to force approximate scoring " + f"(numbers will NOT be official)." + ) + elif declared_stages: + pipeline = declared_stages + sources["score_pipeline"] = "dataset_meta" + else: # no declaration ([] or absent): built-in default pipeline + pipeline = parse_pipeline(args.score_pipeline, "--score_pipeline default") + sources["score_pipeline"] = "default" + + # --- numeric / string knobs (CLI > score_params > default) -------------- + def _knob(cli_name: str, param_name: str, default): + if cli_name in explicit: + sources[cli_name] = "cli" + return getattr(args, cli_name) + if param_name in params: + sources[cli_name] = "dataset_meta" + return params[param_name] + sources[cli_name] = "default" + return default + + numeric_rel_tol = float(_knob("score_numeric_rel_tol", "numeric_rel_tol", args.score_numeric_rel_tol)) + numeric_abs_tol = float(_knob("score_numeric_abs_tol", "numeric_abs_tol", args.score_numeric_abs_tol)) + string_match = str(_knob("score_string_match", "string_match", args.score_string_match)) + if string_match not in _STRING_MATCH_VALUES: + raise ValueError(f"{result_file}: invalid string_match {string_match!r}; allowed: {_STRING_MATCH_VALUES}") + # anls params spell it `threshold`; the rule-stage comparator knob is + # `anls_threshold` — both feed the same resolved value. + param_thresh = params.get("threshold", params.get("anls_threshold")) + if "score_anls_threshold" in explicit: + anls_threshold = float(args.score_anls_threshold) + sources["score_anls_threshold"] = "cli" + elif param_thresh is not None: + anls_threshold = float(param_thresh) + sources["score_anls_threshold"] = "dataset_meta" + else: + anls_threshold = float(args.score_anls_threshold) + sources["score_anls_threshold"] = "default" + + # --- judge rubric degradation note -------------------------------------- + rubric = params.get("rubric") + if rubric and rubric != "binary": + notes.append( + f"official rubric `{rubric}`" + + (f" scale={params['scale']}" if params.get("scale") else "") + + " degraded to a binary llm-judge; official numbers aggregate " + "the fractional/scaled judge scores and are not directly comparable." + ) + + # --- official-protocol stamp + score_note ------------------------------- + if declared is None: + official = None # nothing declared at all + elif declared_stages == []: + official = False # explicit "no official protocol" + notes.append(NO_PROTOCOL_NOTE) + elif sources["score_pipeline"] == "cli" and (unsupported is not None or pipeline != declared_stages): + official = False # declared protocol overridden from the CLI + else: + official = True + score_note = (meta.get("score_note") or "").strip() + if score_note: + notes.append(score_note) # verbatim, per the metadata contract + + return ResolvedProtocol( + pipeline=pipeline, + numeric_rel_tol=numeric_rel_tol, + numeric_abs_tol=numeric_abs_tol, + string_match=string_match, + anls_threshold=anls_threshold, + task_type=meta.get("task_type"), + official_protocol=official, + protocol_notes=notes, + knob_sources=sources, + ) diff --git a/mmeval/scoring/report.py b/mmeval/scoring/report.py new file mode 100644 index 00000000..b1ed1cb1 --- /dev/null +++ b/mmeval/scoring/report.py @@ -0,0 +1,60 @@ +from collections import defaultdict +from typing import Any, Dict, List + + +def build_summary(sample_results: List[Dict[str, Any]]) -> Dict[str, Any]: + total = len(sample_results) + correct = sum(1 for x in sample_results if x.get("is_correct") == 1) + # accuracy divides by ALL samples (invalid rows count as wrong); + # accuracy_valid divides by the gradable samples only — the honest number + # on splits with withheld answers or malformed rows. + accuracy = (correct / total) if total else 0.0 + + by_stage = defaultdict(lambda: {"total": 0, "correct": 0}) + by_question_type = defaultdict(lambda: {"total": 0, "correct": 0}) + invalid = 0 + llm_errors = 0 + judge_probs = [] + for item in sample_results: + stage = item.get("decided_by") or "undecided" + qtype = item.get("question_type") or "unknown" + by_stage[stage]["total"] += 1 + by_stage[stage]["correct"] += int(item.get("is_correct") == 1) + by_question_type[qtype]["total"] += 1 + by_question_type[qtype]["correct"] += int(item.get("is_correct") == 1) + if item.get("status") == "invalid": + invalid += 1 + # Samples whose judge-stage API call failed (API/parse error) — their 0s are + # not verdicts; a non-zero count here means the accuracy is a lower bound. + if str(item.get("reason", "")).startswith("llm_error"): + llm_errors += 1 + if item.get("judge_prob") is not None: + judge_probs.append(item["judge_prob"]) + + # accuracy = strict full-credit rate (mean is_correct). mean_score averages + # the fractional per-sample `score` (metric stages emit true fractions; + # rule/judge stages emit 1.0/0.0) — the OFFICIAL headline for fractional + # protocols. Equal to accuracy whenever no sample carries a fractional score. + mean_score = ( + sum((x["score"] if x.get("score") is not None else float(x.get("is_correct") == 1)) + for x in sample_results) / total + if total else 0.0 + ) + valid = total - invalid + summary = { + "total": total, + "valid": valid, + "correct": correct, + "accuracy": accuracy, + "accuracy_valid": (correct / valid) if valid else 0.0, + "mean_score": mean_score, + "invalid": invalid, + "llm_errors": llm_errors, + "by_stage": dict(by_stage), + "by_question_type": dict(by_question_type), + } + # Mean P(verdict) over judge-graded samples that returned logprobs — + # observability for how confident the judge was, never a verdict input. + if judge_probs: + summary["judge_prob_mean"] = sum(judge_probs) / len(judge_probs) + return summary diff --git a/mmeval/scoring/stages/__init__.py b/mmeval/scoring/stages/__init__.py new file mode 100644 index 00000000..2e1c933e --- /dev/null +++ b/mmeval/scoring/stages/__init__.py @@ -0,0 +1,43 @@ +from mmeval.scoring.stages.base import BaseStage, StageResult, INVALID, PASS, SCORE +from mmeval.scoring.stages.exact_match import ExactMatchStage +from mmeval.scoring.stages.rule_match import RuleMatchStage +from mmeval.scoring.stages.llm import LLMMatchStage, LLMJudgeStage +from mmeval.scoring.stages.metrics import AnlsStage, VqaAccuracyStage + +# Every atomic scoring stage, by pipeline name (mainstream eval vocabulary: +# exact match, answer extraction, LLM-as-judge, VQA accuracy, ANLS). +STAGE_REGISTRY = { + "exact-match": ExactMatchStage, + "rule-match": RuleMatchStage, + # LLM extracts the chosen option, then rule-based set compare (VLMEvalKit). + "llm-match": LLMMatchStage, + # LLM judges whether the response is correct (our approach). + "llm-judge": LLMJudgeStage, + "vqa-accuracy": VqaAccuracyStage, + "anls": AnlsStage, +} + +# Derived from the stage classes' own declarations — the class attribute is +# the single source of truth, so a new stage cannot silently miss these sets. +# uses_llm: the stage calls an LLM API; its presence pulls the judge identity +# into the resume fingerprint. grader: the stage always decides (or +# invalidates) and never abstains, so it is only valid as the last stage. +LLM_STAGE_NAMES = frozenset(n for n, c in STAGE_REGISTRY.items() if c.uses_llm) +GRADER_STAGE_NAMES = frozenset(n for n, c in STAGE_REGISTRY.items() if c.grader) + +__all__ = [ + "BaseStage", + "StageResult", + "PASS", + "SCORE", + "INVALID", + "ExactMatchStage", + "RuleMatchStage", + "LLMMatchStage", + "LLMJudgeStage", + "VqaAccuracyStage", + "AnlsStage", + "STAGE_REGISTRY", + "LLM_STAGE_NAMES", + "GRADER_STAGE_NAMES", +] diff --git a/mmeval/scoring/stages/base.py b/mmeval/scoring/stages/base.py new file mode 100644 index 00000000..2cd59ad4 --- /dev/null +++ b/mmeval/scoring/stages/base.py @@ -0,0 +1,67 @@ +"""The atomic scoring stage: the single primitive behind every pipeline. + +A scoring pipeline is an ordered list of stages. For each sample, stages run +in order; each returns a StageResult with one of three outcomes: + +- PASS — the stage cannot decide this sample; the next stage runs. If every + stage passes, the sample scores 0.0 with decided_by null + (undecided, counted wrong). A pass may carry a reason — notably + "llm_error: ..." markers, which survive into the output row and + summary.llm_errors unless a later stage decides the sample. +- SCORE — the stage decided the sample and the pipeline short-circuits. + `score` is in [0, 1]: rule stages emit 1.0 on a match and never + decide a miss (they pass instead); llm-judge emits 1.0/0.0; + metric stages emit fractional values. is_correct = score >= 1.0. +- INVALID — the sample cannot be graded under this stage's protocol (e.g. + vqa-accuracy without the multi-annotator reference list). Terminal: + the sample is recorded as status=invalid with the stage's reason. +""" +from dataclasses import dataclass, field +from typing import Any, Dict, Optional + +PASS = "pass" +SCORE = "score" +INVALID = "invalid" + + +@dataclass +class StageResult: + outcome: str + score: float = 0.0 + matched: Optional[str] = None + reason: Optional[str] = None + meta: Dict[str, Any] = field(default_factory=dict) + + @classmethod + def passed(cls, reason: Optional[str] = None, meta: Optional[Dict[str, Any]] = None) -> "StageResult": + return cls(outcome=PASS, reason=reason, meta=meta or {}) + + @classmethod + def scored(cls, score: float, matched: Optional[str] = None, + reason: Optional[str] = None, meta: Optional[Dict[str, Any]] = None) -> "StageResult": + return cls(outcome=SCORE, score=float(score), matched=matched, reason=reason, meta=meta or {}) + + @classmethod + def not_gradable(cls, reason: str) -> "StageResult": + return cls(outcome=INVALID, reason=reason) + + +class BaseStage: + name = "base" + # Declared by each stage class; the registry derives LLM_STAGE_NAMES / + # GRADER_STAGE_NAMES from these, so no behavior is keyed off name sets. + uses_llm = False # calls an LLM API: judge identity enters the fingerprint + # grader: the stage always decides (fractional score 0..1) or invalidates, + # never abstains -> only valid as the last pipeline stage, at most one. + # DELIBERATE COUPLING: `grader` bundles "fractional output" with "always + # decides / last-only". If a future stage ever needs one without the + # other, split this flag — known, accepted trade-off, not an oversight. + grader = False + + def __init__(self, args=None): + # Uniform construction: every stage accepts the parsed args; stages + # without external clients ignore them. + self.args = args + + def run(self, sample: Dict[str, Any], context: Dict[str, Any]) -> StageResult: + raise NotImplementedError diff --git a/mmeval/scoring/stages/exact_match.py b/mmeval/scoring/stages/exact_match.py new file mode 100644 index 00000000..7246e0df --- /dev/null +++ b/mmeval/scoring/stages/exact_match.py @@ -0,0 +1,147 @@ +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.extraction import ( + extract_letter_set, + extract_option_strict, + normalize_for_exact, + normalize_open_answer, + parse_gt_letters, + parse_number, +) + + +def extract_letters_strict(pred, candidates): + """Strict-tier letter-set extraction: a clean letters-and-separators answer + ("B", "A, C", "BD"), else a single unambiguous strict-form letter.""" + letters = extract_letter_set(pred, candidates) + if letters: + return letters + single = extract_option_strict(pred, candidates) + return (single,) if single else None + + +def numbers_match(pred: float, gt: float, context) -> bool: + """Exact float equality by default; the protocol's tolerances widen it: + relative (`numeric_rel_tol` — ChartQA 0.05) and/or absolute + (`numeric_abs_tol` — DynaMath 0.001, OlympiadBench-style epsilon). Either + satisfied tolerance matches. Relative tolerance does NOT apply when the + ground truth is 0 — the official relaxed-accuracy protocol falls back to + exact matching there (lmms-eval chartqa/utils.py relaxed_correctness: + `target_float` truthiness sends target==0 to the exact-match branch).""" + if pred == gt: + return True + abs_tol = float(context.get("numeric_abs_tol") or 0.0) + if abs_tol > 0.0 and abs(pred - gt) <= abs_tol: + return True + rel_tol = float(context.get("numeric_rel_tol") or 0.0) + if rel_tol > 0.0 and gt != 0.0: + return abs(pred - gt) / abs(gt) <= rel_tol + return False + + +def numeric_branch_active(context) -> bool: + """The numeric comparator applies when the protocol declares numeric + tolerances or uses the default exact string matching. A protocol that + declares a NON-exact string_match without numeric tolerances (OCRBench: + `contains`) makes that string comparator official for every free-form + row, digit-string answers included (VLMEvalKit + vlmeval/dataset/utils/ocrbench.py:36-42 applies the substring rule to all + categories). A protocol declaring BOTH (ChartQAPro: `anls` + + numeric_rel_tol) dispatches by answer shape, so tolerances keep the + numeric branch alive.""" + return (str(context.get("string_match") or "exact") == "exact" + or float(context.get("numeric_rel_tol") or 0.0) > 0.0 + or float(context.get("numeric_abs_tol") or 0.0) > 0.0) + + +def gt_references(gt) -> list: + """A gt may be a single value or a reference LIST (OCRBench answer lists, + VQA-style multi-annotator answers).""" + if isinstance(gt, (list, tuple)): + return [x for x in gt if x is not None] + return [gt] + + +def open_text_match(pred_text: str, pred_raw, gt_raw, context) -> bool: + """The rule stages' text comparator, per the protocol's `string_match`: + exact (default) — normalized equality against any reference; + contains — OCRBench protocol: a normalized reference appears as a substring + of the normalized prediction; + anls — threshold ANLS against the references (ChartQAPro).""" + mode = context.get("string_match") or "exact" + refs = gt_references(gt_raw) + if mode == "anls": + from mmeval.scoring.stages.metrics import anls + score, error = anls(pred_raw, [str(r) for r in refs], + threshold=float(context.get("anls_threshold") or 0.5)) + return error is None and score > 0.0 + pred_norm = normalize_open_answer(pred_text) + if not pred_norm: + return False + for ref in refs: + ref_norm = normalize_open_answer(ref) + if not ref_norm: + continue + if mode == "contains": + if ref_norm in pred_norm: + return True + elif pred_norm == ref_norm: + return True + return False + + +class ExactMatchStage(BaseStage): + """Strict-form deterministic comparison: near-zero false positives, zero + cost. Matches only unambiguous answers (a bare option letter, a clean + letter set, a pure yes/no, a whole-string number or text); anything it + cannot decide is passed to the next stage.""" + + name = "exact-match" + + def run(self, sample, context): + question_type = context["question_type"] + pred_raw = context["pred"] + gt_raw = context["gt"] + + if gt_raw is None or pred_raw is None: + return StageResult.passed() + + if question_type == "mcq": + # Letter-SET grading: single-select is the |set|=1 special case, + # multi-letter gts like LogicVista's "A, C" the general one. + candidates = context["options"] + # Compact multi-letter gts ("AC") are accepted HERE — the row is + # already typed mcq — while parse_gt_letters stays conservative for + # type inference. Only with REAL parsed options, though: under the + # A-F fallback (options-in-image datasets) word-like gts would + # parse as letter sets ("BED" -> B,D,E). + gt_letters = parse_gt_letters(gt_raw) + if gt_letters is None and context.get("options_parsed"): + gt_letters = extract_letter_set(gt_raw, candidates) + if gt_letters is None: + g = extract_option_strict(gt_raw, candidates) or normalize_for_exact(gt_raw).upper() + gt_letters = (g,) if g else None + pred_letters = extract_letters_strict(pred_raw, candidates) + if gt_letters and pred_letters and pred_letters == gt_letters: + return StageResult.scored(1.0, matched=", ".join(gt_letters)) + return StageResult.passed() + + if question_type == "yes_no": + pred_norm = normalize_open_answer(pred_raw) + if pred_norm in {"yes", "no"} and pred_norm == normalize_open_answer(gt_raw): + return StageResult.scored(1.0, matched=pred_norm) + return StageResult.passed() + + if question_type == "numeric" and numeric_branch_active(context): + # Strict tier: the whole prediction is a number. + gt_num = parse_number(gt_raw) + pred_num = parse_number(pred_raw) + if gt_num is not None and pred_num is not None and numbers_match(pred_num, gt_num, context): + return StageResult.scored(1.0, matched=str(gt_raw)) + return StageResult.passed() + # numeric-shaped rows under a declared non-exact string_match fall + # through to the protocol's string comparator below. + + if open_text_match(str(pred_raw if not isinstance(pred_raw, list) else (pred_raw[0] if pred_raw else "")), + pred_raw, gt_raw, context): + return StageResult.scored(1.0, matched=str(gt_raw)) + return StageResult.passed() diff --git a/mmeval/scoring/stages/llm.py b/mmeval/scoring/stages/llm.py new file mode 100644 index 00000000..4b575ea6 --- /dev/null +++ b/mmeval/scoring/stages/llm.py @@ -0,0 +1,455 @@ +import json +import math +import os +import re +import threading +import time +from typing import Any, Dict + +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.extraction import ( + can_infer_option, + can_infer_text, + extract_letter_set, + extract_option_robust, + normalize_for_exact, + normalize_text, + parse_gt_letters, +) + + +class _LocalJudgeClient: + """In-process open-weight judge (judge_provider=local). Loads judge_model + with the framework's native loading conventions (mmeval/infer/qwen3d5.py: + AutoModelForImageTextToText + AutoProcessor, dtype/device_map auto) inside + the model's registry-declared environment, and exposes the minimal + chat-completions surface the judge stages consume — so prompts, parsing, + retry/llm_error accounting and the semaphore are identical across + providers. Generation is serialized by a lock: the bound is the GPU, not + an API rate limit.""" + + def __init__(self, model_name: str): + import types as _types + from transformers import AutoModelForImageTextToText, AutoProcessor + self._processor = AutoProcessor.from_pretrained(model_name) + self._tok = getattr(self._processor, "tokenizer", self._processor) + self._model = AutoModelForImageTextToText.from_pretrained( + model_name, dtype="auto", device_map="auto") + self._model.eval() + self._lock = threading.Lock() + self.chat = _types.SimpleNamespace( + completions=_types.SimpleNamespace(create=self._create)) + + def _create(self, model, messages, temperature=0.0, max_tokens=512, + logprobs=False, **_ignored): + import torch + from types import SimpleNamespace as NS + try: + # Thinking-mode families (Qwen3/Qwen3.5) reason at length before + # the terse judge/extraction answer; disable it — the verdict + # contract is unchanged and per-call latency drops ~10x. + prompt = self._tok.apply_chat_template( + messages, add_generation_prompt=True, tokenize=False, + enable_thinking=False) + except TypeError: + prompt = self._tok.apply_chat_template( + messages, add_generation_prompt=True, tokenize=False) + inputs = self._tok(prompt, return_tensors="pt").to(self._model.device) + gen_kwargs = dict(max_new_tokens=int(max_tokens), + output_scores=bool(logprobs), + return_dict_in_generate=True, + pad_token_id=self._tok.pad_token_id or self._tok.eos_token_id) + if temperature and float(temperature) > 0: + gen_kwargs.update(do_sample=True, temperature=float(temperature)) + else: + gen_kwargs.update(do_sample=False) # greedy: deterministic verdicts + with self._lock, torch.inference_mode(): + out = self._model.generate(**inputs, **gen_kwargs) + new_tokens = out.sequences[0][inputs["input_ids"].shape[1]:] + text = self._tok.decode(new_tokens, skip_special_tokens=True) + content = None + if logprobs and getattr(out, "scores", None): + content = [NS(token=self._tok.decode([tid]), + logprob=torch.log_softmax(step[0].float(), dim=-1)[tid].item()) + for tid, step in zip(new_tokens.tolist(), out.scores)] + choice = NS(message=NS(content=text), + logprobs=NS(content=content) if content is not None else None) + return NS(choices=[choice]) + + +def local_judge_revision(model_name: str): + """Resolved snapshot commit of a locally cached judge model. Enters the + resume fingerprint for judge_provider=local: the model NAME alone does not + pin the weights the way an API model string does.""" + base = os.path.join( + os.path.expanduser(os.getenv("HF_HOME", "~/.cache/huggingface")), "hub", + "models--" + str(model_name).replace("/", "--"), "refs", "main") + try: + with open(base) as f: + return f.read().strip() + except OSError: + return None + + +_LOCAL_CLIENT_CACHE: Dict[str, "_LocalJudgeClient"] = {} + + +def _build_judge_client(args): + """Build the judge client: local (in-process open-weight model via the + framework's native transformers loading), openai, or azure_openai. + + Returns (client, model_or_deployment_name). + """ + provider = (args.judge_provider or "openai").strip().lower() + if provider == "local": + if not args.judge_model: + raise ValueError("`--judge_model` is required for the local judge.") + # Process-wide cache: stages are rebuilt per result file, and a + # multi-file scoring run must not reload the judge weights each time. + if args.judge_model not in _LOCAL_CLIENT_CACHE: + _LOCAL_CLIENT_CACHE[args.judge_model] = _LocalJudgeClient(args.judge_model) + return _LOCAL_CLIENT_CACHE[args.judge_model], args.judge_model + + try: + from openai import AzureOpenAI, OpenAI + except Exception as exc: # pragma: no cover + raise RuntimeError("openai package is required for API judge providers.") from exc + + if provider == "azure_openai": + key = os.getenv("AZURE_OPENAI_KEY") + endpoint = os.getenv("AZURE_OPENAI_ENDPOINT") + if not key or not endpoint: + raise ValueError("AZURE_OPENAI_KEY and AZURE_OPENAI_ENDPOINT are required for azure_openai judge.") + model = os.getenv("AZURE_OPENAI_DEPLOYNAME", args.judge_model) + client = AzureOpenAI( + api_key=key, + azure_endpoint=endpoint, + api_version=os.getenv("AZURE_OPENAI_API_VERSION", "2024-02-15-preview"), + ) + return client, model + + key = os.getenv("OPENAI_API_KEY") + base_url = os.getenv("OPENAI_BASE_URL") + if not key: + raise ValueError("OPENAI_API_KEY is required for openai judge.") + kwargs = {"api_key": key} + if base_url: + kwargs["base_url"] = base_url + return OpenAI(**kwargs), args.judge_model + + +def _resolve_setting(cli_value, env_name, default): + """Single source of truth for judge knobs: CLI flag > env var > default.""" + if cli_value is not None: + return int(cli_value) + return int(os.getenv(env_name, str(default))) + + +# HTTP statuses worth retrying: timeout/conflict/rate-limit and server errors. +_RETRYABLE_STATUS = {408, 409, 429} +_RETRYABLE_EXC_NAMES = {"APIConnectionError", "APITimeoutError", "RateLimitError", "InternalServerError"} + + +def _is_retryable(exc) -> bool: + status = getattr(exc, "status_code", None) + if isinstance(status, int): + return status in _RETRYABLE_STATUS or status >= 500 + return type(exc).__name__ in _RETRYABLE_EXC_NAMES + + +def _verdict_probability(logprobs_content, verdict) -> Any: + """P of the generated verdict digit, from API token logprobs. The judge reply + is tiny JSON ({"is_correct": 1, ...}) whose FIRST digit token is the verdict; + returns exp(logprob of that token), or None when logprobs are unavailable.""" + if not logprobs_content: + return None + target = str(int(verdict)) + for entry in logprobs_content: + token = str(getattr(entry, "token", "")).strip().strip('"') + if token == target: + lp = getattr(entry, "logprob", None) + return math.exp(lp) if lp is not None else None + if token and token.strip("01") == "" and token != target: + return None # first digit token disagrees with the parsed verdict — don't guess + return None + + +class _LLMStageBase(BaseStage): + uses_llm = True + """Shared client/config + a `_chat` helper with retry and reasoning-model + handling (gpt-5/o* use max_completion_tokens and reject temperature overrides). + + Concurrent LLM calls are capped by a process-wide semaphore (--judge_concurrency / + JUDGE_MAX_CONCURRENCY, default 4), DECOUPLED from the scorer's sample-worker count. + Firing 16+ judge calls at once over a full dataset saturates the API's tokens/min + quota; the gateway then returns degraded-but-valid responses (not errors), silently + tanking accuracy. Capping concurrency keeps sustained throughput under quota. + The semaphore is sized once per process by the first judge stage constructed.""" + + _sem = None + _sem_lock = threading.Lock() + + @classmethod + def _get_sem(cls, n): + if _LLMStageBase._sem is None: + with _LLMStageBase._sem_lock: + if _LLMStageBase._sem is None: + _LLMStageBase._sem = threading.Semaphore(max(1, n)) + return _LLMStageBase._sem + + def __init__(self, args): + self.max_retry = max(1, _resolve_setting(getattr(args, "judge_max_retry", None), "JUDGE_MAX_RETRY", 3)) + self.concurrency = _resolve_setting(getattr(args, "judge_concurrency", None), "JUDGE_MAX_CONCURRENCY", 4) + self.max_tokens = _resolve_setting(getattr(args, "judge_max_tokens", None), "JUDGE_MAX_TOKENS", 2048) + self.temperature = float(args.judge_temperature) + self.include_reason = bool(args.judge_include_reason) + self.client, self.model = _build_judge_client(args) + if not self.model: + raise ValueError("`--judge_model` is required when a judge stage is enabled.") + + # Process-wide: set to False after a provider rejects the logprobs parameter, + # so we don't pay a failed round-trip per sample on such deployments. + _logprobs_supported = True + + def _chat(self, messages, want_logprobs=False): + """Returns (text, error, logprobs): text is None on failure, error describes + the last problem so callers can surface it instead of silently scoring + 'incorrect'. logprobs is the response token-logprob list (or None) — purely + observational, requested only when want_logprobs and the provider allows it.""" + is_reasoning = bool(re.match(r"\s*(gpt-5|o1|o3|o4)", str(self.model), re.I)) + kwargs = {"model": self.model, "messages": messages} + if is_reasoning: + kwargs["max_completion_tokens"] = self.max_tokens + else: + kwargs["max_tokens"] = self.max_tokens + kwargs["temperature"] = self.temperature + use_logprobs = want_logprobs and _LLMStageBase._logprobs_supported + if use_logprobs: + kwargs["logprobs"] = True + sem = self._get_sem(self.concurrency) + last_error = "no_attempt" + for attempt in range(self.max_retry): + try: + with sem: # cap concurrent API calls; don't hold it during backoff + resp = self.client.chat.completions.create(**kwargs) + text = resp.choices[0].message.content or "" + if text.strip(): + lp = getattr(resp.choices[0], "logprobs", None) + content = getattr(lp, "content", None) if lp is not None else None + return text, None, content + last_error = "empty_response" + except Exception as exc: + last_error = f"{type(exc).__name__}: {exc}" + if not _is_retryable(exc): + if use_logprobs and getattr(exc, "status_code", None) == 400: + # 400 with logprobs requested: some gateways reject the + # parameter outright. Drop it (process-wide) and redo the + # call — the verdict matters more than the observability + # field. Auth/permission errors (401/403) fail fast below. + print(f"[{self.name}] provider rejected logprobs; disabling: {last_error}", flush=True) + _LLMStageBase._logprobs_supported = False + return self._chat(messages, want_logprobs=False) + # auth / bad request / not found — retrying cannot help + print(f"[{self.name}] non-retryable API error: {last_error}", flush=True) + return None, last_error, None + print(f"[{self.name}] attempt {attempt + 1}/{self.max_retry} failed: {last_error}", flush=True) + if attempt + 1 < self.max_retry: + time.sleep(1.5 * (2 ** attempt)) # 1.5s, 3s, 6s exponential backoff + return None, last_error, None + + +class LLMJudgeStage(_LLMStageBase): + """The LLM directly judges whether the response is correct given + QUESTION + MODEL_RESPONSE + GROUND_TRUTH, returning is_correct 0/1. + Both verdicts DECIDE the sample (score 1.0/0.0); only transport/parse + failures pass, carrying an llm_error marker.""" + + name = "llm-judge" + + def run(self, sample: Dict[str, Any], context: Dict[str, Any]) -> StageResult: + pred = normalize_text(context["pred"], extract_boxed=False, strip_latex_commands=False) + gt = normalize_text(context["gt"], extract_boxed=False, strip_latex_commands=False) + question = normalize_text(context["question"], extract_boxed=False, strip_latex_commands=False) + question_type = context["question_type"] + + prompt = self._build_prompt(question_type, question, pred, gt, context) + messages = [ + {"role": "system", "content": "You are a strict evaluator for VLM outputs."}, + {"role": "user", "content": prompt}, + ] + + text, error, logprobs = self._chat(messages, want_logprobs=True) + parsed = self._parse_response(text) if text else None + if parsed is None: + # Transport/parse failure, NOT a "judged incorrect" verdict — the reason + # marker keeps it distinguishable in score.json and in the summary. + detail = error or "unparseable_judge_response" + return StageResult.passed(reason=f"llm_error: {detail}") + + is_correct = int(parsed.get("is_correct", 0)) == 1 + # Observability only: P(verdict token) from API logprobs; never a + # model-self-reported number, never an input to the verdict. + prob = _verdict_probability(logprobs, parsed["is_correct"]) + meta = {"judge_prob": prob} + reason = parsed.get("reason") if self.include_reason else None + if is_correct: + return StageResult.scored(1.0, matched=str(context["gt"]), reason=reason, meta=meta) + return StageResult.scored(0.0, matched=None, reason=reason, meta=meta) + + def _build_prompt(self, question_type, question, pred, gt, context): + options = context.get("option_texts", {}) + options_text = "" + if question_type == "mcq" and options: + rendered = [f"{k}. {v}" for k, v in options.items()] + options_text = "\nOptions:\n" + "\n".join(rendered) + + reason_instruction = "Include a concise reason." if self.include_reason else "Set reason to an empty string." + return ( + "Judge whether MODEL_RESPONSE is correct given QUESTION and GROUND_TRUTH.\n" + f"QuestionType: {question_type}\n" + f"QUESTION: {question}\n" + f"{options_text}\n" + f"MODEL_RESPONSE: {pred}\n" + f"GROUND_TRUTH: {gt}\n\n" + "Return JSON only with schema:\n" + '{"is_correct": 0_or_1, "reason": "string"}\n' + f"{reason_instruction}\n" + "No markdown, no extra fields." + ) + + def _parse_response(self, text): + text = text.strip() + if not text: + return None + try: + return self._normalize_judge_obj(json.loads(text)) + except Exception: + pass + match = re.search(r"\{.*\}", text, flags=re.DOTALL) + if match: + try: + return self._normalize_judge_obj(json.loads(match.group(0))) + except Exception: + return None + return None + + @staticmethod + def _normalize_judge_obj(obj): + if not isinstance(obj, dict): + return None + raw = obj.get("is_correct", 0) + if isinstance(raw, bool): + value = 1 if raw else 0 + else: + try: + value = int(raw) + except Exception: + value = 0 + reason = obj.get("reason", "") + return {"is_correct": 1 if value == 1 else 0, "reason": str(reason)} + + +class LLMMatchStage(_LLMStageBase): + """VLMEvalKit's approach: the LLM only EXTRACTS which option(s) the response + picks, then we rule-based exact-compare against the ground truth. Mirrors + extract_answer_from_item for mcq (rule-based can_infer prefetch, GPT extraction + only as a fallback) and the LogicVista letters extractor for multi-letter gts + (vlmeval/dataset/utils/logicvista.py:15-21, sorted-letters compare :50-68).""" + + name = "llm-match" + + # Modeled on VLMEvalKit's LogicVista extraction instruction (logicvista.py:15-21). + EXTRACT_MULTI_TMPL = ( + 'You are an AI assistant who will help me to match an answer with several ' + 'options of a multiple-selection question. You are provided with a question, ' + 'several options, and an answer, and you need to output which option letters ' + 'the answer selects. Output ONLY the chosen uppercase letters separated by ' + 'commas (e.g. "A, C"). If the answer selects none of the options, output Z.\n' + 'Question: {}?\nOptions: {}\nAnswer: {}\nYour output: ' + ) + + # Verbatim from VLMEvalKit (vlmeval/dataset/utils/multiple_choice.py build_prompt). + EXTRACT_TMPL = ( + 'You are an AI assistant who will help me to match ' + 'an answer with several options of a single-choice question. ' + 'You are provided with a question, several options, and an answer, ' + 'and you need to find which option is most similar to the answer. ' + 'If the meaning of all options are significantly different from the answer, output Z. ' + 'Your should output a single uppercase character in A, B, C, D (if they are valid options), and Z. \n' + 'Example 1: \n' + 'Question: What is the main object in image?\nOptions: A. teddy bear B. rabbit C. cat D. dog\n' + 'Answer: a cute teddy bear\nYour output: A\n' + 'Example 2: \n' + 'Question: What is the main object in image?\nOptions: A. teddy bear B. rabbit C. cat D. dog\n' + 'Answer: Spider\nYour output: Z\n' + 'Example 3: \n' + 'Question: {}?\nOptions: {}\nAnswer: {}\nYour output: ' + ) + + def run(self, sample: Dict[str, Any], context: Dict[str, Any]) -> StageResult: + if context["question_type"] != "mcq": + return StageResult.passed(reason="llm-match supports mcq only") + + pred = normalize_text(context["pred"], extract_boxed=False, strip_latex_commands=False) + question = normalize_text(context["question"], extract_boxed=False, strip_latex_commands=False) + # Drop media placeholders from the text prompt. + question = re.sub(r"<(?:image|video)>", " ", question).strip() + choices = [str(c).strip().upper() for c in (context["options"] or [])] + option_texts = context.get("option_texts", {}) + + # Compact gts only with real parsed options (see exact.py: the A-F + # fallback would read word-like gts as letter sets). + gt_letters = parse_gt_letters(context["gt"]) + if gt_letters is None and context.get("options_parsed"): + gt_letters = extract_letter_set(context["gt"], choices) + if gt_letters is not None and len(gt_letters) > 1: + # Multi-letter gt (LogicVista): rule prefetch, then the letters-extraction + # prompt; official sorted-set compare (utils/logicvista.py:50-68). + got = extract_letter_set(pred, choices) + error = None + if got is None: + rendered = " ".join(f"{k}. {v}" for k, v in option_texts.items()) + prompt = self.EXTRACT_MULTI_TMPL.format(question, rendered, pred) + text, error, _ = self._chat([{"role": "user", "content": prompt}]) + if text: + got = extract_letter_set(text, choices) + if got and got == gt_letters: + return StageResult.scored(1.0, matched=", ".join(gt_letters)) + reason = f"llm_error: {error}" if error else None + return StageResult.passed(reason=reason) + + # Single-letter gt: VLMEvalKit extract_answer_from_item protocol. + gt_letter = (gt_letters[0] if gt_letters else None) or \ + extract_option_robust(context["gt"], choices, option_texts) or \ + normalize_for_exact(context["gt"]).upper() + + opt, error = self._extract_option(question, pred, choices, option_texts) + if opt and gt_letter and opt == gt_letter: + return StageResult.scored(1.0, matched=gt_letter) + reason = f"llm_error: {error}" if error else None + return StageResult.passed(reason=reason) + + def _extract_option(self, question, pred, choices, option_texts): + """Returns (option_or_None, api_error_or_None).""" + # Prefetch with rules (VLMEvalKit can_infer = can_infer_option | can_infer_text). + # Original case, matching VLMEvalKit — uppercasing first would turn every + # article "a" into a hit for option A. + opt = can_infer_option(str(pred), choices) or can_infer_text(str(pred), option_texts) + if opt: + return opt, None + # GPT extraction fallback. + rendered = [f"{k}. {v}" for k, v in option_texts.items()] if option_texts else [] + options_str = " ".join(rendered) + prompt = self.EXTRACT_TMPL.format(question, options_str, pred) + text, error, _ = self._chat([{"role": "user", "content": prompt}]) + if not text: + return None, error + valid = choices + ["Z"] + got = can_infer_option(text.upper(), valid) + if got: + return got, None + # The extractor replies with a bare letter; accept any valid candidate (not + # just A-D — benchmarks with options E/F+ must not fall off this cliff). + m = re.search(rf"\b([{re.escape(''.join(valid))}])\b", text.upper()) + return (m.group(1) if m else None), None + diff --git a/mmeval/scoring/stages/metrics.py b/mmeval/scoring/stages/metrics.py new file mode 100644 index 00000000..22024ebb --- /dev/null +++ b/mmeval/scoring/stages/metrics.py @@ -0,0 +1,126 @@ +"""Fractional grader stages. Both always DECIDE (score 0..1) or mark the +sample invalid — they never pass — so they are only valid as a pipeline's +last stage. is_correct = 1 only at full credit; the official headline +aggregate for fractional protocols is summary.mean_score. + +Protocol sources (implemented 1:1): +- vqa-accuracy: lmms-eval lmms_eval/tasks/vqav2/utils.py:15-42 + (vqav2_process_results) — whitespace-normalize, EvalAIAnswerProcessor + punctuation+digit/article normalization, then the official leave-one-out + consensus accuracy: for each reference answer, acc = min(1, #matching OTHER + references / 3); sample score = mean over references. +- anls: lmms-eval lmms_eval/api/metrics.py:293-321 (anls) — per + reference, normalized Levenshtein distance over lowercased + whitespace-collapsed strings; score = 1 - min distance, zeroed below the + threshold (default 0.5). +""" +from typing import Any, List, Optional, Tuple + +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.vqa_eval_metric import EvalAIAnswerProcessor + +_VQA_PROCESSOR = EvalAIAnswerProcessor() + + +def _reference_list(gt: Any) -> Optional[List[str]]: + if isinstance(gt, (list, tuple)) and gt: + return [str(x) for x in gt] + if isinstance(gt, str) and gt.strip(): + return [gt] + return None + + +def _vqa_normalize(text: str) -> str: + # lmms-eval vqav2_process_results: \n/\t -> space, strip, then the official + # EvalAI punctuation and digit/article passes. + text = text.replace("\n", " ").replace("\t", " ").strip() + text = _VQA_PROCESSOR.process_punctuation(text) + text = _VQA_PROCESSOR.process_digit_article(text) + return text + + +def vqa_consensus(pred: Any, gt: Any) -> Tuple[Optional[float], Optional[str]]: + """Official VQA consensus accuracy. Returns (score, error): score is None + on error. Requires the multi-annotator reference list (VQAv2/OK-VQA/ + TextVQA/VizWiz ship 10 answers) — the leave-one-out formula is undefined + for a single reference.""" + refs = _reference_list(gt) + if refs is None or len(refs) < 2: + return None, "vqa-accuracy requires the multi-annotator answer list (>=2 references) as gt" + pred_text = pred[0] if isinstance(pred, list) and pred else pred + if not isinstance(pred_text, str): + return None, "vqa-accuracy: prediction is not text" + res = _vqa_normalize(pred_text) + norm_refs = [_vqa_normalize(r) for r in refs] + accs = [] + for i in range(len(norm_refs)): + others = norm_refs[:i] + norm_refs[i + 1:] + matching = sum(1 for o in others if o == res) + accs.append(min(1.0, matching / 3.0)) + return sum(accs) / len(accs), None + + +def _levenshtein(s1: str, s2: str) -> int: + # lmms-eval api/metrics.py levenshtein_distance, verbatim algorithm. + if len(s1) > len(s2): + s1, s2 = s2, s1 + distances = range(len(s1) + 1) + for i2, c2 in enumerate(s2): + distances_ = [i2 + 1] + for i1, c1 in enumerate(s1): + if c1 == c2: + distances_.append(distances[i1]) + else: + distances_.append(1 + min((distances[i1], distances[i1 + 1], distances_[-1]))) + distances = distances_ + return distances[-1] + + +def anls(pred: Any, gt: Any, threshold: float = 0.5) -> Tuple[Optional[float], Optional[str]]: + """ANLS (lmms-eval api/metrics.py:293-321): 1 - min normalized Levenshtein + over the references, zeroed below `threshold`. Accepts a single reference + string or a reference list.""" + refs = _reference_list(gt) + if refs is None: + return None, "anls: empty ground truth" + pred_text = pred[0] if isinstance(pred, list) and pred else pred + if not isinstance(pred_text, str): + return None, "anls: prediction is not text" + det = " ".join(pred_text.strip().lower().split()) + values = [] + for answer in refs: + gt_answer = " ".join(answer.strip().lower().split()) + dist = _levenshtein(gt_answer, det) + length = max(len(answer.upper()), len(pred_text.upper())) + values.append(0.0 if length == 0 else float(dist) / float(length)) + score = 1 - min(values) + if score < threshold: + score = 0.0 + return score, None + + +class VqaAccuracyStage(BaseStage): + """Official VQA consensus accuracy (fractional grader).""" + + name = "vqa-accuracy" + grader = True + + def run(self, sample, context): + score, error = vqa_consensus(context["pred"], context["gt"]) + if score is None: + return StageResult.not_gradable(error) + return StageResult.scored(score) + + +class AnlsStage(BaseStage): + """ANLS similarity against the references (fractional grader).""" + + name = "anls" + grader = True + + def run(self, sample, context): + score, error = anls(context["pred"], context["gt"], + threshold=float(context.get("anls_threshold") or 0.5)) + if score is None: + return StageResult.not_gradable(error) + return StageResult.scored(score) diff --git a/mmeval/scoring/stages/rule_match.py b/mmeval/scoring/stages/rule_match.py new file mode 100644 index 00000000..a3ff7135 --- /dev/null +++ b/mmeval/scoring/stages/rule_match.py @@ -0,0 +1,117 @@ +import re + +from mmeval.scoring.stages.base import BaseStage, StageResult +from mmeval.scoring.stages.exact_match import (gt_references, numbers_match, + numeric_branch_active, open_text_match) +from mmeval.scoring.extraction import ( + extract_letter_set, + extract_option_robust, + extract_yes_no, + normalize_for_exact, + normalize_open_answer, + normalize_text, + parse_gt_letters, + parse_number, +) + +# Answer cues that introduce the model's final value in a chain of thought. +_NUMERIC_CUE = re.compile(r"(?:final answer|the answer is|answer is|answer\s*[:\-]|=)") +# Optional trailing % so extracted percentages keep the official /100 +# semantics (see extraction.parse_number). +_NUMBER_RE = re.compile(r"-?\d+(?:\.\d+)?%?") +_OPEN_ANSWER_RES = ( + re.compile(r"(?:final answer|answer is|the answer is)\s*[:\-]?\s*(.+)$", re.IGNORECASE), + re.compile(r"^\s*(.+?)\s*$", re.IGNORECASE), +) + + +class RuleMatchStage(BaseStage): + """Robust rule-based answer extraction: per-type extraction tuned for + verbose / chain-of-thought output, then deterministic comparison. Still + fully rule-based — no API calls.""" + + name = "rule-match" + + def run(self, sample, context): + if context["gt"] is None or context["pred"] is None: + return StageResult.passed() + + handler = { + "mcq": self._match_mcq, + "yes_no": self._match_yes_no, + "numeric": self._match_numeric, + }.get(context["question_type"], self._match_open) + return handler(context) + + def _match_mcq(self, context): + # Letter-SET grading (single-select = |set|=1; LogicVista multi-letter + # answers compare sorted sets, vlmeval/dataset/utils/logicvista.py:50-68). + pred_raw = context["pred"] + gt_raw = context["gt"] + candidates = context["options"] + option_texts = context.get("option_texts", {}) + + # Compact multi-letter gts ("AC") are accepted at match time (the row + # is already typed mcq) — but only with REAL parsed options: under the + # A-F fallback, word-like gts would parse as letter sets ("BED" -> B,D,E). + gt_letters = parse_gt_letters(gt_raw) + if gt_letters is None and context.get("options_parsed"): + gt_letters = extract_letter_set(gt_raw, candidates) + if gt_letters is None: + g = extract_option_robust(gt_raw, candidates, option_texts) or normalize_for_exact(gt_raw).upper() + gt_letters = (g,) if g else None + + # Clean letters-and-separators segments first ("A, C", "BD", + # "the answers are A and C"); then the robust single-letter extractor + # (VLMEvalKit can_infer + our patches: tags, cues, CJK, \boxed, + # option-text restatement). + pred_letters = extract_letter_set(pred_raw, candidates) + if pred_letters is None: + option = extract_option_robust(pred_raw, candidates, option_texts) + pred_letters = (option,) if option else None + + if gt_letters and pred_letters and pred_letters == gt_letters: + return StageResult.scored(1.0, matched=", ".join(gt_letters)) + return StageResult.passed() + + def _match_yes_no(self, context): + # VLMEvalKit YOrN_Extraction parity (vlmeval/dataset/utils/yorn.py:254-261): + # word-level yes XOR no; a response containing both defers to the next + # stage (upstream marks it 'Unknown'). + gt_norm = normalize_open_answer(context["gt"]) + pred_yn = extract_yes_no(context["pred"]) + if pred_yn is not None and pred_yn == gt_norm: + return StageResult.scored(1.0, matched=pred_yn) + return StageResult.passed() + + def _match_numeric(self, context): + if not numeric_branch_active(context): + # e.g. OCRBench: the declared non-exact string comparator is the + # official rule for digit-string answers too (see exact_match.py). + return self._match_open(context) + gt_nums = [n for n in (parse_number(r) for r in gt_references(context["gt"])) if n is not None] + if not gt_nums: + return StageResult.passed() + pred_norm = normalize_open_answer(context["pred"]) + # Prefer the segment after the LAST answer cue (a CoT usually walks + # through intermediate numbers first); fall back to the whole text. + cues = list(_NUMERIC_CUE.finditer(pred_norm)) + segment = pred_norm[cues[-1].end():] if cues else pred_norm + nums = _NUMBER_RE.findall(segment) or _NUMBER_RE.findall(pred_norm) + pred_num = parse_number(nums[0]) if nums else None + if pred_num is not None and any(numbers_match(pred_num, g, context) for g in gt_nums): + return StageResult.scored(1.0, matched=str(context["gt"])) + return StageResult.passed() + + def _match_open(self, context): + pred_norm_text = normalize_text(context["pred"]) + + # generic "final answer is ..." extraction, then whole-string restatement; + # each candidate compared per the protocol's string_match (exact/contains/anls). + for pattern in _OPEN_ANSWER_RES: + m = pattern.search(pred_norm_text) + if not m: + continue + if open_text_match(m.group(1), m.group(1), context["gt"], context): + return StageResult.scored(1.0, matched=str(context["gt"])) + return StageResult.passed() diff --git a/mmeval/scoring/vqa_eval_metric.py b/mmeval/scoring/vqa_eval_metric.py new file mode 100644 index 00000000..a885657a --- /dev/null +++ b/mmeval/scoring/vqa_eval_metric.py @@ -0,0 +1,218 @@ +# Verbatim copy of lmms-eval's EvalAIAnswerProcessor +# (lmms_eval/tasks/_task_utils/vqa_eval_metric.py @ EvolvingLMMs-Lab/lmms-eval, +# itself copied from facebookresearch/mmf pythia/tasks/processors.py#L897) — +# the OFFICIAL VQA-accuracy answer normalization. Do not edit: protocol fidelity +# requires the exact contraction/digit/punctuation tables. +import re + + +class EvalAIAnswerProcessor: + """ + Processes an answer similar to Eval AI + copied from + https://github.com/facebookresearch/mmf/blob/c46b3b3391275b4181567db80943473a89ab98ab/pythia/tasks/processors.py#L897 + """ + + CONTRACTIONS = { + "aint": "ain't", + "arent": "aren't", + "cant": "can't", + "couldve": "could've", + "couldnt": "couldn't", + "couldn'tve": "couldn't've", + "couldnt've": "couldn't've", + "didnt": "didn't", + "doesnt": "doesn't", + "dont": "don't", + "hadnt": "hadn't", + "hadnt've": "hadn't've", + "hadn'tve": "hadn't've", + "hasnt": "hasn't", + "havent": "haven't", + "hed": "he'd", + "hed've": "he'd've", + "he'dve": "he'd've", + "hes": "he's", + "howd": "how'd", + "howll": "how'll", + "hows": "how's", + "Id've": "I'd've", + "I'dve": "I'd've", + "Im": "I'm", + "Ive": "I've", + "isnt": "isn't", + "itd": "it'd", + "itd've": "it'd've", + "it'dve": "it'd've", + "itll": "it'll", + "let's": "let's", + "maam": "ma'am", + "mightnt": "mightn't", + "mightnt've": "mightn't've", + "mightn'tve": "mightn't've", + "mightve": "might've", + "mustnt": "mustn't", + "mustve": "must've", + "neednt": "needn't", + "notve": "not've", + "oclock": "o'clock", + "oughtnt": "oughtn't", + "ow's'at": "'ow's'at", + "'ows'at": "'ow's'at", + "'ow'sat": "'ow's'at", + "shant": "shan't", + "shed've": "she'd've", + "she'dve": "she'd've", + "she's": "she's", + "shouldve": "should've", + "shouldnt": "shouldn't", + "shouldnt've": "shouldn't've", + "shouldn'tve": "shouldn't've", + "somebody'd": "somebodyd", + "somebodyd've": "somebody'd've", + "somebody'dve": "somebody'd've", + "somebodyll": "somebody'll", + "somebodys": "somebody's", + "someoned": "someone'd", + "someoned've": "someone'd've", + "someone'dve": "someone'd've", + "someonell": "someone'll", + "someones": "someone's", + "somethingd": "something'd", + "somethingd've": "something'd've", + "something'dve": "something'd've", + "somethingll": "something'll", + "thats": "that's", + "thered": "there'd", + "thered've": "there'd've", + "there'dve": "there'd've", + "therere": "there're", + "theres": "there's", + "theyd": "they'd", + "theyd've": "they'd've", + "they'dve": "they'd've", + "theyll": "they'll", + "theyre": "they're", + "theyve": "they've", + "twas": "'twas", + "wasnt": "wasn't", + "wed've": "we'd've", + "we'dve": "we'd've", + "weve": "we've", + "werent": "weren't", + "whatll": "what'll", + "whatre": "what're", + "whats": "what's", + "whatve": "what've", + "whens": "when's", + "whered": "where'd", + "wheres": "where's", + "whereve": "where've", + "whod": "who'd", + "whod've": "who'd've", + "who'dve": "who'd've", + "wholl": "who'll", + "whos": "who's", + "whove": "who've", + "whyll": "why'll", + "whyre": "why're", + "whys": "why's", + "wont": "won't", + "wouldve": "would've", + "wouldnt": "wouldn't", + "wouldnt've": "wouldn't've", + "wouldn'tve": "wouldn't've", + "yall": "y'all", + "yall'll": "y'all'll", + "y'allll": "y'all'll", + "yall'd've": "y'all'd've", + "y'alld've": "y'all'd've", + "y'all'dve": "y'all'd've", + "youd": "you'd", + "youd've": "you'd've", + "you'dve": "you'd've", + "youll": "you'll", + "youre": "you're", + "youve": "you've", + } + + NUMBER_MAP = { + "none": "0", + "zero": "0", + "one": "1", + "two": "2", + "three": "3", + "four": "4", + "five": "5", + "six": "6", + "seven": "7", + "eight": "8", + "nine": "9", + "ten": "10", + } + ARTICLES = ["a", "an", "the"] + PERIOD_STRIP = re.compile(r"(?!<=\d)(\.)(?!\d)") + COMMA_STRIP = re.compile(r"(?<=\d)(\,)+(?=\d)") + PUNCTUATIONS = [ + ";", + r"/", + "[", + "]", + '"', + "{", + "}", + "(", + ")", + "=", + "+", + "\\", + "_", + "-", + ">", + "<", + "@", + "`", + ",", + "?", + "!", + ] + + def __init__(self, *args, **kwargs): + pass + + def word_tokenize(self, word): + word = word.lower() + word = word.replace(",", "").replace("?", "").replace("'s", " 's") + return word.strip() + + def process_punctuation(self, in_text): + out_text = in_text + for p in self.PUNCTUATIONS: + if (p + " " in in_text or " " + p in in_text) or (re.search(self.COMMA_STRIP, in_text) is not None): + out_text = out_text.replace(p, "") + else: + out_text = out_text.replace(p, " ") + out_text = self.PERIOD_STRIP.sub("", out_text, re.UNICODE) + return out_text + + def process_digit_article(self, in_text): + out_text = [] + temp_text = in_text.lower().split() + for word in temp_text: + word = self.NUMBER_MAP.setdefault(word, word) + if word not in self.ARTICLES: + out_text.append(word) + else: + pass + for word_id, word in enumerate(out_text): + if word in self.CONTRACTIONS: + out_text[word_id] = self.CONTRACTIONS[word] + out_text = " ".join(out_text) + return out_text + + def __call__(self, item): + item = self.word_tokenize(item) + item = item.replace("\n", " ").replace("\t", " ").strip() + item = self.process_punctuation(item) + item = self.process_digit_article(item) + return item diff --git a/mmeval/utils/argparser.py b/mmeval/utils/argparser.py index d0f83a54..2781ecee 100644 --- a/mmeval/utils/argparser.py +++ b/mmeval/utils/argparser.py @@ -1,5 +1,7 @@ import argparse import inspect +import os +import sys import warnings from dataclasses import dataclass, field, fields from typing import Dict, Optional, Sequence, get_args @@ -7,6 +9,7 @@ @dataclass class ModelArguments: model_name_or_path: Optional[str] = field(default=None) + model_series: Optional[str] = field(default=None) # parameters for model low_cpu_mem_usage: Optional[bool] = field(default=None, metadata={"help": "Tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model."}) @@ -14,6 +17,8 @@ class ModelArguments: # parameters for model inference dtype: Optional[str] = field(default=None, metadata={"help": "Override the default torch.dtype and load the model under a specific dtype."}) + min_pixels: Optional[int] = field(default=None, metadata={"help": "Image processor min_pixels override (Qwen-VL family); None keeps the runner's default."}) + max_pixels: Optional[int] = field(default=None, metadata={"help": "Image processor max_pixels override (Qwen-VL family); None keeps the runner's default."}) device_map: Optional[str] = field(default=None, metadata={"help": "A map that specifies where each submodule should go."}) # parameters that control the length of the output @@ -84,7 +89,48 @@ class ExperimentArguments: no_conda: bool = field(default=False, metadata={"help": "use current python env instead of conda"}) -ARGUMENT_DATACLASSES = (ModelArguments, DataArguments, InferenceArguments, ExperimentArguments) +@dataclass +class ScoreRuntimeArguments: + # Score-mode CLI convention: every dest is prefixed `score_` (scoring + # pipeline/IO/knobs) or `judge_` (the LLM-judge client group — already + # score-mode-only and self-descriptive; `score_judge_*` would stutter). + # No unprefixed or infer-colliding dests. + score_out_dir: Optional[str] = field(default=None, metadata={"help": "output directory containing result.json files"}) + score_parallel_per_task: int = field(default=4, metadata={"help": "number of sample workers inside each result.json"}) + + +@dataclass +class ScoreArguments: + score_result_glob: str = field(default="**/result.json", metadata={"help": "glob pattern for result files in score mode"}) + score_output_name: str = field(default="score.json", metadata={"help": "output score json file name"}) + score_progress_bar: bool = field(default=True, metadata={"help": "show progress bar while scoring samples"}) + score_resume: bool = field(default=True, metadata={"help": "resume scoring from score tmp/final files when available"}) + score_save_freq: int = field(default=20, metadata={"help": "flush frequency for score resume tmp file"}) + score_debug: bool = field(default=False, metadata={"help": "write per-stage trace into score outputs"}) + score_pipeline: str = field(default="exact-match,rule-match", metadata={"help": "ordered scoring pipeline: comma-separated atomic stage names (exact-match, rule-match, llm-match, llm-judge, vqa-accuracy, anls)"}) + + score_gt_field: str = field(default="answer", metadata={"help": "field name for ground-truth in result sample"}) + score_pred_field: str = field(default="messages[-1].response", metadata={"help": "field path for prediction in result sample"}) + score_question_type: str = field(default="auto", metadata={"help": "question type: auto|mcq|yes_no|numeric|open. auto = read the sample's question_type field (recognized values only), then infer from gt shape"}) + score_numeric_rel_tol: float = field(default=0.0, metadata={"help": "relative tolerance for numeric answers (0 = exact float equality)"}) + score_numeric_abs_tol: float = field(default=0.0, metadata={"help": "absolute tolerance for numeric answers (0 = off; DynaMath uses 0.001)"}) + score_string_match: str = field(default="exact", metadata={"help": "rule-chain text comparison: exact|contains|anls (contains = OCRBench substring protocol; anls = threshold ANLS)"}) + score_anls_threshold: float = field(default=0.5, metadata={"help": "ANLS threshold (string_match=anls and the anls grader)"}) + + judge_provider: Optional[str] = field(default=None, metadata={"help": "llm judge provider: local (in-process open-weight model, framework-native loading) | openai | azure_openai"}) + judge_model: Optional[str] = field(default=None, metadata={"help": "llm judge model/deployment name"}) + # For the three knobs below, resolution order is: CLI flag > env var > default + # (env names JUDGE_MAX_RETRY / JUDGE_MAX_CONCURRENCY / JUDGE_MAX_TOKENS). + judge_max_retry: Optional[int] = field(default=None, metadata={"help": "llm judge max attempts per call (default: env JUDGE_MAX_RETRY or 3)"}) + judge_temperature: float = field(default=0.0, metadata={"help": "llm judge generation temperature"}) + judge_concurrency: Optional[int] = field(default=None, metadata={"help": "max concurrent llm judge calls, process-wide (default: env JUDGE_MAX_CONCURRENCY or 4)"}) + judge_max_tokens: Optional[int] = field(default=None, metadata={"help": "llm judge max completion tokens (default: env JUDGE_MAX_TOKENS or 2048)"}) + judge_include_reason: bool = field(default=False, metadata={"help": "include llm judge reason in output"}) + + +ARGUMENT_DATACLASSES = (ModelArguments, DataArguments, InferenceArguments, ExperimentArguments, ScoreArguments) +INFER_ARGUMENT_DATACLASSES = (ModelArguments, DataArguments, InferenceArguments, ExperimentArguments) +SCORE_ARGUMENT_DATACLASSES = (ScoreRuntimeArguments, ScoreArguments) BOOL_DEFAULTS = { @@ -100,9 +146,8 @@ class ExperimentArguments: ) } -def parse_args(): - parser = argparse.ArgumentParser() - for dc in ARGUMENT_DATACLASSES: +def _add_dataclass_arguments(parser, dataclasses, suppress_defaults=False): + for dc in dataclasses: for f in fields(dc): tp = f.type type_args = getattr(tp, '__args__', None) @@ -114,23 +159,57 @@ def parse_args(): if f.default is True: parser.add_argument(f"--no-{cli_name}", f"--no_{f.name}", dest=f.name, action="store_false", - default=True, help=help_text) + default=argparse.SUPPRESS if suppress_defaults else True, + help=help_text) elif f.default is False: parser.add_argument(f"--{cli_name}", f"--{f.name}", dest=f.name, action="store_true", - default=False, help=help_text) + default=argparse.SUPPRESS if suppress_defaults else False, + help=help_text) else: parser.add_argument(f"--{cli_name}", f"--{f.name}", dest=f.name, action="store_true", - default=None, help=help_text) + default=argparse.SUPPRESS if suppress_defaults else None, + help=help_text) parser.add_argument(f"--no-{cli_name}", f"--no_{f.name}", - dest=f.name, action="store_false") + dest=f.name, action="store_false", + **({"default": argparse.SUPPRESS} if suppress_defaults else {})) else: parser.add_argument(f"--{cli_name}", f"--{f.name}", dest=f.name, type=tp, - default=f.default, help=help_text) - return parser.parse_args() + default=argparse.SUPPRESS if suppress_defaults else f.default, + help=help_text) + + +def _track_explicit_flags(args, argv): + """Record which score-mode dests the user explicitly provided (shadow parse + with suppressed defaults). The scorer's per-knob resolution needs this: + explicit CLI flag > dataset_meta from result.json > built-in default.""" + shadow = argparse.ArgumentParser() + _add_dataclass_arguments(shadow, SCORE_ARGUMENT_DATACLASSES, suppress_defaults=True) + args._explicit_flags = frozenset(vars(shadow.parse_known_args(argv)[0]).keys()) + return args + +def parse_args(mode: str = "auto"): + mode = (mode or "auto").strip().lower() + if mode == "auto": + script_name = os.path.basename(sys.argv[0]) + mode = "score" if script_name == "score.py" else "infer" + + if mode == "infer": + dataclasses = INFER_ARGUMENT_DATACLASSES + elif mode == "score": + dataclasses = SCORE_ARGUMENT_DATACLASSES + else: + raise ValueError(f"Unsupported parse mode: {mode}. Expected infer|score|auto.") + + parser = argparse.ArgumentParser() + _add_dataclass_arguments(parser, dataclasses) + args = parser.parse_args() + if mode == "score": + args = _track_explicit_flags(args, sys.argv[1:]) + return args def parse_model_kwargs(args, default_kwargs=None): default_kwargs = default_kwargs or {} model_kwargs = {} diff --git a/scripts/examples/hf_dataset.sh b/scripts/examples/hf_dataset.sh index bdcbc341..649cd238 100644 --- a/scripts/examples/hf_dataset.sh +++ b/scripts/examples/hf_dataset.sh @@ -1,9 +1,37 @@ export PYTHONPATH=./:$PYTHONPATH +# +# The committed outputs under work_dirs/examples/hf_dataset/ are exactly what +# this script produces: a deterministic 20-sample subset of the MMBench dev +# split (dev carries answers; the test split withholds them), selected with +# the framework's native sampling flags — see docs/en/USAGE.md "Run a subset +# of samples". python mmeval/run.py \ --model_name_or_path Qwen/Qwen3-VL-2B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ - --split test \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ + --split dev \ + --sample_num 20 \ + --sample_order random \ + --sample_seed 42 \ --out_dir work_dirs/examples/hf_dataset/MMBench_en_V11 \ --gpu_per_parallel 1 \ - --parallel_per_task 1 \ No newline at end of file + --parallel_per_task 1 + +# Score the run. The explicit rule pipeline keeps this credential-free +# (MMBench's declared pipeline includes an LLM judge stage, which needs API +# keys); the override is recorded in score.json's knob_sources. +python mmeval/score.py \ + --score_out_dir work_dirs/examples/hf_dataset/MMBench_en_V11 \ + --score_result_glob 'result.json' \ + --score_pipeline exact-match,rule-match \ + --no_score_resume + +# Fixture gate: every sample must be gradable — a fixture whose rows are all +# invalid would exercise nothing but the missing-gt path. +python3 - <<'PY' +import json +s = json.load(open("work_dirs/examples/hf_dataset/MMBench_en_V11/score.json"))["summary"] +assert s["invalid"] == 0, f"fixture has {s['invalid']} invalid samples" +print(f"fixture OK: {s['total']} samples, invalid=0, accuracy={s['accuracy']:.2f}") +PY diff --git a/scripts/examples/llm_judge_api_reference.md b/scripts/examples/llm_judge_api_reference.md new file mode 100644 index 00000000..6f69a1fb --- /dev/null +++ b/scripts/examples/llm_judge_api_reference.md @@ -0,0 +1,44 @@ +# LLM-as-Judge API Reference (simple-mmeval scoring) + +LLM judge used for `template,llm-match` scoring of MCQ / open-ended benchmarks. +Provider: **Azure OpenAI**, accessed via ByteDance internal modelhub proxy. + +## Environment variables + +```bash +# === Azure OpenAI judge === +export AZURE_OPENAI_KEY="" # never commit real keys +export AZURE_OPENAI_ENDPOINT="" +export AZURE_OPENAI_DEPLOYNAME="gpt-5.4-mini-2026-03-17" +export AZURE_OPENAI_API_VERSION="2024-02-01" + +# === call-side concurrency / retry control === +export JUDGE_MAX_CONCURRENCY=4 # cap concurrency to avoid quota saturation / degradation +export JUDGE_MAX_RETRY=3 +export JUDGE_MAX_TOKENS=2048 +``` + +## Scoring command + +```bash +python mmeval/score.py --out_dir --score_result_glob 'result.json' \ + --matching_order template,llm-match \ + --judge_provider azure_openai \ + --judge_model gpt-5.4-mini-2026-03-17 \ + --score_output_name score_llm.json \ + --parallel_per_task 16 --no_score_resume +``` + +## Notes + +- **judge model**: `gpt-5.4-mini-2026-03-17` (Azure deployment name is identical). +- **matching_order = `template,llm-match`**: try rule/template answer-matching first; + fall back to the LLM judge only when the rule match fails. Fewer calls, more stable. +- **provider = `azure_openai`**; the endpoint may be a proxy rather than public Azure — + set AZURE_OPENAI_ENDPOINT to whatever your deployment exposes. +- `JUDGE_MAX_CONCURRENCY=4` keeps concurrency low to avoid saturating the judge quota + (which otherwise degrades throughput/quality). + +> Set the key/endpoint via environment variables; never commit real values. The +> reusable part is the *methodology*: rule-first + LLM fallback, the judge model, +> and the concurrency/retry settings. diff --git a/scripts/examples/prompt_template.sh b/scripts/examples/prompt_template.sh index 8da791da..11afcce8 100644 --- a/scripts/examples/prompt_template.sh +++ b/scripts/examples/prompt_template.sh @@ -1,4 +1,13 @@ export PYTHONPATH=./:$PYTHONPATH +# +# NOTE on committed example outputs: the result.json/score.json checked into +# work_dirs/examples/ are the outputs of these commands as written — the HF +# example (Test 4) selects a deterministic 20-sample subset of the answered +# dev split with the framework's native sampling flags +# (--sample_num/--sample_order/--sample_seed, see docs/en/USAGE.md "Run a +# subset of samples"), scores it, and asserts every sample is gradable. +# Score the local-JSON outputs (Tests 1-2) with: +# PYTHONPATH=. python3 mmeval/score.py --score_out_dir --no_score_resume # Test 1: Local JSON with template file path python mmeval/run.py \ @@ -34,13 +43,32 @@ python mmeval/run.py \ --gpu_per_parallel 1 \ --parallel_per_task 1 -# Test 4: HuggingFace MMBench-en-V11 +# Test 4: HuggingFace MMBench-V11 (en subset), 20-sample subset via native +# sampling. The dev split carries answers (test withholds them), so the +# committed fixture exercises real grading. python mmeval/run.py \ --model_name_or_path Qwen/Qwen3-VL-2B-Instruct \ - --dataset mmeval_hf@mm-eval/MMBench-en-V11 \ - --split test \ + --dataset mmeval_hf@mm-eval/MMBench-V11 \ + --subset en \ + --split dev \ + --sample_num 20 \ + --sample_order random \ + --sample_seed 42 \ --out_dir work_dirs/examples/prompt_template/hf_template \ --gpu_per_parallel 1 \ --parallel_per_task 1 +# Score Test 4 and gate the fixture: every sample must be gradable. +python mmeval/score.py \ + --score_out_dir work_dirs/examples/prompt_template/hf_template \ + --score_result_glob 'result.json' \ + --score_pipeline exact-match,rule-match \ + --no_score_resume +python3 - <<'PY' +import json +s = json.load(open("work_dirs/examples/prompt_template/hf_template/score.json"))["summary"] +assert s["invalid"] == 0, f"fixture has {s['invalid']} invalid samples" +print(f"fixture OK: {s['total']} samples, invalid=0, accuracy={s['accuracy']:.2f}") +PY + diff --git a/scripts/examples/run_score_evalkit_llm.sh b/scripts/examples/run_score_evalkit_llm.sh new file mode 100755 index 00000000..d56a634a --- /dev/null +++ b/scripts/examples/run_score_evalkit_llm.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash + +set -euo pipefail + +# Example script: run scoring (exact -> template -> llm-judge) on evalkit outputs. +# Defaults are intentionally conservative for cost/time. +# +# Usage: +# OPENAI_API_KEY=xxx bash scripts/examples/run_score_evalkit_llm.sh +# +# Optional env vars: +# OUT_DIR=work_dirs/evalkit_all_qwen +# SCORE_RESULT_GLOB='VStarBench/result.json' # or '*/result.json' +# SCORE_OUTPUT_NAME=score_llm.json +# PARALLEL_PER_TASK=1 # sample workers inside each result.json +# PIPELINE='exact-match,rule-match,llm-judge' +# JUDGE_PROVIDER='openai' +# JUDGE_MODEL='gpt-5' +# JUDGE_INCLUDE_REASON='false' +# SCORE_RESUME='true' +# SCORE_SAVE_FREQ=20 + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")"/../.. && pwd)" +cd "${ROOT_DIR}" + +export PYTHONPATH="./:${PYTHONPATH:-}" + +OUT_DIR="${OUT_DIR:-work_dirs/evalkit_all_qwen}" +SCORE_RESULT_GLOB="${SCORE_RESULT_GLOB:-VStarBench/result.json}" +SCORE_OUTPUT_NAME="${SCORE_OUTPUT_NAME:-score_llm.json}" +PARALLEL_PER_TASK="${PARALLEL_PER_TASK:-1}" +PIPELINE="${PIPELINE:-exact-match,rule-match,llm-judge}" +JUDGE_PROVIDER="${JUDGE_PROVIDER:-openai}" +JUDGE_MODEL="${JUDGE_MODEL:-gpt-5}" +JUDGE_INCLUDE_REASON="${JUDGE_INCLUDE_REASON:-false}" +SCORE_RESUME="${SCORE_RESUME:-true}" +SCORE_SAVE_FREQ="${SCORE_SAVE_FREQ:-20}" + +if [[ "${MATCHING_ORDER}" == *"llm"* ]]; then + if [[ -z "${OPENAI_API_KEY:-}" && "${JUDGE_PROVIDER}" == "openai" ]]; then + echo "ERROR: OPENAI_API_KEY is required when using llm matcher with openai provider." + exit 1 + fi +fi + +echo "Scoring config:" +echo " OUT_DIR=${OUT_DIR}" +echo " SCORE_RESULT_GLOB=${SCORE_RESULT_GLOB}" +echo " SCORE_OUTPUT_NAME=${SCORE_OUTPUT_NAME}" +echo " PARALLEL_PER_TASK=${PARALLEL_PER_TASK} (sample workers per result.json)" +echo " MATCHING_ORDER=${MATCHING_ORDER}" +echo " JUDGE_PROVIDER=${JUDGE_PROVIDER}" +echo " JUDGE_MODEL=${JUDGE_MODEL}" +echo " JUDGE_INCLUDE_REASON=${JUDGE_INCLUDE_REASON}" +echo " SCORE_RESUME=${SCORE_RESUME}" +echo " SCORE_SAVE_FREQ=${SCORE_SAVE_FREQ}" +echo + +CMD=( + python mmeval/score.py + --score_out_dir "${OUT_DIR}" + --score_result_glob "${SCORE_RESULT_GLOB}" + --parallel_per_task "${PARALLEL_PER_TASK}" + --score_pipeline "${PIPELINE}" + --score_output_name "${SCORE_OUTPUT_NAME}" + --judge_provider "${JUDGE_PROVIDER}" + --judge_model "${JUDGE_MODEL}" + --score_save_freq "${SCORE_SAVE_FREQ}" +) + +if [[ "${JUDGE_INCLUDE_REASON}" == "true" ]]; then + CMD+=(--judge_include_reason) +fi + +if [[ "${SCORE_RESUME}" == "false" ]]; then + CMD+=(--no_score_resume) +fi + +"${CMD[@]}" diff --git a/scripts/templates/vlmevalkit_mcq.txt b/scripts/templates/vlmevalkit_mcq.txt new file mode 100644 index 00000000..0e2f142a --- /dev/null +++ b/scripts/templates/vlmevalkit_mcq.txt @@ -0,0 +1,5 @@ +{{ question }}{% if options %} +Options: +{% for k, v in options.items() %}{{ k }}. {{ v }} +{% endfor %}{% endif %} +Please select the correct answer from the options above. \ No newline at end of file diff --git a/work_dirs/examples/evalkit_dataset/LLaVABench/score.json b/work_dirs/examples/evalkit_dataset/LLaVABench/score.json new file mode 100644 index 00000000..9d93dfd6 --- /dev/null +++ b/work_dirs/examples/evalkit_dataset/LLaVABench/score.json @@ -0,0 +1,905 @@ +{ + "summary": { + "total": 60, + "valid": 0, + "correct": 0, + "accuracy": 0.0, + "accuracy_valid": 0.0, + "mean_score": 0.0, + "invalid": 60, + "llm_errors": 0, + "by_stage": { + "undecided": { + "total": 60, + "correct": 0 + } + }, + "by_question_type": { + "mcq": { + "total": 60, + "correct": 0 + } + } + }, + "config": { + "score_pipeline": [ + "exact-match", + "rule-match" + ], + "score_gt_field": "answer", + "score_pred_field": "messages[-1].response", + "score_question_type": "auto", + "score_numeric_rel_tol": 0.0, + "score_numeric_abs_tol": 0.0, + "score_string_match": "exact", + "score_anls_threshold": 0.5, + "task_type": null, + "official_protocol": null, + "knob_sources": { + "score_pipeline": "default", + "score_numeric_rel_tol": "default", + "score_numeric_abs_tol": "default", + "score_string_match": "default", + "score_anls_threshold": "default" + }, + "judge_provider": null, + "judge_model": null, + "judge_temperature": 0.0, + "judge_include_reason": false, + "score_progress_bar": true, + "score_resume": false, + "score_save_freq": 20, + "score_parallel_per_task": 1, + "resumed_count": 0, + "provenance": { + "git_commit": null, + "python": "3.13.13", + "openai": "2.44.0" + }, + "protocol_notes": [ + "Per-sample scoring; official aggregations noted per dataset in these notes are not reproduced. LLM failures are marked wrong and counted in summary.llm_errors (never randomly guessed, unlike VLMEvalKit)." + ] + }, + "samples": [ + { + "eval-id": 0, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is the famous landmark known as the Diamond Head, a volcanic crater located in Honolulu, Hawaii. It is a popular tourist attraction and a key feature of the city's skyline." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 1, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is an aerial photograph capturing a dramatic and scenic coastal landscape, likely taken from a high vantage point on a mountain or cliff overlooking the ocean. The image is rich in detail and color, showcasing the contrast between the rugged landmass and the vibrant sea.\n\nThe central focus is a large, steep, and rugged mountain that dominates the mid-ground. Its surface is a mix of dark, earthy tones\u2014browns, tans, and dark reds\u2014indicating a heavily eroded volcanic rock. The mountain has a distinct, steep-sided crater or bowl shape, with a large, prominent crater that appears to be the result" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 2, + "question_type": "mcq", + "gt": null, + "pred": [ + "The image you've provided shows the iconic **Dolphin's Back**, a natural landform located in the **Hawaiian Islands**. This is a classic example of a **volcanic crater** formed by the collapse of a volcano.\n\nHere are the possible reasons for the formation of this sight:\n\n---\n\n### 1. **Volcanic Origin**\n- The most likely cause is that this is a **volcanic crater** formed during the eruption of a volcano.\n- The area is part of the **Hawaiian Islands**, which are formed by the slow movement of the Pacific Plate over a hotspot (a region of" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 3, + "question_type": "mcq", + "gt": null, + "pred": [ + "### **A Journey to the Heart of Hawaii: Exploring the Majestic Haleakal\u0101 and the Cultural Tapestry of Maui**\n\n**By [Your Name] | May 2024**\n\nIf you're a fan of breathtaking landscapes, vibrant culture, and a truly unforgettable experience, then your dream destination might just be the island of Maui, Hawaii. This is not just a vacation; it's a journey into a world where nature and tradition harmoniously coexist. I had the privilege of visiting Maui this past spring, and the experience was nothing short of magical. From the awe-inspiring Haleakal\u0101 volcano to the" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 4, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the visual characteristics in the image, this is a **mangosteen**.\n\nHere are the key features that identify it:\n\n- **Appearance:** The fruit has a deep, dark purple or maroon skin.\n- **Stem:** It has a distinctive green, leaf-like cap at the top, which is the fruit's calyx.\n- **Internal Structure:** The fruit is cut in half, revealing a very white, fleshy, and star-shaped interior. This is the edible flesh, which is known for its sweet and slightly tangy taste.\n- **Texture:** The flesh is described as juicy and soft," + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 5, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, we can count the uncut fruits by identifying the whole, intact fruits.\n\n- There are three whole mangosteens with their green tops still attached. They are located in the upper left, middle, and lower right portions of the image.\n- There is one cut mangosteen in the foreground, showing its white, star-shaped interior. This is not an uncut fruit.\n- There is one cut mangosteen on the right side, also showing its white interior. This is also not an uncut fruit.\n- The other fruit is the one in the lower right, which is cut in half and" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 6, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a detailed, close-up photograph of several mangosteens (also known as mangosteens or mangosteens) arranged on a rustic wooden surface. The image is composed to highlight the fruit's unique and appealing characteristics.\n\nThe central focus is on the mangosteens, which are shown in various states of presentation:\n- **Whole fruits:** Several whole mangosteens are visible, each with a deep, dark purple to almost black skin. The skin is smooth and has a slight sheen, and the top of each fruit features a distinctive, leaf-like crown. This crown is composed of a cluster of green, fleshy" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 7, + "question_type": "mcq", + "gt": null, + "pred": [ + "The scent of the mangosteen is a delightful and unique blend of tropical freshness and a hint of subtle sweetness, which is truly unforgettable. If you were to describe it to someone who has never had this fruit before, you might say:\n\n\"Imagine a crisp, clean aroma of fresh citrus and sweet, tropical fruit, like a blend of lemon and lime, but with a hint of a slightly sweet, almost floral note. It\u2019s not overpowering, but rather delicate and refreshing, like the scent of a warm, sunlit orchard. This fragrance is so distinct and pleasant that it\u2019s almost like a burst of sunshine on a warm" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 8, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a digitally altered image that is a parody of the famous Mona Lisa, but with a dog instead of a woman. The image is a portrait of a long-haired, brown dog, likely a dachshund or a similar breed, dressed in a manner that mimics the Renaissance-era fashion of the Mona Lisa. The dog is depicted with a human-like expression, with large, dark, and expressive eyes that look directly at the viewer. Its fur is a rich brown color, and it has a long, flowing coat with a slightly shaggy texture.\n\nThe dog is wearing a dark, draped garment that resembles the famous Mona" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 9, + "question_type": "mcq", + "gt": null, + "pred": [ + "This painting is a humorous and creative reinterpretation of the iconic \"Mona Lisa,\" but with a canine subject. The intended effect is to create a playful, absurd, and entertaining piece of art that combines the classic elements of the original painting with a modern, unexpected twist.\n\nHere are the key aspects of the intended effect:\n\n- **Absurdity and Satire**: By replacing the human figure with a dog, the artist subverts the traditional narrative of the Mona Lisa. The dog's expression, though it appears to be a \"smile,\" is more like a neutral, slightly sad or confused look, which is a perfect contrast to" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 10, + "question_type": "mcq", + "gt": null, + "pred": [ + "This image presents a striking and humorous reinterpretation of Leonardo da Vinci's *Mona Lisa*, but its meaning is likely to be interpreted differently by various audiences based on their perspectives, cultural backgrounds, and personal experiences.\n\nFor many, especially those familiar with the original artwork, the image is a playful and affectionate parody. The central figure, a dog with a human-like expression and a classic pose, is a clear homage to the Mona Lisa's enigmatic smile and the artistic style of the Renaissance. The dog's gaze, with its soft, slightly melancholic eyes, mirrors the Mona Lisa's subtle, enigmatic smile. This twist is" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 11, + "question_type": "mcq", + "gt": null, + "pred": [ + "The man in the photo is Elon Musk, the CEO of Tesla, SpaceX, and other companies." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 12, + "question_type": "mcq", + "gt": null, + "pred": [ + "This meme is a parody of the iconic scene from the 1999 Disney animated film *The Lion King*.\n\nThe scene depicted is where the character Simba, a young lion, is on the Pride Rock, looking up at the sky and holding a small, golden lion cub (often referred to as \"the cub\") in his arms. The cub is the same one that is being held by the character, which is a reference to the \"Doge\" meme, a popular internet meme featuring a cartoon dog with a human face.\n\nThe meme is a clever blend of two different cultural references: the *The Lion King*" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 13, + "question_type": "mcq", + "gt": null, + "pred": [ + "This meme reflects a blend of satire, internet culture, and commentary on Elon Musk's public image, personality, and actions through the juxtaposition of his public persona with a fictional, comedic character.\n\nHere's a breakdown of how it comments on him:\n\n- **Satire of the \"Doge\" Culture**: The central figure is a dog with the face of a Doge, a popular internet meme that combines the image of a dog with the face of a meme character. The meme uses this to satirize Musk's often exaggerated public persona. The Doge face is a symbol of internet humor, and its use here suggests Musk's" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 14, + "question_type": "mcq", + "gt": null, + "pred": [ + "Of course. Here is a detailed explanation of the meme.\n\nThis image is a classic internet meme that uses a simple, relatable scene to convey a common feeling of exhaustion or dread. It is a well-known \"just...monday\" meme, popularized on platforms like Reddit and Imgflip.\n\n### The Visual Elements\n\nThe image is composed of a photograph and text overlaid on it.\n\n- **The Subject:** The central figure is a small, light-brown or cream-colored French Bulldog (also known as a Frenchie). The dog is lying on its back on a wooden deck. Its body is relaxed and it appears" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 15, + "question_type": "mcq", + "gt": null, + "pred": [ + "The meme \"Monday. Just... Monday.\" expresses the common sentiment of feeling overwhelmed, stressed, or drained by the start of the week. Here are several other ways someone might express the same sentiment:\n\n- **\"I'm not even sure I'm awake anymore. It's like I've been in a time loop.\"** This conveys a sense of mental fatigue and disorientation, similar to the dog's state of exhaustion.\n\n- **\"This week is just a blur of tasks and deadlines. I'm not even sure what I'm supposed to do.\"** This expresses the feeling of being overwhelmed by the volume of responsibilities.\n\n- **" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 16, + "question_type": "mcq", + "gt": null, + "pred": [ + "Yes, the painting you're referring to is *The Mona Lisa* by Leonardo da Vinci. It is one of the most famous artworks in the world, created in the early 16th century." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 17, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a detailed description of the painting, which is a famous work by the Italian Renaissance artist Leonardo da Vinci.\n\nThe painting is a portrait of a woman, commonly known as the Mona Lisa, painted in the early 1500s. The artwork is renowned for its enigmatic expression, the subtle use of light and shadow, and its innovative use of perspective.\n\n**Subject and Composition:**\nThe central figure is a woman with long, dark brown hair that falls over her shoulders and frames her face. Her gaze is directed slightly to the right of the viewer, creating a sense of engagement and mystery. Her facial expression is one" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 18, + "question_type": "mcq", + "gt": null, + "pred": [ + "The painting you've shown is *The Mona Lisa*, created by Leonardo da Vinci in the early 1500s. It is one of the most famous and influential works in the history of art. Here's a discussion of its historical impact and significance.\n\n### Historical Impact\n\n1. **Revolution in Artistic Technique**:\n - *The Mona Lisa* is a masterclass in the use of sfumato, a technique that involves the delicate blending of tones to create a soft, almost invisible transition between colors and shadows. This technique allowed da Vinci to create a sense of depth and realism that was groundbreaking at the time" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 19, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a detailed photograph of the iconic Seattle Space Needle, a famous landmark in Seattle, Washington. The image is taken from a low angle, looking up towards the top of the structure, which emphasizes its height and architectural design.\n\nThe Space Needle is the central focus of the image. It is a white, modern skyscraper with a distinctive saucer-shaped top. The top section is a wide, circular platform with a flat, white surface. A red, white, and blue American flag is flying from the top of the structure. The main body of the needle is a tall, slender, white tower with three vertical support legs, which" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 20, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, here is an analysis of the best weather, season, time of day, and whether the photo was taken at a good time to visit the Space Needle.\n\n### Weather and Season\nThe weather appears to be **sunny and pleasant**. The sky is a clear, bright blue with no clouds, and the lighting is bright and even, suggesting a sunny day. This type of weather is ideal for outdoor activities and sightseeing.\n\nThe season is likely **late spring or summer**. This is inferred from the lush green foliage on the trees and the overall vibrant appearance of the city. While the photo was taken" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 21, + "question_type": "mcq", + "gt": null, + "pred": [ + "The character in the image is Conan Edogawa, a fictional detective from the anime and manga series *Detective Conan* (also known as *Case Closed*). He is known for his sharp intellect and keen observational skills, often solving complex mysteries." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 22, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, the character is **Kaito Kid**, a central figure in the anime and manga series *Case Closed* (also known as *Detective Conan*). The image shows him in his signature attire, which is a key element of his character design.\n\nHere is an analysis of his personality and the design elements that contributed to his popularity:\n\n### Personality\n\nKaito Kid is a complex and multifaceted character whose personality is defined by a combination of traits:\n\n- **Intense Focus and Determination**: Kaito is driven by a singular purpose: to solve crimes and bring justice to the world." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 23, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, this appears to be a serene and scenic location, likely a lake or a calm body of water in a mountainous or forested region. While the image itself is beautiful and peaceful, it's important to be cautious about the potential risks and considerations when visiting such a place. Here are some things to be mindful of:\n\n---\n\n### 1. **Weather and Environmental Conditions**\n- **Extreme Weather:** The landscape is surrounded by mountains and forests, which can be exposed to sudden weather changes. Even on a seemingly calm day, there could be a sudden storm, heavy rain, or snowfall. The sky in" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 24, + "question_type": "mcq", + "gt": null, + "pred": [ + "If I were a photographer aiming to capture the essence of this location, I would choose **early morning on a calm, overcast day**. Here\u2019s why:\n\n- **Time of Day: Early Morning (around 6:30 AM)**\n - The soft, diffused light of early morning is ideal for this scene. It avoids the harsh shadows and intense sunlight of midday, creating a gentle, even illumination that enhances the natural colors and textures of the landscape. The light is also less likely to create glare on the water, allowing for a clearer view of the wooden dock and the surrounding environment.\n - The air" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 25, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a candid, humorous photograph taken on a city street, likely in New York City, capturing a man in an unconventional and amusing situation.\n\nThe central focus is a man with a bald head and glasses, wearing a bright yellow long-sleeved shirt. He is standing on the rear bumper of a yellow SUV, which appears to be a Ford Explorer or similar model, with a silver lower bumper. The vehicle is parked on a street, and the man is actively ironing a blue piece of clothing, which looks like a shirt or a towel, onto a small, makeshift ironing board. This board is constructed from a metal folding" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 26, + "question_type": "mcq", + "gt": null, + "pred": [ + "The image is unusual because it depicts a man ironing clothes on the back of a yellow taxi, which is parked on the street. This is not a typical scene of someone ironing at home or in a laundry room, but rather an unexpected and unconventional situation. The man is using a portable ironing board, which is placed on the back of the taxi, and is ironing a blue shirt. The yellow taxi is parked on the street, and there are other vehicles and buildings in the background. The image is likely a humorous or ironic depiction of a man who has taken his ironing to the streets, possibly as a form of" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 27, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, the fruit in the left part of the refrigerator is **strawberries**.\n\nThey are located in a clear plastic container on the lower left side of the fridge, next to a bag of carrots. The strawberries are red and appear to be fresh." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 28, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, the brand of the blueberry-flavored yogurt is **Fay-eh!**.\n\nThis is visible on the label of the white tub of yogurt in the refrigerator, which is labeled \"GRADE A\" and \"Fay-eh!\"." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 29, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on a careful examination of the image provided, there is no strawberry-flavored yogurt in the fridge.\n\nHere is a breakdown of the contents visible in the refrigerator:\n\n- **In the foreground:** A clear plastic container holds several strawberries.\n- **On the right side:** There is a container of yogurt. The label is clearly visible and reads:\n - **\"All Natural\"**\n - **\"Tropical Blueberry\"**\n - **\"Green Stewed Yogurt\"**\n - The flavor is explicitly **blueberry**, not strawberry.\n- **In the background:** There is a large" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 30, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a detailed description of the image:\n\nThe photograph captures the interior of a refrigerator, focusing on the shelves filled with various food items. The scene is well-lit, likely by the refrigerator's internal lighting, which highlights the contents.\n\n- **Left Side:** The leftmost shelf holds a clear plastic container filled with fresh strawberries. The strawberries are a mix of red and green, and they are arranged in a neat, circular pattern. Above the container of strawberries, there is a colorful bag of baby carrots, which are orange and wrapped in a purple and yellow plastic bag. The bag is slightly crumpled.\n\n- **Center:**" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 31, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the items visible in the refrigerator, you have a variety of ingredients that can be used to cook a range of meals. Here's a breakdown of what you can make with these ingredients:\n\n### **Protein & Carbohydrates**\n- **Protein:** This is a strong foundation for many meals.\n - **Chicken or Turkey:** You can use the chicken or turkey from the frozen package (likely a frozen chicken breast or turkey cutlet) to make a simple stir-fry, a chicken and vegetable stew, or a chicken and vegetable bake.\n - **Tuna or Salmon:** The tuna is a great source of protein" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 32, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, we can count the coffee mugs.\n\n- There are three mugs visible in the image, arranged in a row.\n- The mug on the far left is partially cut off by the frame.\n- The mug in the middle is the most clearly focused.\n- The mug on the right is also clearly visible.\n\nTherefore, there are **3** coffee mugs in the set." + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 33, + "question_type": "mcq", + "gt": null, + "pred": [ + "### **The Ultimate Mario Mug: A Celebration of Fun and Flavor**\n\nStep into the world of endless adventures with the **Mario Mug**\u2014a vibrant, high-quality ceramic mug that brings the magic of the Mushroom Kingdom to your daily coffee ritual! Designed for the ultimate gaming and coffee lovers, this mug is more than just a drinkware item\u2014it's a piece of fun, a nostalgic tribute, and a stylish addition to any kitchen or office.\n\n---\n\n### **Why the Mario Mug Stands Out**\n\n- **Iconic Design, Realistic Detail**: Every mug features a stunning, high-resolution print of Mario in his classic red cap and blue" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 34, + "question_type": "mcq", + "gt": null, + "pred": [ + "Of course! Based on the image provided, this is a delicious and visually appealing dish of baked lobster tails. Here is a detailed recipe for this dish.\n\n---\n\n### **Baked Lemon Herb Lobster Tails**\n\nThis recipe is a simple, elegant way to prepare lobster tails, highlighting their rich flavor with a bright, zesty lemon and fresh herbs. The dish is best enjoyed as a main course, but it's also delicious as a side dish or a hearty appetizer.\n\n#### **Ingredients**\n\n- **For the lobster tails:**\n - 1 (12-14 oz) lobster tail, preferably from a fresh," + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 35, + "question_type": "mcq", + "gt": null, + "pred": [ + "Of course! This meme is a clever and humorous take on a common internet joke, using a visual metaphor to create a funny and relatable scenario.\n\nHere is a detailed breakdown of the meme:\n\n### The Visual: The \"World\" of Fried Chicken\n- The image shows a baking sheet with several pieces of breaded and fried chicken, likely chicken nuggets or nugget-style chicken.\n- The pieces are arranged on the sheet to form a recognizable silhouette of the world map.\n- The \"countries\" are formed by the shapes of the chicken pieces:\n - The **United States** is represented by the two large, elongated pieces" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 36, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the image provided, the two machine learning concepts mentioned are:\n\n- **Statistical Learning**: This is the top section of the meme, where a stick figure is explaining that people are overgeneralizing because the VC-Dimension of the kernel is too high. This is a concept from the theory of statistical learning, which deals with the generalization ability of models.\n\n- **Neural Networks**: This is the bottom section of the meme, where a stick figure is advocating for \"stack more layers\" to solve the problem. Neural networks are a type of machine learning model that are based on artificial neurons, and stacking more layers is a" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 37, + "question_type": "mcq", + "gt": null, + "pred": [ + "This meme is a satirical and humorous take on the challenges and misconceptions in machine learning, specifically comparing two different approaches to model complexity and generalization.\n\nThe meme is composed of two panels, each with a stick figure representing a different approach to machine learning, and a chart or diagram illustrating the underlying concept.\n\n---\n\n### Panel 1: The \"Statistical Learning\" Approach\n\n- **Title:** `STATISTICAL LEARNING`\n- **Figure:** A stick figure with an angry expression, representing a traditional machine learning approach.\n- **Text in speech bubble:** \n - \"People, our learner overgeneralizes because the VC-D" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 38, + "question_type": "mcq", + "gt": null, + "pred": [ + "Of course. This comic is funny because it uses a clever, layered joke about the history and evolution of machine learning, playing on the contrast between two different approaches to solving problems. Let's break it down step by step.\n\n### Step 1: The Setup - The Two Approaches\n\nThe comic presents two distinct methods for solving a problem, which are presented as \"statistical learning\" and \"neural networks.\"\n\n- **Top Panel: Statistical Learning**\n - The speaker is a stick figure with a serious, frustrated expression.\n - The topic is \"statistical learning,\" which is a well-known field in machine learning.\n " + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 39, + "question_type": "mcq", + "gt": null, + "pred": [ + "This image is a collage of three photographs that document the use of a specific electronic adapter. The overall theme is the adaptation of a modern device (a smartphone) to an older, legacy system (a VGA monitor or projector). The collage is presented in a way that highlights the unusual and somewhat humorous nature of this connection.\n\nHere is a detailed description of each panel:\n\n---\n\n- **Panel 1 (Left):** This is the largest panel, showing a close-up of a smartphone with a black, protective case. The phone is positioned at an angle, with its charging port facing the viewer. A white, flexible cable is plugged into the" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 40, + "question_type": "mcq", + "gt": null, + "pred": [ + "Of course, here is a detailed description of the funny aspects of the image, broken down panel by panel.\n\nThe image is a composite of three photos that together tell a humorous story about a specific, and very awkward, electronic accessory.\n\n---\n\n### Panel 1 (Left, Main Image)\nThis panel shows a smartphone, which appears to be a modern model with a black screen and a protective case, connected to a **blue VGA (or DVI) cable**. The cable is plugged into the phone's charging port. The cable's plug is a standard, bulky blue VGA connector, which is designed for connecting a computer to a monitor" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 41, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the visual characteristics of the creature, it appears to be made of a combination of materials, primarily designed to look like a 3D printed or sculpted object.\n\nHere's a breakdown of the materials and textures visible:\n\n- **The Body and Fur:** The main body of the creature is a vibrant, fiery red-orange color. The surface has a smooth, glossy finish, which suggests a material like **plastic resin** or **acrylic**. The way the light reflects off it gives it a polished, almost metallic sheen, which is characteristic of these materials. The texture is also smooth, but it's not perfectly" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 42, + "question_type": "mcq", + "gt": null, + "pred": [ + "The image you've provided is not the logo of LLaVA (Large Language and Vision Assistant), but rather a stylized, cartoonish figurine that appears to be a digital creation, possibly an AI-generated character. The character is a fiery, red creature with a flamboyant, cartoonish design. It has large, expressive eyes, a pair of red-rimmed glasses, and is surrounded by flames, giving it a dynamic, energetic appearance. The creature stands on two legs, with a textured, rocky base that suggests it might be a creature from a fantasy or sci-fi world. The overall design is whimsical and imaginative" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 43, + "question_type": "mcq", + "gt": null, + "pred": [ + "Based on the painting, the animals are a bear, a cat, and a rabbit. They are gathered around a tree stump that serves as a small table, enjoying a meal together.\n\nHere is a breakdown of what each animal is doing:\n\n- **The Bear**: On the left, the bear is sitting and holding a small plate of cookies in its paws. It is also holding a small cup, possibly a teacup, to its mouth. It appears to be eating or drinking from the cup.\n- **The Cat**: In the center, the cat is sitting upright, looking at the bear and the rabbit. It" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 44, + "question_type": "mcq", + "gt": null, + "pred": [ + "### The Tale of the Forest's Last Tea\n\nIn the heart of a lush, sun-dappled forest, where the trees whispered secrets to the wind and the leaves rustled like secrets, lived a trio of unlikely friends: a wise bear, a curious cat, and a gentle rabbit. They were not the usual forest dwellers, but they were the guardians of the forest's quiet magic. The bear, named Thistle, was a stout guardian of the forest's ancient roots, while the cat, named Marmot, was a silent observer of the world, and the rabbit, named Pippin, was the quietest" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 45, + "question_type": "mcq", + "gt": null, + "pred": [ + "This is a hand-drawn, minimalist sketch of a website layout, likely for a personal or humorous website. The entire drawing is contained within a rectangular frame, mimicking the appearance of a page in a notebook.\n\nThe sketch is organized with a clear structure, using a hand-drawn line to separate the title from the content.\n\n**Top Section: The Title**\n- The title, \"My Joke website,\" is written in a casual, handwritten script.\n- The text is centered at the top of the page.\n- The title is enclosed within a horizontal line, which is drawn with a single, thick line across the top of the" + ], + "decided_by": null, + "matched": null, + "score": 0.0, + "is_correct": 0, + "status": "invalid", + "reason": "missing_gt_or_pred" + }, + { + "eval-id": 46, + "question_type": "mcq", + "gt": null, + "pred": [ + "```html\n\n\n\n \n \n My Joke Website\n