Openlist-Ani Metadata Evaluation

This repository contains offline evaluation assets for Openlist-Ani metadata parsing.

It is intentionally separate from the business repository so large manual gold datasets, exploratory silver datasets, collection scripts, and evaluation reports do not increase the size or churn of TwooSix/Openlist-Ani.

Contents

• data/metadata_parser/manual_gold_labeled.jsonl: manually reviewed gold labels.
• data/metadata_parser/title_visible_silver_labeled.jsonl: exploratory silver labels.
• data/metadata_parser/mikan_bangumi_raw_titles.jsonl: raw Mikan title samples.
• data/metadata_parser/reports/: historical evaluation and review reports.
• tools/metadata_parser/: collection, labeling, and evaluation scripts.
• tests/: tests for the evaluation scripts and large gold accuracy gate.

Run Against A Local Openlist-Ani Checkout

The evaluation code imports parser implementations from Openlist-Ani. Clone this repository next to Openlist-Ani, or set OPENLIST_ANI_REPO_PATH explicitly.

OPENLIST_ANI_REPO_PATH=/path/to/Openlist-Ani uv run pytest -q

If OPENLIST_ANI_REPO_PATH is not set, tests look for ../Openlist-Ani.

Scope

These tests evaluate the title extraction flow:

resource title -> ReleaseTitleParseResult

TMDB lookup and title canonicalization are evaluated separately by tools/metadata_parser/evaluate_regex_tmdb_against_library.py.