Seo geo prerender

.claude/agents/parser-fixture-writer.md

@@ -0,0 +1,75 @@
+---
+name: parser-fixture-writer
+description: >-
+  Write fixture-based regression tests for one open-wine-map parser
+  (FR cahier extractor, ES national-pliego parser, a country's
+  national-spec parser, fiche_technique, etc.). Use during Phase 5 of the
+  improvements plan, or after any parser regression ("add fixture tests
+  for the RO parser", "lock in the uppercase-title guard with a test").
+  Produces tests/fixtures/<cc>_*.txt|.html + tests/test_<cc>_parser.py.
+  The agent reads cached raw/ documents to harvest real excerpts but must
+  redact them to minimal sections and never commit anything from raw/
+  itself.
+tools: Read, Grep, Glob, Bash, Write
+---
+
+You write regression tests for ONE parser at a time in open-wine-map, a
+pipeline that extracts wine-appellation facts from public regulator
+documents. Parsers here are regex/keyword-routing functions that have
+historically regressed when a tweak for country N broke country M — your
+tests exist to make that impossible.
+
+## Inputs you are given
+
+- The parser module path (e.g. `scripts/_lib/es/national_pliego.py`,
+  `scripts/02_extract_cahiers.py`, `scripts/_lib/fiche_technique.py`).
+- Optionally: known-tricky cases (a slug, a git commit that fixed a bug,
+  a CURATOR_TODO note).
+
+## Method
+
+1. **Read the parser.** List its public entry points and the template /
+   heading variants it claims to handle (docstrings and comment tables
+   enumerate them — e.g. national_pliego.py names the JCCM / INCAVI /
+   AGACAL / ITACyL… heading variants; 02_extract_cahiers handles
+   uppercase-title guards, ordinal/bullet grape prefixes, cidre
+   `1) DENOMINATION` variants).
+2. **Mine git history for past regressions:**
+   `git log --oneline -- <parser-path>` — every "fix"-flavoured commit is
+   a mandatory test case.
+3. **Harvest fixtures from raw/ caches** (these exist locally but are
+   gitignored): find a document exercising each variant, extract the
+   MINIMAL section the parser routes on (target < 200 lines), and save as
+   `tests/fixtures/<cc>_<variant>_<slug>.txt` (pdftotext output) or
+   `.html` (OJ pages). These are excerpts of public, licence-clear
+   regulator documents — fine to commit. If raw/ lacks a document for a
+   variant, synthesize a minimal fixture from the parser's own expected
+   shape and mark it `# synthetic` in a header comment.
+4. **Write `tests/test_<cc>_parser.py`.** Import the parser the same way
+   existing tests do (see tests/test_content_block.py:
+   `sys.path.insert(0, …/scripts)` then import). Assert on:
+   - section routing (the right text lands in the right semantic role),
+   - grape parsing: principal vs accessory split, threshold percentages,
+     prefix stripping,
+   - style/colour detection where the parser does it,
+   - one regression test per mined bug-fix commit, named
+     `test_regression_<short-description>`.
+   Keep assertions on STRUCTURE (keys, counts, specific slugs), not on
+   full-output snapshots — snapshots rot.
+5. **Run:**
+   `uv run python -m pytest tests/test_<cc>_parser.py -v` → all pass, and
+   the full suite `uv run python -m pytest -q` stays green.
+   `uv run ruff check tests/` → clean.
+
+## Hard rules
+
+- Never commit whole documents or anything under `raw/`; fixtures are
+  short redacted excerpts only.
+- Never modify the parser to make a test pass. If the parser's actual
+  behaviour disagrees with what its docs/comments claim, write the test
+  against ACTUAL behaviour and flag the discrepancy in your final report.
+- No network access needed or allowed — everything comes from the local
+  raw/ cache or is synthetic.
+- One country/parser per run; report at the end: fixtures added, cases
+  covered, variants NOT covered (missing raw documents), discrepancies
+  found.

.claude/skills/ruff-zero/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: ruff-zero
+description: >-
+  Drive open-wine-map's ruff error count to zero WITHOUT changing behaviour.
+  Use when executing Phase 1 of the improvements plan, or whenever
+  `uv run ruff check scripts/ tests/` reports errors. Covers the safe
+  procedure for each rule class present in this repo (F401, F541, F601,
+  E741, E702, E731, F841, E402), with a mandatory special procedure for
+  F601 duplicate keys in scripts/_lib/grape_lexicon.py — those are DATA,
+  not style, and blind deletion can silently change grape-synonym folding.
+---
+
+# ruff-zero
+
+Goal state: `uv run ruff check scripts/ tests/` → `All checks passed!` and
+`uv run python -m pytest -q` → all pass, after EVERY commit.
+
+## Order of operations
+
+1. `uv run ruff check scripts/ tests/ --fix` (auto-fixes F401 unused
+   imports, F541 empty f-strings). Run pytest. Commit:
+   `fix: ruff auto-fixes (unused imports, empty f-strings)`
+2. F601 in `scripts/_lib/grape_lexicon.py` — see special procedure below.
+   Own commit.
+3. Remaining classes, ONE RULE CLASS PER COMMIT, pytest between each:
+   - **E741** (`l`, `I`, `O` as names): rename to something contextual
+     (`l` → `line`, `lst`, `layer`, `lon`…). Rename ONLY within the
+     function scope shown by ruff; use exact-match find within that
+     function, never file-wide sed.
+   - **E702** (semicolon-joined statements): split onto separate lines,
+     same indentation.
+   - **E731** (lambda assignment): convert to a `def` with the same name.
+   - **F841** (unused variable): if the right-hand side has side effects
+     (function call), keep the call and drop the assignment; if pure,
+     delete the line.
+   - **E402** (import not at top): only move the import if nothing between
+     file top and the import mutates `sys.path` or env that the import
+     needs. In this repo several scripts do
+     `sys.path.insert(0, …/scripts)` BEFORE importing `_lib` — those
+     imports must stay put; silence with `# noqa: E402` instead.
+
+## F601 special procedure (grape_lexicon.py)
+
+Each finding = the same dict key literal appears twice. Python keeps the
+LAST one. This file maps grape-name slugs to canonical slugs — a wrong
+deletion changes which VIVC identity a grape folds into.
+
+Per finding:
+
+1. Find BOTH occurrences: `grep -n '"<key>"' scripts/_lib/grape_lexicon.py`
+2. Compare the two VALUES.
+   - **Identical values** → delete the LATER occurrence (the one ruff
+     points at). If the later line carries a more informative comment
+     (e.g. a VIVC number), move that comment to the surviving line.
+   - **Different values** → DO NOT TOUCH. Append the key + both
+     line numbers + both values to a report list and continue. These are
+     real data conflicts requiring VIVC research (hand off to the
+     `grape-colour-researcher` agent or a human).
+3. After all findings:
+   ```
+   uv run ruff check scripts/_lib/grape_lexicon.py --select F601
+   uv run python -m pytest -q        # tests/test_no_duplicate_keys.py guards semantics
+   ```
+4. Commit: `fix: dedupe repeated grape_lexicon keys (identical-value F601s)`
+   — include the skipped-conflict report (if any) in the commit body.
+
+## Freezing at zero
+
+After all classes are clean, add to `pyproject.toml`:
+
+```toml
+[tool.ruff.lint]
+select = ["E", "F", "W"]
+```
+
+Run the check again — adding `W` may surface new findings; fix them the
+same way (one class per commit). Do NOT add rule families beyond E/F/W
+without being asked.
+
+## Pitfalls
+
+- Never run file-wide search-replace for renames; ruff gives exact
+  line/col — edit surgically.
+- `scripts/04_build_maps.py` and `scripts/_lib/map_template.py` are
+  5.9k/4k lines; load only the relevant region (read_file with offset),
+  not the whole file.
+- If pytest fails after a change, `git checkout -- <file>` and redo that
+  single finding; do not stack fixes on a broken tree.
+- `done2.json` / `todo.json` in the worktree are translation round-trip
+  artifacts — never commit them.

.claude/skills/stage04-extraction/SKILL.md

@@ -0,0 +1,99 @@
+---
+name: stage04-extraction
+description: >-
+  Move-only refactor procedure for carving functions out of
+  scripts/04_build_maps.py into scripts/_lib/ modules without changing any
+  build output. Use when executing Phase 6 of the improvements plan
+  ("extract augmenters", "modularize stage 04") or any time a function
+  must move out of 04_build_maps.py / map_template.py. The danger this
+  skill defends against: stage 04 failures are often SILENT (a country or
+  feature just disappears from the map), so verification relies on the
+  golden-output comparator, not on "it didn't crash".
+---
+
+# stage04-extraction
+
+## Iron rules
+
+1. **Move, never edit.** The function body is copied verbatim. No renames,
+   no cleanups, no type-hint additions, no f-string fixes — those are
+   separate commits OUTSIDE this skill.
+2. **One module per commit.**
+3. **The build output is the test.** A move is proven correct only by
+   `scripts/compare_build_output.py` reporting `identical` after a full
+   stage-04 rebuild (Task "final proof" below). py_compile + pytest are
+   intermediate gates only.
+
+## Prerequisite
+
+A golden snapshot must exist (plan Phase 0):
+`tmp/golden-before/` populated from the CURRENT `wiki/` output, and
+`scripts/compare_build_output.py` present. If missing, create them first —
+never extract without a golden snapshot.
+
+## Procedure (per module)
+
+1. **Map the block.** Identify the function(s) to move and EVERYTHING they
+   reference: module-level constants, helper functions, imports. Use:
+   ```
+   grep -n "<name>" scripts/04_build_maps.py
+   ```
+   for every symbol used inside the body. A symbol used by BOTH the moved
+   block and remaining code stays in a shared location (move it to the new
+   module and import it back, or leave it and import it into the new
+   module — prefer whichever produces fewer cross-imports).
+2. **Create the module** under `scripts/_lib/` (e.g.
+   `scripts/_lib/augment/es.py`). Top of file: only the imports the moved
+   code actually needs. Note: `_lib` modules use relative imports
+   (`from .grape_lexicon import …`) — follow the pattern of existing
+   `_lib` files; check one (e.g. `scripts/_lib/lieu_dit.py`) first.
+3. **Replace in 04_build_maps.py** the moved block with an import near the
+   other `_lib` imports:
+   ```python
+   from _lib.augment.es import augment_es_records_with_national_pliegos
+   ```
+   (Match how 04_build_maps.py already imports from `_lib` — copy an
+   existing import line's style exactly.)
+4. **Intermediate gates:**
+   ```
+   uv run python -m py_compile scripts/04_build_maps.py scripts/_lib/augment/*.py
+   uv run python -m pytest -q
+   uv run ruff check scripts/
+   ```
+   All three must pass. (04_build_maps.py cannot be imported by module
+   name — it starts with a digit — hence py_compile.)
+5. **Smoke the entry point** without a full build:
+   ```
+   uv run scripts/04_build_maps.py --help 2>&1 | head -5
+   ```
+   If the script has no argparse, run it for ~20 seconds and Ctrl-C after
+   the record-loading phase prints; an ImportError/NameError appears
+   immediately.
+6. **Commit:** `refactor: extract <names> to _lib/augment/<cc>.py (no-op)`
+
+## Final proof (once per phase, after the LAST module move)
+
+```
+uv run scripts/04_build_maps.py
+uv run scripts/compare_build_output.py tmp/golden-before wiki
+```
+
+Expected: `identical`, exit 0. ANY diff = a move changed behaviour →
+bisect by `git stash` / re-applying module moves one at a time. Do not
+rationalize a diff away; stage 04 output is deterministic by design (the
+codebase sorts set-derived structures specifically to guarantee this).
+
+## Known coupling traps in 04_build_maps.py
+
+- Several `augment_*` functions read module-level path constants
+  (`ROOT`, `RAW`, …) defined near the top of the file — they must be
+  imported or re-derived (`Path(__file__).resolve()` depth changes when
+  the file moves from `scripts/` to `scripts/_lib/augment/`! Re-derive
+  carefully: `parents[2]` from `scripts/_lib/augment/x.py` = repo root,
+  vs `parents[1]` from `scripts/04_build_maps.py`).
+- `_backfill_it_nonstub_from_masaf` is a private helper of the IT
+  augmenter — moves with it.
+- `synthesize_it_sottozone_records` is called between other IT steps in
+  `main()`; preserve call ORDER in main() exactly.
+- tqdm/print progress lines inside moved functions stay verbatim (curators
+  grep build logs for them).

.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: ci
+on:
+  push:
+    branches: [main, seo-geo-prerender]
+  pull_request:
+
+jobs:
+  checks:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+      - name: Sync dev dependencies
+        run: uv sync --group dev
+      - name: Ruff (lint, pinned at zero)
+        run: uv run ruff check scripts/ tests/
+      - name: Pytest
+        run: uv run python -m pytest -q
+
+  eslint:
+    # Advisory only — the map app JS lives in scripts/_lib/assets/app.js
+    # (extracted from the Python template in Phase 4). Build correctness is
+    # guarded by the byte-identity golden check, not by lint, so this job is
+    # non-blocking (continue-on-error).
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+      - name: ESLint (map app JS)
+        run: npx --yes eslint@9 scripts/_lib/assets/app.js

.gitignore

@@ -6,6 +6,11 @@ __pycache__/
 .DS_Store
 *.log
 
+# Bunny CDN access logs pulled via scripts/fetch_bunny_logs.py (v2 logging API)
+logs/
+
+.hermes/*
+
 # secrets — PISTE / Légifrance OAuth credentials, etc.
 .env
 .env.*
@@ -37,6 +42,9 @@ raw/wikipedia/*
 *.pmtiles
 *.zip
 *.html
+# …except test fixtures (short, redacted, licence-clear OJ-page HTML
+# excerpts) — these ARE committed; see tests/fixtures/README.md.
+!tests/fixtures/*.html
 
 # stage 02c round-trip work files (todo.json emitted by --emit-todo,
 # done*.json returned by the translator) — ephemeral