Gi terms and analytics

.gitignore

@@ -64,8 +64,8 @@ done*.json
 .claude/settings.local.json
 .claude/scheduled_tasks.lock
 
-# gettext catalogs — UI-localisation layer; not committed in the minimal
-# bootstrap. Stage 04 falls back to the FR source when these are absent.
-locale/
+# gettext catalogs: the .po sources + .pot are committed (a fresh checkout must
+# render EN/ES/NL, not the FR msgids); only the compiled .mo is derived.
+locale/**/*.mo
 
 tmp/*

README.md

@@ -2,24 +2,25 @@
 
 A reference wiki + map of European wine appellations, generated mechanically
 from public regulator data. France (INAO + JORF) is the canonical pipeline;
-Spain (eAmbrosia + EUR-Lex) lives under `scripts/es/`; Portugal (eAmbrosia +
-IVV) under `scripts/pt/`; Italy under `scripts/it/`; Austria under
-`scripts/at/`; Slovenia under `scripts/si/`. Every per-record fact traces
-back to a public-source document — nothing here is hand-written narrative.
+the other 20 countries, the United Kingdom included, each have a sibling
+pipeline under `scripts/<cc>/` sourced from the eAmbrosia EU register, the
+national regulator, the Swiss federal repertoire or the UK GI register. The
+per-country list, coverage and mechanics live in `CLAUDE.md`. Every
+per-record fact traces back to a public-source document — nothing here is
+hand-written narrative.
 
 ## Status
 
 The FR pipeline runs end-to-end across the full AOC/AOP/IGP corpus, emitting
 per-denomination markdown pages (one per appellation plus one per DGC —
 Muscadet sub-crus, Côtes du Rhône Villages, Alsace grands crus, Chablis
-premier-cru climats, etc.). The ES pipeline covers the ~149 wine GIs in
-eAmbrosia (106 DOP + 43 IGP); coverage is a function of which wines have an
-EU-OJ "documento único" — see `CLAUDE.md` for the curator workflow. The PT
-pipeline covers the 44 wine GIs (30 DOP + 14 IGP) sourced from eAmbrosia +
-the IVV cadernos master indexes. Italy (531 wine GIs), Austria (32) and
-Slovenia (17 — 14 DOP + 3 IGP) follow the same eAmbrosia + EU-OJ
-single-document pattern. Stage 04 merges all six streams into a single
-four-locale interactive map (FR / EN / ES / NL). The site is deployed at
+premier-cru climats, etc.). The other 20 country pipelines follow the same
+stage layout (00 fetch → 01 fetch documents → 02 extract → 02d/02e terroir
+facts → 03 wiki), each with its own document source and geometry chain;
+coverage per country is a function of which wines have a fetchable
+specification, and `CLAUDE.md` documents every country's sources, coverage
+and curator workflow. Stage 04 merges all 21 streams into a single
+four-locale interactive map (EN / FR / ES / NL). The site is deployed at
 <https://www.openwinemap.com>.
 
 ## Setup
@@ -160,19 +161,22 @@ Same `--emit-todo` / `--import` flags apply to 02d and 02e.
 ### Internationalisation (map UI chrome)
 
 Sidebar labels, panel headings, and style chip names are translated via
-gettext. Catalogs live under `locale/<lang>/LC_MESSAGES/messages.po` and are
-hand-editable. Stage 04 recompiles `messages.mo` automatically when the `.po`
-is newer.
+gettext. Catalogs live under `locale/<lang>/LC_MESSAGES/messages.po`, are
+committed to the repo, and are hand-editable. Stage 04 recompiles
+`messages.mo` automatically when the `.po` is newer.
 
 ```
-uv run pybabel extract -F locale/babel.cfg -o locale/messages.pot scripts/_lib/
-uv run pybabel update  -i locale/messages.pot -d locale
-uv run pybabel init    -i locale/messages.pot -d locale -l <lang>   # new locale
+.venv/bin/python -m babel.messages.frontend extract -F locale/babel.cfg -o locale/messages.pot scripts/_lib/
+.venv/bin/python -m babel.messages.frontend update  -i locale/messages.pot -d locale --no-fuzzy-matching
+.venv/bin/python -m babel.messages.frontend init    -i locale/messages.pot -d locale -l <lang>   # new locale
 ```
 
 Always extract from the directory, not a single file — `style_taxonomy.py`
 carries msgid anchors that are silently dropped otherwise (and pybabel
-update will mark them obsolete).
+update will mark them obsolete). Always pass `--no-fuzzy-matching` to
+`update`: without it pybabel fills every new msgid with a guess borrowed from
+an unrelated existing entry and marks it fuzzy, which is harder to spot than
+an empty msgstr. Then set the new msgstrs by hand in each locale.
 
 ## Public data sources
 
@@ -195,13 +199,16 @@ for the full rules.
 | **EUR-Lex — OJ single documents** — `eur-lex.europa.eu` (HTML) | Canonical pliego de condiciones (documento único) for ES wines: zona geográfica, variedades, vínculo, rendimientos. Stage 01 fetches each wine's `publications[0].uri`; stage 01b uses headless Chromium to solve the CloudFront WAF challenge on the blocked subset | EU public sector information |
 | **MAPA + CCAA national pliegos** — `mapa.gob.es`, JCCM, INCAVI, AGACAL, ITACyL, Aragón, Navarra, GVA, Canarias, Andalucía, Euskadi, Madrid, Extremadura (per-region PDFs) | Secondary/accessory grape varieties not published in the EU-OJ documento único (stage 02f) | Public domain (national/regional gazettes) |
 | **SIGPAC vineyard parcels** — `fega.es` (per-comarca shapefiles) | Pliego-cited polygon inclusions for fine-grained ES geometry (e.g. Priorat vs Montsant overlap resolution) | Licence-clear under MAPA terms |
-| **GISCO LAU 2021** — Eurostat | EU-wide municipio polygons for ES IGP commune-list / province-wide / CCAA-wide geometry fallback | © EuroGeographics for the administrative boundaries (free reuse) |
+| **GISCO LAU 2024** — Eurostat | EU-wide municipality polygons for the commune-list / province-wide / region-wide geometry fallbacks (ES IGPs and the other eAmbrosia countries) | © EuroGeographics for the administrative boundaries (free reuse) |
 | **Bétard 2022 EU PDO geometry** — [Figshare](https://figshare.com/) (`EU_PDO.gpkg`) | Pre-Nov-2021 EU PDO polygons; covers ~99 of 106 ES DOPs and all 30 PT DOPs | CC0 |
 | **IVV cadernos de especificações** — `ivv.gov.pt` (per-DOP/IGP PDFs) | Canonical legal definition of every Portuguese wine GI — área delimitada, castas, rendimentos, relação com a área geográfica | Public domain (Portuguese state) |
 | **DGT CAOP 2025** — `geo2.dgterritorio.gov.pt` (Continente + RAA + RAM GPKGs) | Portuguese commune-precision boundaries for future PT IGP commune-list geometry | CC BY 4.0 |
 | **Wikipedia** — `<lang>.wikipedia.org` REST API | Sidepanel tooltips for grape varieties and distinctive styles (stages 02b/grapes, 02b/styles); per-AOC pages used as a sommelier-vocabulary salience hint for terroir-fact extraction (stage 02b/aocs → 02d) | CC BY-SA 4.0 |
 | **VIVC** — [Vitis International Variety Catalogue](https://www.vivc.de/), Julius Kühn-Institut Geilweilerhof | Canonical grape-variety names + VIVC variety numbers driving the per-AOC pill's "canonical bracket" (e.g. *Aragonez (Tempranillo Tinto)*), and synonym-aware Wikipedia search (stage 02g + 02b/grapes). Cite: Röckel et al., Vitis International Variety Catalogue — www.vivc.de | Factual citation only — JKI publishes no explicit data licence. We ship VIVC IDs + prime names; verbatim synonym strings are *not* republished pending JKI confirmation. |
-| **Anthropic Messages API** — `claude-haiku-4-5` | Cahier-summary translation (02c), terroir-fact extraction from cahier section X + Wikipedia (02d), terroir-fact translation (02e), grape-tooltip translation (02b/grapes-translate); each stage can be swapped to Ollama or to manual round-trip | n/a — outputs are derivatives of the cahier (public domain) and Wikipedia (CC BY-SA 4.0) |
+| **MASAF *Elenco alfabetico dei vini DOP*** — `masaf.gov.it` | Source of the Italian traditional term attached to each DOP as a whole (DOCG vs DOC) | Italian public-sector information (MASAF) |
+| **MAPA *Listado de DOP e IGP de vinos*** — `mapa.gob.es` | Source of the Spanish traditional term attached to each GI as a whole (DOCa / DOQ, DO, Vino de Pago, Vino de Calidad, Vino de la Tierra) | Spanish public-sector information (MAPA) |
+| **Reg. (EC) 607/2009 Annex XII** — [legislation.gov.uk copy](https://www.legislation.gov.uk/eur/2009/607/annex/XII/adopted/data.xht) | Reference list of the traditional terms registered per member state (Reg. 1308/2013 Art. 112(a)); backs the definition and source shown when hovering a term | EU public sector information |
+| **Anthropic Messages API** — `claude-sonnet-4-6` (default model) | Terroir-fact extraction from the regulator's terroir text + Wikipedia (02d) and terroir-fact translation (02e); cahier-summary translation (02c) and grape/style-tooltip translation (02b/grapes-translate, 02b/styles-translate) can run here too, but in practice 02c goes through the manual round-trip (a human translator) and the tooltips run mostly on Ollama with Mistral Small 3.2, with a Claude residual; each stage can be swapped to Ollama, Mistral or manual round-trip | n/a — outputs are derivatives of the regulator text (public sector) and Wikipedia (CC BY-SA 4.0) |
 
 The map UI displays attribution alongside any Wikipedia extract ("via
 Wikipedia · CC BY-SA 4.0"), any translated summary ("Machine translated
@@ -212,6 +219,25 @@ carry per-bullet provenance (`cahier` / `wiki` / `both`); bullets grounded
 in Wikipedia render the CC BY-SA 4.0 attribution inline, the rest default
 to the cahier-PDF footer link.
 
+### Appellation names: traditional term and legal scheme
+
+Every appellation carries two names, and they mean two different things:
+the **traditional term** the country's regulator attaches to the GI as a
+whole (DOCG, DOQ, AOC, Vinho Regional, …; Reg. 1308/2013 Art. 112(a),
+registered in Reg. (EC) 607/2009 Annex XII) and the **scheme** it is
+registered under (the EU's PDO / PGI). The map shows them as
+`TERM (SCHEME)`, for example "DOCG (PDO)" or "DOQ (PDO)", with the scheme
+word localised per UI language ("DOCG (AOP)" in French, "DOCG (BOB)" in
+Dutch). Where a country has no term of its own (Germany's Mosel, the UK),
+only the scheme is shown. Swiss AOCs sit outside the EU scheme and carry no
+bracket; the United Kingdom registers under its own GI scheme, which keeps
+the words PDO and PGI; French eaux-de-vie are spirit-drink GIs, not wine
+PDOs. Lot-level quality grades (Qualitätswein, Prädikatswein, kakovostno
+vino) are not terms in this sense and are not shown, and neither are scheme
+abbreviations (ΠΟΠ, CHOP, ЗНП, BOB). Hovering a term in the panel shows its
+definition and source. The stored `kind` token (AOC / DOP / IGP / EDV) is
+unchanged; both axes (`eu_scheme`, `national_term`) are derived at stage 04.
+
 ## Licence
 
 - **Code** (`scripts/`, `pyproject.toml`, etc.) — MIT, see `LICENSE`.

VERIFICATION.md

@@ -884,3 +884,32 @@ the **GOV.UK "protected food and drink names" register** run by DEFRA.
 # Independent check: https://www.gov.uk/protected-food-drink-names
 #   → filter Register = "Wines", Country of origin = "United Kingdom".
 ```
+
+## 2026-09-07 — traditional terms vs the national rosters (IT MASAF elenco, ES MAPA listado)
+
+Independent cross-check of the `national_term` axis introduced with the two-axis naming
+layer (`scripts/_lib/gi_terms.py`, `scripts/_lib/traditional_terms.json`).
+
+| check | corpus (parents, wines) | reference | result |
+|---|---:|---|---|
+| IT DOCG | 79 | MASAF *Elenco alfabetico dei vini DOP* agg. 18.03.2026: 79 DOCG rows | ✅ equal |
+| IT DOC | 333 | elenco 332 + Valtènesi (registered 2026-03-18, Reg. (EU) 2026/572, pinned) | ✅ equal |
+| IT IGT | 111 | eAmbrosia IT PGI records (112 incl. the Salemi stub not on the map) | ✅ |
+| ES DOCa/DOQ | 1 + 1 | MAPA listado 2 July 2026: DOCa 2 (Rioja, Priorat; Priorat shown as DOQ) | ✅ |
+| ES Vino de Pago | 28 | listado VP 27 + Urbezo (VP per MAPA 2024-10-25, listado still DO; pinned) | ✅ |
+| ES Vino de Calidad | 7 | listado VC 7 | ✅ |
+| ES DO | 69 | listado DO 70 − Urbezo | ✅ |
+| ES Vino de la Tierra | 43 | listado VT 43 (all 43 ES PGIs) | ✅ |
+| AT DAC | 18 | BML DAC-Verordnungen page 17 + Wagram (RIS) | ✅ |
+| sub == parent | 1,242 subs | every sub-denomination's (eu_scheme, national_term) equals its parent's | ✅ 0 mismatches |
+| kind invariant | AOC 1404 / AOP 1 / DOP 944 / EDV 28 / IGP 537 | same counts in `appellations{,-villages}.geojson` before and after | ✅ unchanged |
+
+Recipe:
+
+```
+.venv/bin/python scripts/it/00_fetch_data.py        # refreshes raw/it/masaf-elenchi/ (dated ServeAttachment link)
+.venv/bin/python scripts/es/00_fetch_data.py        # refreshes raw/es/mapa/listado-dop-igp-vinos.pdf
+.venv/bin/python scripts/04_build_maps.py
+.venv/bin/python scripts/audit_gi_terms.py --strict
+grep -o '"kind": *"[A-Z]*"' wiki/map-data/appellations.geojson | sort | uniq -c
+```

docs/analytics.md

@@ -0,0 +1,83 @@
+# Analytics — Plausible events, goals and how to read the numbers
+
+The map ships a self-hosted [Plausible](https://plausible.io/) snippet
+(`analytics.dev.devloed.com`, injected by `_TEMPLATE` in
+[scripts/_lib/map_template.py](../scripts/_lib/map_template.py)); the site id
+is **`openwinemap.com`** (Plausible strips the `www.`). Custom events are sent
+through the `track(name, props)` helper in
+[scripts/_lib/assets/app.js](../scripts/_lib/assets/app.js), which no-ops
+when the script is blocked. Every prop is a bounded slug vocabulary — never
+raw user text — so breakdowns stay useful and nothing personal can leak.
+
+## Events sent by app.js
+
+| Event | Props | Fired when |
+|---|---|---|
+| `Appellation Viewed` | `slug`, `country`, `kind`, `region`, `stacked`, `stack_size`, `via`, `locale` | the detail panel opens or the stack focus changes. `via` ∈ `map` (click on the map), `cycle` (re-click cycling a stack), `facet`, `omnisearch`, `panel-link` (parent/child link inside a card). **Not** fired for the page-load open of a `/<lang>/<slug>` landing (the pageview already records it) nor for the localStorage restore. |
+| `Appellation Opened` | `slug`, `via` (`facet` / `omnisearch`), `locale` | an explicit pick — the cleanest interest signal |
+| `Filter Applied` | `facet`, `value`, `locale` | a facet checkbox / chip / country / region / appellation filter changes |
+| `Filters Reset` | `locale` | the reset button |
+| `Kind Toggled` | `kind` (`igp` / `spirits`), `enabled`, `via` (`reveal-hint`, optional), `locale` | the IGP / spirits switches |
+| `View Mode Switched` | `mode` (`simple` / `advanced`), `locale` | the mode toggle |
+| `Grape Scope Toggled` | `scope` (`main` / `all`), `locale` | principal-only vs all grapes |
+| `Omnisearch Used` | `result_count`, `had_match`, `groups` (a/g/r/s), `query_len`, `locale` | 1 s after typing stops in the omnisearch |
+| `Omnisearch Result Picked` | `type` (`grape` / `region` / `style` / `classification`), `locale` | a non-appellation suggestion is picked (appellation picks fire `Appellation Opened`) |
+| `Search Used` | `result_count`, `had_match`, `query_len`, `locale` | the legacy sidebar search (superseded by the omnisearch in June 2026) |
+| `Theme Changed` | `theme`, `locale` | light / dark toggle |
+| `Feedback Clicked` | `channel` (`github` / `email`), `locale` | a link tagged `data-feedback` in the sidebar disclaimer or the About dialog |
+| `Outbound Link: Click` | `url` | Plausible's own outbound-link tracking (source PDFs, Wikipedia, interprofessions, GitHub) |
+
+## Goals — what the dashboard and the Stats API can see
+
+Plausible stores every custom event but only **shows** the ones configured as
+a goal (Site settings → Goals → *Add goal* → *Custom event*, exact name).
+Adding a goal is retroactive — history appears immediately. Configured on
+2026-09-11: `Appellation Viewed`, `Filter Applied`, `Outbound Link: Click`,
+`View Mode Switched`, `Kind Toggled`, `Filters Reset`, `Search Used`.
+
+**Still to add** (sent since June 2026 but invisible until then):
+
+- `Appellation Opened`
+- `Omnisearch Used`
+- `Omnisearch Result Picked`
+- `Theme Changed`
+- `Grape Scope Toggled`
+- `Feedback Clicked` (new, 2026-09)
+
+Until `Omnisearch Used` is a goal, search usage is unmeasured — `Search Used`
+went to zero when the omnisearch replaced the sidebar search on 2026-06-19.
+
+## Reading the numbers — artefacts to keep in mind
+
+- **Pageviews on entity paths are entries only.** Opening an appellation
+  rewrites the URL with `history.replaceState`, which Plausible does not count
+  as a pageview (by design — panel opens are not page loads). `/en/bourgogne`
+  showing 85 visitors and 10 pageviews means 10 landings and 85 people whose
+  panel was on Bourgogne at some point. Use the `Appellation Viewed` slug
+  breakdown, not the Pages report, for what people look at.
+- **Bounce on entity landings was ~0 by construction until 2026-09-11**: the
+  page-load open fired `Appellation Viewed`, and any second event ends a
+  bounce. From this build on, a landing that only reads the SSR page counts
+  as a bounce again, so expect the organic bounce rate to rise from ~8 % to
+  something honest.
+- **Empty entry page + 0 pageviews** (≈ 7 % of visits) are map tabs resumed
+  after the 30-minute session timeout; they show as *Direct*. Real, engaged
+  sessions — not a bug.
+- **`Appellation Viewed.slug` is the stack focus**, i.e. the smallest
+  bounding box under the click, so an umbrella appellation whose aire equals
+  a famous one gets the credit (Coteaux Champenois ≡ Champagne). Filter on
+  `via` = `facet` / `omnisearch` — or use `Appellation Opened` — for intent.
+- A wrongly-huge simple-mode polygon shows up in this breakdown before it
+  shows up anywhere else (Pouilly-Loché, 2026-09: a 0.4 km² AOC drawn
+  across all of Burgundy by a stage-02 aire parse miss). Stage 04's
+  `[villages-guard]` log line now flags the pattern; the analytics is the
+  second line of defence.
+
+## Querying
+
+The Stats API v2 (`POST /api/v2/query`, `Authorization: Bearer <key>`) covers
+aggregates, time series and breakdowns by any prop or visit dimension; no DB
+tunnel is needed for insights. Keys are created per session in Plausible's
+settings and revoked afterwards — never commit or store one. A ClickHouse
+tunnel (port 8123) is only worth it for session-sequence questions
+("searched, then opened?") that the API cannot express.