Skip to content

FR: eAmbrosia register as a second cahier source + keyed CARTO basemap - #24

Merged
bdevloed merged 1 commit into
mainfrom
fr-eambrosia-register
Sep 6, 2026
Merged

bdevloed merged 1 commit into
mainfrom
fr-eambrosia-register

Conversation

@bdevloed

@bdevloed bdevloed commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Two independent pieces of work that were sitting together in the working tree. Both are documented in full in CLAUDE.md. See the note at the bottom — say the word and I'll split them into separate PRs.


1. eAmbrosia register as a fallback cahier source (FR)

BO Agri is the FR pipeline's primary cahier source and its long tail is expensive: hand-curated URLs, a Légifrance cookie fetcher, OCR over professional-org mirrors. The eAmbrosia EU GI register serves the same INAO cahier PDF per GI as an attachment, with none of that — so it is wired as a self-service last tier, not a replacement. BO Agri and the curator overrides still resolve first, so every currently-canonical resolution stays canonical (today the tier fires for 0 of 466).

The join key — 01d_resolve_register.py

France is the one country sourced from INAO rather than eAmbrosia, so its records carry no id_eambrosia. Stage 01d supplies it by matching the SIQO appellation name against the register's protectedName.

A wrong bind attaches another appellation's cahier, which is worse than a gap, so _lib/fr/register_match.py refuses rather than guesses — three guards:

  • Product-type partition. Calvados is both a Wine PGI (the Normandy IGP) and a Spirit-drink PGI (the eau-de-vie); without partitioning by qualityProductType, either could claim the other's cahier.
  • No fuzzy matching. Exact normalised alias parts only (splitting SIQO's " ou " / " et " / commas, and / on the register side). Exact matching alone would miss ~20 appellations; fuzzy would risk a wrong bind — the alias fold does the work.
  • One-to-one. Two appellations claiming one file number means at least one bind is wrong, so both go back to the queue.

Result: 466 / 466 parents resolved, 0 unresolved — 444 on the full name, 20 on an alias, 2 pinned in the checked-in register_overrides.json.

The fallback tier — RegisterTier in stage 01

Runs only after BO Agri and the curator overrides miss. Attachments cache by sha256 outside raw/inao/cahiers/, deliberately: stage 02's build_global_segment_index indexes every PDF in that directory, so dumping a few hundred register cahiers there would silently rewrite unrelated records. Only a cahier that actually wins a resolution is promoted. Provenance is written only for entries the register won, so an appellation still served by BO Agri keeps a byte-identical manifest entry.

The shadow report — audit_fr_register_shadow.py

Compares the register cahier against the in-build record without writing into the build. v1 sweep over all 466 parents:

step count share
resolved to a register file number 464 99.6 %
register serves a cahier attachment 361 77.5 %
register cahier extracts 352 75.5 %
lien_au_terroir byte-identical to the build 92 26 % of extracted
differs substantively 215

The register is systematically the older vintage — of the 215 substantive differences the newer text is on the BO Agri side in 199. That settles the open question: it must stay a last tier and must not be promoted. The determinism control came back 356 / 356 byte-identical, so every difference is real, not extractor noise. Three records where the register is materially richer are queued in CURATOR_TODO.md.

Endpoint gotchas

In _lib/eambrosia_register.py: the internal id is not derivable from giIdentifier, the detail path has no /v1/, the attachment path does, and an explicit Accept: application/pdf trips the anti-bot gate.


2. Keyed CARTO basemap + staged OpenFreeMap successor

CARTO retired key-less raster access in 2026-08: an unkeyed tile still returns HTTP 200 but with "API KEY REQUIRED" stamped into the PNG — a failure invisible to any status-code check, visible only by looking at the image.

Tile URLs now carry ?key=, injected at build time from CARTO_BASEMAP_KEY in the repo-root .env via the new _lib/env.py (which also takes over _load_dotenv from _lib/batch). Unset ⇒ no key and a watermarked basemap, so stage 04 warns loudly on stderr. CARTO + OpenStreetMap attribution stays visible, as the free tier requires. The key-free vector successor is vendored at _lib/vendor/openfreemap-{positron,dark}.json for a later switch.


Tests

tests/test_fr_register_{match,tier}.py — 30 tests covering the matcher's three guards and the tier's ordering + cache isolation. Full suite 368 passing, ruff clean.

Note on scope

This bundles two unrelated features because that is how they sat in the working tree. Splitting them means hunk-level surgery across five shared files (CLAUDE.md, 04_build_maps.py, app.js, map_template.py, batch.py); I judged a single honest commit safer than risking a mis-split, but I'm happy to split it into two PRs with each half verified in a throwaway worktree.

🤖 Generated with Claude Code

…basemap

Two independent pieces of work that were sitting together in the working
tree. Both are described in full in CLAUDE.md.

1. eAmbrosia register as a fallback cahier source (FR)

BO Agri is the FR pipeline's primary cahier source and its long tail is
expensive: hand-curated URLs, a Légifrance cookie fetcher, OCR over
professional-org mirrors. The eAmbrosia EU GI register serves the *same*
INAO cahier PDF per GI as an attachment, with none of that — so it is
wired as a self-service **last tier**, not a replacement. BO Agri and the
curator overrides still resolve first, so every currently-canonical
resolution stays canonical (today the tier fires for 0 of 466).

- 01d_resolve_register.py supplies the join key FR lacks (INAO-sourced
  records carry no id_eambrosia): SIQO name → register protectedName.
  A wrong bind attaches another appellation's cahier, which is worse than
  a gap, so _lib/fr/register_match.py refuses rather than guesses —
  product-type partition (Calvados is both a Wine PGI and a Spirit-drink
  PGI), no fuzzy matching (exact normalised alias parts only), and a
  one-to-one check. 466/466 parents resolved, 0 unresolved: 444 on the
  full name, 20 on an alias, 2 pinned in register_overrides.json.
- RegisterTier in 01_scrape_cahiers.py runs only after BO Agri AND the
  curator overrides miss. Attachments cache by sha256 OUTSIDE
  raw/inao/cahiers/ so stage 02's global segment index cannot silently
  rewrite unrelated records; only a winning cahier is promoted.
  Provenance is written only for entries the register won, so an
  appellation still served by BO Agri keeps a byte-identical manifest.
- audit_fr_register_shadow.py compares register vs in-build cahier
  without writing into the build. v1 sweep over all 466 parents: the
  register is systematically the OLDER vintage (newer text on the BO Agri
  side in 199 of 215 substantive differences), which settles the open
  question — it must stay a last tier and must not be promoted. The
  determinism control came back 356/356 byte-identical, so the
  differences are real, not extractor noise. Three records where the
  register is materially richer are queued in CURATOR_TODO.
- Endpoint gotchas live in _lib/eambrosia_register.py: the internal id is
  not derivable from giIdentifier, the detail path has no /v1/, the
  attachment path does, and an explicit Accept: application/pdf trips the
  anti-bot gate.

2. Keyed CARTO basemap + staged OpenFreeMap successor

CARTO retired key-less raster access in 2026-08: an unkeyed tile still
returns HTTP 200 but with "API KEY REQUIRED" stamped into the PNG — a
failure invisible to any status-code check. Tile URLs now carry ?key=,
injected at build time from CARTO_BASEMAP_KEY in the repo-root .env via
the new _lib/env.py (which also takes over _load_dotenv from _lib/batch).
Unset ⇒ no key and a watermarked basemap, so stage 04 warns loudly.
CARTO + OpenStreetMap attribution stays visible, as the free tier
requires. The key-free vector successor is vendored at
_lib/vendor/openfreemap-{positron,dark}.json for a later switch.

Tests: tests/test_fr_register_{match,tier}.py (30 tests) cover the
matcher's three guards and the tier's ordering + cache isolation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bdevloed
bdevloed merged commit fbe16b0 into main Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant