FR: eAmbrosia register as a second cahier source + keyed CARTO basemap - #24
Merged
Merged
Conversation
…basemap
Two independent pieces of work that were sitting together in the working
tree. Both are described in full in CLAUDE.md.
1. eAmbrosia register as a fallback cahier source (FR)
BO Agri is the FR pipeline's primary cahier source and its long tail is
expensive: hand-curated URLs, a Légifrance cookie fetcher, OCR over
professional-org mirrors. The eAmbrosia EU GI register serves the *same*
INAO cahier PDF per GI as an attachment, with none of that — so it is
wired as a self-service **last tier**, not a replacement. BO Agri and the
curator overrides still resolve first, so every currently-canonical
resolution stays canonical (today the tier fires for 0 of 466).
- 01d_resolve_register.py supplies the join key FR lacks (INAO-sourced
records carry no id_eambrosia): SIQO name → register protectedName.
A wrong bind attaches another appellation's cahier, which is worse than
a gap, so _lib/fr/register_match.py refuses rather than guesses —
product-type partition (Calvados is both a Wine PGI and a Spirit-drink
PGI), no fuzzy matching (exact normalised alias parts only), and a
one-to-one check. 466/466 parents resolved, 0 unresolved: 444 on the
full name, 20 on an alias, 2 pinned in register_overrides.json.
- RegisterTier in 01_scrape_cahiers.py runs only after BO Agri AND the
curator overrides miss. Attachments cache by sha256 OUTSIDE
raw/inao/cahiers/ so stage 02's global segment index cannot silently
rewrite unrelated records; only a winning cahier is promoted.
Provenance is written only for entries the register won, so an
appellation still served by BO Agri keeps a byte-identical manifest.
- audit_fr_register_shadow.py compares register vs in-build cahier
without writing into the build. v1 sweep over all 466 parents: the
register is systematically the OLDER vintage (newer text on the BO Agri
side in 199 of 215 substantive differences), which settles the open
question — it must stay a last tier and must not be promoted. The
determinism control came back 356/356 byte-identical, so the
differences are real, not extractor noise. Three records where the
register is materially richer are queued in CURATOR_TODO.
- Endpoint gotchas live in _lib/eambrosia_register.py: the internal id is
not derivable from giIdentifier, the detail path has no /v1/, the
attachment path does, and an explicit Accept: application/pdf trips the
anti-bot gate.
2. Keyed CARTO basemap + staged OpenFreeMap successor
CARTO retired key-less raster access in 2026-08: an unkeyed tile still
returns HTTP 200 but with "API KEY REQUIRED" stamped into the PNG — a
failure invisible to any status-code check. Tile URLs now carry ?key=,
injected at build time from CARTO_BASEMAP_KEY in the repo-root .env via
the new _lib/env.py (which also takes over _load_dotenv from _lib/batch).
Unset ⇒ no key and a watermarked basemap, so stage 04 warns loudly.
CARTO + OpenStreetMap attribution stays visible, as the free tier
requires. The key-free vector successor is vendored at
_lib/vendor/openfreemap-{positron,dark}.json for a later switch.
Tests: tests/test_fr_register_{match,tier}.py (30 tests) cover the
matcher's three guards and the tier's ordering + cache isolation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent pieces of work that were sitting together in the working tree. Both are documented in full in
CLAUDE.md. See the note at the bottom — say the word and I'll split them into separate PRs.1. eAmbrosia register as a fallback cahier source (FR)
BO Agri is the FR pipeline's primary cahier source and its long tail is expensive: hand-curated URLs, a Légifrance cookie fetcher, OCR over professional-org mirrors. The eAmbrosia EU GI register serves the same INAO cahier PDF per GI as an attachment, with none of that — so it is wired as a self-service last tier, not a replacement. BO Agri and the curator overrides still resolve first, so every currently-canonical resolution stays canonical (today the tier fires for 0 of 466).
The join key —
01d_resolve_register.pyFrance is the one country sourced from INAO rather than eAmbrosia, so its records carry no
id_eambrosia. Stage 01d supplies it by matching the SIQO appellation name against the register'sprotectedName.A wrong bind attaches another appellation's cahier, which is worse than a gap, so
_lib/fr/register_match.pyrefuses rather than guesses — three guards:qualityProductType, either could claim the other's cahier." ou "/" et "/ commas, and/on the register side). Exact matching alone would miss ~20 appellations; fuzzy would risk a wrong bind — the alias fold does the work.Result: 466 / 466 parents resolved, 0 unresolved — 444 on the full name, 20 on an alias, 2 pinned in the checked-in
register_overrides.json.The fallback tier —
RegisterTierin stage 01Runs only after BO Agri and the curator overrides miss. Attachments cache by sha256 outside
raw/inao/cahiers/, deliberately: stage 02'sbuild_global_segment_indexindexes every PDF in that directory, so dumping a few hundred register cahiers there would silently rewrite unrelated records. Only a cahier that actually wins a resolution is promoted. Provenance is written only for entries the register won, so an appellation still served by BO Agri keeps a byte-identical manifest entry.The shadow report —
audit_fr_register_shadow.pyCompares the register cahier against the in-build record without writing into the build. v1 sweep over all 466 parents:
lien_au_terroirbyte-identical to the buildThe register is systematically the older vintage — of the 215 substantive differences the newer text is on the BO Agri side in 199. That settles the open question: it must stay a last tier and must not be promoted. The determinism control came back 356 / 356 byte-identical, so every difference is real, not extractor noise. Three records where the register is materially richer are queued in
CURATOR_TODO.md.Endpoint gotchas
In
_lib/eambrosia_register.py: the internalidis not derivable fromgiIdentifier, the detail path has no/v1/, the attachment path does, and an explicitAccept: application/pdftrips the anti-bot gate.2. Keyed CARTO basemap + staged OpenFreeMap successor
CARTO retired key-less raster access in 2026-08: an unkeyed tile still returns HTTP 200 but with "API KEY REQUIRED" stamped into the PNG — a failure invisible to any status-code check, visible only by looking at the image.
Tile URLs now carry
?key=, injected at build time fromCARTO_BASEMAP_KEYin the repo-root.envvia the new_lib/env.py(which also takes over_load_dotenvfrom_lib/batch). Unset ⇒ no key and a watermarked basemap, so stage 04 warns loudly on stderr. CARTO + OpenStreetMap attribution stays visible, as the free tier requires. The key-free vector successor is vendored at_lib/vendor/openfreemap-{positron,dark}.jsonfor a later switch.Tests
tests/test_fr_register_{match,tier}.py— 30 tests covering the matcher's three guards and the tier's ordering + cache isolation. Full suite 368 passing, ruff clean.Note on scope
This bundles two unrelated features because that is how they sat in the working tree. Splitting them means hunk-level surgery across five shared files (
CLAUDE.md,04_build_maps.py,app.js,map_template.py,batch.py); I judged a single honest commit safer than risking a mis-split, but I'm happy to split it into two PRs with each half verified in a throwaway worktree.🤖 Generated with Claude Code