Fix scientific data preparation, cache ownership, and heatmap geometry

CHANGELOG.md

@@ -5,6 +5,45 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Fixed
+
+- Regional plots select rows before axis scaling, labels and LD lookup, preserve
+  lead identity and honor shared/per-panel LD settings. Invalid stacked lead
+  positions are rejected; canonical and legacy frames can share a stack.
+- Label eligibility is resolved once from the selected lead row. Fresh reference
+  LD replaces an existing `R2` column without losing coloring or changing row order.
+- Colocalization projects fields from their declared source before merging, so
+  unrelated metadata cannot change effect colors or satisfy a missing column.
+  Disabling effect coloring also disables its effect-column requirements.
+- GTEx preserves chromosome and absolute position; relative TSS distance is no
+  longer accepted as an absolute coordinate. GTF/GFF3 display names are preserved
+  independently of attribute order.
+- Genome-wide column resolution reaches QQ compositions and Miami hover. Missing
+  categorical groups remain visible, including groups omitted from a custom order.
+  Pandas categorical columns render correctly in categorical Manhattan plots.
+- PLINK resolves paths before changing directory and recognizes registered Ensembl
+  species aliases. Unknown chromosome-set support is rejected explicitly.
+- Recombination plotting reads caller directories without replacing files or
+  applying an implicit liftover. Managed maps reject unavailable target assemblies.
+- Archive ingestion rejects links and unsafe or duplicate members, streaming map
+  text into generated canonical paths. Concurrent downloads use private staging.
+- Gene and exon caches publish together and retain the previous entry after an
+  interrupted write. Legacy separate CSV pairs are treated as cache misses.
+- Matplotlib and Bokeh heatmap cells share their boundaries with SNP highlights.
+- Example comparison generates outside the checkout, preserves manual edits and
+  returns a failing status for unexpected differences.
+
+### Changed
+
+- FINEMAP/CAVIAR loaders preserve PIPs and supplied membership without inventing
+  credible sets. Figures with no membership use the existing PIP-only display.
+- `calculate_colocalization_overlap` matches chromosome plus absolute position.
+  Position-only inputs require `common_chrom`; output columns are canonical.
+- `Species.plink_flags=None` means unknown support. An explicit empty tuple keeps
+  PLINK's human chromosome defaults.
+
 ## [3.1.1] - 2026-09-08
 
 ### Added

CONTRIBUTING.md

@@ -65,16 +65,19 @@ markdown, mermaid, link-check, or test failures will not merge.
 - **Base branch:** open PRs against `main`.
 - **Tests first:** follow test-driven development — add or update tests in
   `tests/` before (or alongside) the implementation. Mock PLINK calls rather
-  than requiring a local install; see `tests/test_ld.py` for the pattern.
+  than requiring a local install; see `tests/test_ld_process.py` for the pattern.
 - **Changelog:** add an entry to `CHANGELOG.md` under the `## [Unreleased]`
   section, using the `Added` / `Changed` / `Fixed` / `Removed` categories.
 - **Docs:** update `README.md`, `docs/USER_GUIDE.md`, `docs/ARCHITECTURE.md`,
   or `docs/CONFIGURATION.md` when your change alters public APIs, adds
   features, or changes behavior that users rely on.
 - **Example plots:** if your change touches a backend or a panel, run
   `scripts/example_diff.sh`. It regenerates the examples and lists the exports
-  whose content changed after normalising generated ids; the suite does not
-  see serialised output. Commit only the files it reports, using `--keep`.
+  whose content changed after normalising generated ids. Generation runs in a
+  temporary directory and leaves the checkout untouched. Exit 1 reports differences;
+  exit 2 reports a failure. Review the changes, then use `--keep` to accept only
+  changed exports. Acceptance refuses to overwrite manually modified exports.
+  Commit the accepted outputs with the source change.
 - **Commits:** keep messages focused on *what* changed and *why*. Do not
   include AI or tool attribution.
 - **Scope:** one logical change per PR. Refactors and feature work belong in

docs/ARCHITECTURE.md

@@ -14,6 +14,19 @@ of three interchangeable backends: matplotlib (static PNG/PDF), plotly
 backend-pluggable pipeline: validation → data preparation → backend-agnostic
 plot assembly → backend-specific rendering.
 
+Regional preparation resolves each panel's columns and LD options, selects its
+rows once and carries the selected lead onward. Genome-wide preparation projects
+configured roles to canonical columns before sharing layout with QQ and Miami.
+Colocalization projects each source before merging, so caller metadata cannot
+rename internal fields. The backends use the shared `cell_edges` geometry for
+heatmap cells and highlights.
+
+Reference-data ownership is explicit. Caller map directories are read-only;
+managed caches alone may download and replace generations. Download writers have
+private staging files, map archives stream regular members into canonical names,
+and gene/exon pairs publish as one atomically replaced ZIP. See
+[ADR 0009](adr/0009-resolved-inputs-and-owned-publication.md).
+
 ## Component Diagram
 
 ```mermaid
@@ -32,6 +45,8 @@ graph TD
 
     subgraph Prepare["Data Preparation"]
         DATA[_data.py: shared p-value intake]
+        REGIONAL["plotter.py: selected regional inputs and resolved leads"]
+        CACHE["_gene_cache.py: atomic gene and exon archive"]
         LD[ld.py: PLINK wrapper]
         RECOMB[recombination.py: maps + CanFam4 liftover]
         ENSEMBL["reference_genes.py:<br/>gene fetch by build<br/>(ensembl.py, ucsc.py)"]
@@ -72,15 +87,18 @@ graph TD
     SCHEMA --> EQTLV
     EQTLV --> COLORS
     LD --> COLORS
-    COLORS --> LZ
-    DATA --> LZ
+    COLORS --> PANELS
+    DATA --> REGIONAL
+    LZ --> REGIONAL
+    REGIONAL --> LD
+    REGIONAL --> PANELS
+    ENSEMBL --> CACHE
     COLORS --> MP
     COLORS --> SP
     COLORS --> MIAMI
     COLORS --> LDH
     COLORS --> CP
     RECOMB --> LZ
-    LZ --> PROTO
     LZ --> PANELS
     MP --> PANELS
     SP --> PANELS
@@ -229,7 +247,7 @@ stages:
 | Regional panels | Internal modules | `src/pylocuszoom/panels/{association,finemapping,eqtl,genes,heatmap}.py` | One module per panel type, each holding its value type, the constructor it builds itself through, and the `draw` method that draws it. A panel carries its resolved mode, region, hover contract and layout, so drawing inspects no columns |
 | `MiamiRequest`, `MiamiPanel`, `miami_plan` | Internal module | `src/pylocuszoom/panels/miami.py` | The Miami figure: a request the plotter resolves, a panel that draws one mirrored Manhattan half with its SNP annotations, and the builder that lays two of them on a `FigurePlan` with the cross-panel highlights |
 | `PhewasPanel`, `ForestPanel` | Internal module | `src/pylocuszoom/panels/stats.py` | The PheWAS and forest panels, each built through `from_frame` and drawing itself. Every family is a panel value with `draw` on a `FigurePlan`; no family holds a renderer class |
-| `ColocPanel` | Internal module | `src/pylocuszoom/panels/coloc.py` | The colocalization scatter: the merged frame, its resolved column names and lead index, drawing itself with both threshold lines through `add_significance_line` |
+| `ColocPanel` | Internal module | `src/pylocuszoom/panels/coloc.py` | The colocalization scatter: the projected frame with fixed source-owned column roles and lead index, drawing itself with both threshold lines through `add_significance_line` |
 | `LDHeatmapPanel` | Internal module | `src/pylocuszoom/panels/ld_heatmap.py` | The standalone heatmap: the matrix, its ids, and the lead and highlight indices, drawing itself |
 | `ManhattanPlotter` | Class | `src/pylocuszoom/manhattan_plotter.py` | Genome-wide Manhattan and QQ plots |
 | `StatsPlotter` | Class | `src/pylocuszoom/stats_plotter.py` | PheWAS and forest plots |

docs/CODEMAP.md

@@ -199,7 +199,8 @@ Data transformation between validated input and backend-ready primitives.
 | 3d | get_recombination_rate_for_region | Region-filtered recomb rate | [recombination.py](../src/pylocuszoom/recombination.py) |
 | 3d | download_canine_recombination_maps | Lazy-download bundled maps | [recombination.py](../src/pylocuszoom/recombination.py) |
 | 3d | recomb_for_region, RecombResult | The one place the skip-the-overlay decision is made, reported as a value | [recombination.py](../src/pylocuszoom/recombination.py) |
-| 3d | download_recombination_maps, RecombSource | Species-generic download, extract and publish; the record carries everything that varies | [recombination.py](../src/pylocuszoom/recombination.py) |
+| 3d | download_recombination_maps, RecombSource | Species-generic download, canonical member streaming and publication; the record carries everything that varies | [recombination.py](../src/pylocuszoom/recombination.py) |
+| 3e | prepare_genomewide_frames | Per-input column projection before genome-wide layout and composition | [manhattan.py](../src/pylocuszoom/manhattan.py) |
 | 3e | prepare_manhattan_frames | Cumulative-position Manhattan prep against one shared `GenomeLayout` | [manhattan.py](../src/pylocuszoom/manhattan.py) |
 | 3e | GenomeLayout | Chromosome order, offsets, colours, ticks, and x limits for every panel of a figure | [manhattan.py](../src/pylocuszoom/manhattan.py) |
 | 3f | prepare_qq_data | Observed vs expected QQ data | [qq.py](../src/pylocuszoom/qq.py) |
@@ -208,12 +209,14 @@ Data transformation between validated input and backend-ready primitives.
 | 3h | source_for, get_genes_for_build | The build-to-source routing and the one fetch-and-cache orchestration | [reference_genes.py](../src/pylocuszoom/reference_genes.py) |
 | 3h | ensembl_source, fetch_overlap_frames | Ensembl REST client | [ensembl.py](../src/pylocuszoom/ensembl.py) |
 | 3h | ucsc_source, fetch_track_frames | UCSC track client, used for CanFam3.1, CanFam4 and FelCat9 | [ucsc.py](../src/pylocuszoom/ucsc.py) |
-| 3h | gene cache | On-disk cache shared by both gene sources | [_gene_cache.py](../src/pylocuszoom/_gene_cache.py) |
-| 3j | enrich_with_ld | Calls PLINK for lead-SNP R² and merges it into the GWAS frame under one recovery policy | [_ld_plotting.py](../src/pylocuszoom/_ld_plotting.py) |
+| 3h | gene cache | Atomic gene/exon archive cache shared by both gene sources | [_gene_cache.py](../src/pylocuszoom/_gene_cache.py) |
+| 3j | _AssociationInput | Region-selected data and resolved per-panel options | [plotter.py](../src/pylocuszoom/plotter.py) |
+| 3j | enrich_with_ld | Calls PLINK for lead-SNP R² and assigns values by SNP ID while preserving selected rows | [_ld_plotting.py](../src/pylocuszoom/_ld_plotting.py) |
 | 3j | prepare_pvalue_data | Shared p-value intake: filtering, zero-value mode, finite `-log10` | [_data.py](../src/pylocuszoom/_data.py) |
 | 3j | prepare_eqtl_for_plotting | eQTL panel prep | [eqtl.py](../src/pylocuszoom/eqtl.py) |
-| 3j | calculate_colocalization_overlap | Colocalisation overlap between two association frames | [eqtl.py](../src/pylocuszoom/eqtl.py) |
-| 3j | add_snp_labels | SNP label placement and lead-proximity filtering | [labels.py](../src/pylocuszoom/labels.py) |
+| 3j | calculate_colocalization_overlap | Significant coordinate overlap on chromosome and absolute position | [eqtl.py](../src/pylocuszoom/eqtl.py) |
+| 3j | select_label_candidates | Shared lead-proximity eligibility for regional and standalone SNP labels | [_label_data.py](../src/pylocuszoom/_label_data.py) |
+| 3j | add_snp_labels | SNP label ranking and placement | [labels.py](../src/pylocuszoom/labels.py) |
 | 3j | liftover | CanFam3.1 to CanFam4 coordinate lift for recombination maps | [_liftover.py](../src/pylocuszoom/_liftover.py) |
 | 3j | UNSET, resolve_threshold | The significance-threshold sentinel every threshold-bearing plotter uses, which keeps `None` meaning "draw no line" | [_plotter_utils.py](../src/pylocuszoom/_plotter_utils.py) |
 | 3i | Regional panels | The five regional panel value types, each with the `draw` method that draws it, one per module | [panels/](../src/pylocuszoom/panels/) |

docs/CONFIGURATION.md

@@ -43,18 +43,20 @@ Implementation:
   `liftover` leaf holding downloaded chain files
 - [`_gene_cache.cache_root()`](../src/pylocuszoom/_gene_cache.py)
 
-You can also override the cache location explicitly by passing `output_dir`
-to `download_canine_recombination_maps()` / `ensure_recomb_maps()` — this
-bypasses the environment variables entirely.
+To pre-download maps into a chosen directory, call
+`download_canine_recombination_maps(output_dir="/path/to/maps")`.
+Passing `recomb_data_dir` to the plotter, or `data_dir` to the map helpers,
+selects a read-only caller directory. It never downloads or replaces files there,
+and its coordinates must already use the requested build. With no directory,
+`ensure_recomb_maps()` manages the platform cache and may download built-in maps.
 
 ## Programmatic Configuration (Pydantic Models)
 
-The user-facing API uses plain keyword arguments (`plot()`,
-`plot_stacked()`). Internally these kwargs are validated by frozen
-Pydantic models defined in
-[`src/pylocuszoom/config.py`](../src/pylocuszoom/config.py). You normally
-do not construct these directly, but they define the canonical set of
-options and their defaults.
+The plotting methods take frozen Pydantic values such as `ColumnConfig`,
+`DisplayConfig`, `LDConfig` and `PanelInputs`, defined in
+[`src/pylocuszoom/config.py`](../src/pylocuszoom/config.py). Region coordinates
+and per-panel overrides remain keyword arguments. The public values can be
+reused across calls; each call resolves its effective options before rendering.
 
 ### `RegionConfig` — genomic region (required)
 
@@ -176,8 +178,9 @@ If you need per-environment behaviour, do it at the caller level, e.g.:
 
 - Set `XDG_CACHE_HOME` / `LOCALAPPDATA` per machine to control where
   reference data is cached.
-- Pre-download reference data in CI with `ensure_recomb_maps()` pointing at
-  a shared directory, then set `output_dir=` accordingly at runtime.
+- Pre-download canine maps with `download_canine_recombination_maps(output_dir=...)`,
+  then pass that directory as `recomb_data_dir` to the plotter. Caller maps must
+  already use the requested genome build.
 - On Databricks, the `/dbfs/FileStore/reference_data/recombination_maps`
   path is selected automatically.
 

docs/DEVELOPMENT.md

@@ -64,7 +64,7 @@ The common development commands are:
 | `uv run pre-commit run --all-files` | Run the full pre-commit suite against every file in the repo. |
 | `uv build` | Build the wheel and sdist via hatchling into `dist/`. |
 | `uv run python examples/generate_example_plots.py` | Regenerate example plots shown in the README. |
-| `scripts/example_diff.sh [--keep]` | Regenerate the examples and list the exports whose content changed after id normalisation; the equivalence check for backend and panel changes. |
+| `scripts/example_diff.sh [--keep]` | Generate outside the checkout and compare exports with HEAD. Exit 1 means differences; exit 2 means failure. `--keep` accepts generated changes only when affected exports have no manual edits. |
 | `uv lock` | Refresh `uv.lock` after changing dependencies in `pyproject.toml`. |
 
 See [CONTRIBUTING.md](../CONTRIBUTING.md) for the full pre-commit and pre-PR checklists.

docs/TESTING.md

@@ -65,7 +65,9 @@ Tests live under `tests/`. Files follow the `test_*.py` naming convention and ma
 - `tests/test_data_intake.py` — the shared p-value intake policy
 - `tests/test_manhattan_plotter.py`, `tests/test_qq.py`, `tests/test_manhattan.py` — Manhattan/QQ coverage
 - `tests/test_stats_plotter.py`, `tests/test_phewas.py`, `tests/test_forest.py` — statistical plots
-- `tests/test_ld.py` — PLINK wrapper (driven through the `fake_plink` fixture; no real PLINK binary required)
+- `tests/test_ld.py` — PLINK command construction and species flags
+- `tests/test_ld_parsing.py` — single-lead and pairwise PLINK output parsing
+- `tests/test_ld_process.py` — PLINK file validation, executable discovery and process execution, using `fake_plink` without a real PLINK binary
 - `tests/test_backends.py` — the shared `PlotBackend` surface and the matplotlib backend
 - `tests/test_plotly_backend.py`, `tests/test_bokeh_backend.py` — the interactive backends
 - `tests/test_notebook_backends.py` — Plotly/Bokeh notebook compatibility, parametrised over both backends through `tests/figure_probes.py`
@@ -105,7 +107,7 @@ Hypothesis strategies shared across tests live in `tests/strategies.py`.
 ### Guidelines
 
 - **Assert on observable outputs, not mock call counts.** Check returned figures, DataFrame columns/shapes, written files, and raised exceptions. Reserve `assert_called_once_with` for true system boundaries (PLINK subprocess, HTTP, filesystem dispatch).
-- **Drive PLINK through `fake_plink`** — tests must not require a real PLINK installation. The `fake_plink` fixture in `conftest.py` patches `subprocess.run` and writes a real `.ld` file at the path the command asked for, so command construction, output parsing and the R2 merge all stay inside the test. Assert on the frame `calculate_ld` returns, not on what the mock received: a command flag is already pinned by `TestBuildLdCommand` and `TestBuildPairwiseLdCommand`, which call the pure builders and assert on the list they return.
+- **Drive PLINK through `fake_plink`** — tests must not require a real PLINK installation. The `fake_plink` fixture in `conftest.py` patches `subprocess.run` and writes a real `.ld` file at the path the command asked for, so command construction, output parsing and the R2 assignment all stay inside the test. Assert on the frame `calculate_ld` returns, not on what the mock received: a command flag is already pinned by `TestBuildLdCommand` and `TestBuildPairwiseLdCommand`, which call the pure builders and assert on the list they return.
 - **State a rendering behaviour once, not once per backend.** A fact about the figure (which marker, whether it hovers, whether it exports) belongs in one `@pytest.mark.parametrize("backend_name", INTERACTIVE_BACKENDS)` test reading a probe from `tests/figure_probes.py`. Hand-written per-backend twins drift: the bokeh eQTL marker test used to pass with the negative-effect glyph never drawn. A genuine library-specific regression still belongs in `test_plotly_backend.py` or `test_bokeh_backend.py`.
 - **Cover edge cases**: empty DataFrames, missing required columns, mismatched list lengths, single-SNP regions, and cross-chromosome filtering.
 - **Respect the 30s timeout.** If a test is legitimately slow, override with `@pytest.mark.timeout(60)` rather than raising the global default.