Robust on-disk artifacts: atomic .gvlfa cache + dataset creation/validation (closes #21)

[d-laub]

Summary

Makes GenVarLoader's generated on-disk artifacts — the .gvlfa FASTA cache and gvl.write dataset directories — self-describing, safe under concurrent creation, and resilient to format drift. Closes #21.

This PR landed in two stages: the robust .gvlfa FASTA cache, then the atomic-creation + dataset-validation + concurrency follow-ups it set up.

Stage 1 — Robust .gvlfa FASTA cache (self-describing, fingerprint-validated)

Replaces the brittle mtime-validated, sibling-only .fa.gvl flat FASTA cache with a self-describing .gvlfa/ directory cache that fingerprints its source, resolves it three ways, auto-rebuilds when stale/corrupt, and auto-migrates legacy caches — fully backwards compatible.

• New module python/genvarloader/_fasta_cache.py owns the on-disk format, validation, build, and migration behind a single entry point ensure_cache(path) -> (FastaCache, data_path).
  • .gvlfa/ holds metadata.json (pydantic FastaCache: format version, gvl version, contig lengths, source hints, fingerprint) + sequence.bin (numpy memmap of all contigs).
  • Fingerprint: blake2b of the first 1 MiB + total file size.
  • Three-way source resolution: sibling → relative → absolute.
  • Validity states: fresh / stale / unvalidated; auto-rebuild on stale or size-corrupt; format-too-new raises consistently from both entry points (never silently downgrades).
  • Legacy migration: reuses .fa.gvl bytes via move, but only after verifying the legacy byte count matches the current source — a stale/truncated legacy cache is left untouched and rebuilt fresh.
• Fasta (_fasta.py) and Reference.from_path (_dataset/_reference.py) rewired to ensure_cache; both now also accept a .gvlfa directory directly as their path.
• Old _valid_cache / _write_to_cache / _get_sequences machinery removed.

This intentionally mirrors the existing SvarLink robustness idiom (fingerprint + three-way resolution + legacy migration) already used for .svar back-references in dataset metadata.

Stage 2 — Atomic creation, dataset validation, and concurrency safety (closes #21)

• New single-responsibility primitive python/genvarloader/_atomic.py — atomic_dir(dest, *, overwrite, lock, timeout) builds each artifact into a private sibling temp dir and publishes it with an atomic os.replace. A best-effort filelock avoids N redundant concurrent builds but is never load-bearing for correctness — the atomic rename is the guarantee, so a lock timeout or a network-FS no-op just means "build anyway". SkipPublish aborts publishing to reuse an already-valid dest; overwrite=True uses move-aside-then-rename. Reused by both artifacts.
• FASTA cache (_fasta_cache.py) — build/migrate_legacy now publish through atomic_dir; ensure_cache rebuild paths go through a locked, double-checked helper (_ensure_built) so concurrent builders don't all rebuild and never corrupt. The cache auto-rebuilds (source available).
• gvl.write (_dataset/_write.py) — the whole dataset is built into a temp dir and published atomically (with atomic_dir(...)), so an interrupted or racing write never leaves a partial dataset. Metadata gains a format_version field (default None for back-compat) and the module records DATASET_FORMAT_VERSION = 1.0.0.
• Validation on open (_dataset/_validate.py + _open.py) — Dataset.open now runs validate_dataset: a format-version major gate (incompatible / too-new / too-old → actionable ValueError; missing → treated as 1.0.0) plus structural/size integrity (required files present; regions.npy shape (n_regions, 4); genotype offsets.npy byte-length matches n_regions·ploidy·n_samples+1). Datasets never auto-rebuild (no retained source); they raise telling you to regenerate with gvl.write.
• Out-of-scope (documented): genoray .gvi and pysam .fai/.gzi index files are created by those libraries and are not made atomic/locked by gvl.

Adds a filelock>=3.12 dependency. Skill (skills/genvarloader/SKILL.md) updated to document .gvlfa support, atomic/locked creation, the dataset format gate, and the index-file limitation.

Test plan

• tests/unit/test_fasta_cache.py — fingerprint, source resolution, build/load round-trip, byte-equality vs pysam, stale/unvalidated/format-too-new/corrupt classification, legacy migration (incl. stale-bytes guard), ensure_cache decision matrix, atomic-publish + no-partial-cache + double-check reuse.
• tests/unit/test_fasta.py — .gvlfa creation, direct .gvlfa input, missing-source on-demand read error, legacy migration, in-memory-no-cache path, Reference.from_path .gvlfa round-trip.
• tests/unit/test_atomic.py (8) — clean publish, sibling temp, exception cleanup, FileExistsError, overwrite replace, SkipPublish, lock-file persistence, concurrent-loser-discarded.
• tests/unit/dataset/test_validate.py (9) — valid pass, missing/too-new/too-old format version, missing/wrong-shape regions.npy, genotype offsets wrong/correct length, genotypes-without-ploidy.
• tests/unit/dataset/test_write_atomic.py (7) — format_version field/constant/round-trip, on-disk stamp, atomic no-temp-left, failure leaves no partial artifacts, overwrite-false raises.
• tests/unit/test_concurrency.py (2, @slow) — the Lock GVL files to avoid multiple file creation/deletion in multi-job settings #21 regression: N processes building the same .gvlfa cache produce a byte-identical result with no orphans; N processes writing the same dataset path (overwrite=True) leave exactly one valid, openable dataset with no orphans.
• Full fast suite green: 584 passed, 39 skipped, 4 xfailed, 0 failures; slow concurrency tier: 2 passed.
• ruff check + ruff format clean on all touched files. (Pre-existing pyrefly errors in _bigwig.py/_flat.py/_ragged.py, plus import-resolution false-positives on _fasta_cache/new modules, are unrelated — the Rust ext and seqpro stubs aren't resolvable in the hook env.)

🤖 Generated with Claude Code