Repository navigation
feat(preprocess): tile-internal chunking + mmap accumulator (v1.9.0) - #8
Merged
Merged
Conversation
Closes v1.8.4's third v1.6 blocker (memory ceiling at 10 m + 100 km
buffer on the GB10's 119 GB unified memory) end-to-end across both
preprocess and rank stages.
Preprocess side:
- derive_horizon_profile gains azimuth_chunk_size + out= parameters
so the per-tile working buffer can be capped at chunk_size azimuths
(~17 GB at 10 m, chunk=9) instead of the full 70 GB output array.
- run_tiled_per_region opens the accumulator with
np.lib.format.open_memmap against an on-disk .npy file at fine
resolutions, so the per-tile output never materialises in unified
memory.
- default_azimuth_chunk_size heuristic auto-detects the chunk: no
chunking at v1.5 resolutions (240/80/20m for 8 of 9 polygons),
chunk=9 at 10m, chunk=2 at 5m.
- Cache format split: at fine resolutions, write
<base>.npy (mmap-able bulk array) + <base>.meta.npz (small
metadata sidecar). Legacy single-file <base>.npz still written at
v1.5 resolutions. New existing_horizon_cache_path() resolver picks
whichever pair is on disk.
Rank side:
- _load_horizon_profile_npz mmap-loads the .npy form without
decompressing — without this, the rank stage would re-OOM by
materialising the 70 GB compressed array at load time even though
preprocess avoided it.
- compute_earth_visibility_fraction gains row_chunk_size: iterates
over y-slabs (~1500 rows at 10m, ~5 GB working buffer per slab),
uploads slab to GPU, runs the libration sweep, calls
madvise(MADV_DONTNEED) between slabs to evict OS page cache and
keep steady-state pressure tractable.
- lat/lon/grid-convergence downcast to float32 (sub-cm precision
over a 220 km tile, well under 10 m). Halves persistent rank-tile
memory.
- --lola-source flag mirrored from v1.8.3's preprocess so a single
PGDA mosaic flows through both stages without renaming files.
Numerically identical to the v1.5 single-pass path: 21 synthetic-data
correctness tests covering chunked-vs-unchunked equivalence (per-azimuth
preprocess and per-row rank), mmap round-trips, cache-resolver fallback
behaviour, and CLI flag plumbing.
10 m end-to-end diagnostic on the 3 cleanly-covered USGS polygons
(HW, SP, G2):
- 18 selene sites (vs 20 at v1.5 20m); HW 10/SP 7/G2 1.
- 11/18 (61.1%) matched within 5 km of an in-scope Wueller site
(vs 13/20 = 65.0% at 20m).
- Wueller-side 11/29 matched (37.9%, identical at 20m and 10m).
- Median matched-pair distance 1.29 km (vs 1.88 km at 20m,
-31% tightening). Per-region medians: HW 1.16 km (vs 1.77,
-34%); SP 1.77 km (vs 2.03, -13%); G2 still 0/1 matched.
- Wall-clock on GB10: preprocess 38m, rank 28m, compare 3s.
v1.6 itself remains deferred only on blocker 1 (polygon coverage; 3
of 9 polygons fit at 10m). v1.9 closes the engineering gap and
demonstrates the methodology converges (tightens) at 10m where data
is available. Methodology, default weights, and v1.5/v1.7/v1.8
headline numbers (56/69 = 81.2%, median 1.69 km, full 9-region 20m
run) unchanged.
Test count: 412 collected, 403 passed (+8 new), 9 skipped (LEND +
Robbins data-gated).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the third v1.6 blocker (memory ceiling at 10m + 100km buffer on
GB10's 119 GB unified memory) end-to-end across preprocess and rank.
derive_horizon_profile'snew
azimuth_chunk_size+out=params (~17 GB peak at 10m,chunk=9).
run_tiled_per_regionwrites throughnp.lib.format.open_memmapso per-tile output never materializes inunified memory. Cache split:
<base>.npy(mmap-able) +<base>.meta.npz(sidecar) at fine resolutions; legacy single-file.npzpreserved at v1.5 resolutions._load_horizon_profile_npzmmap-loads.npyform withoutdecompressing.
compute_earth_visibility_fractiongainsrow_chunk_sizey-slab iteration withmadvise(MADV_DONTNEED)between slabs. lat/lon/grid-convergence downcast to float32
(sub-cm precision over a 220 km tile).
--lola-sourceflag mirroredfrom v1.8.3's preprocess.
tests/test_per_azimuth_chunking.pycover chunked-vs-unchunkedequivalence (preprocess + rank), mmap round-trips, cache-resolver
fallback, CLI plumbing.
10m diagnostic (HW, SP, G2 — 3 cleanly-covered USGS polygons)
Wueller site (vs 13/20 = 65.0% at 20m on the same subset).
-31% tightening). HW: 1.16 km (-34%). SP: 1.77 km (-13%).
v1.6 remains deferred only on blocker 1 (polygon coverage). v1.5/v1.7/
v1.8 headline numbers (56/69 = 81.2%, 1.69 km median, full 9-region
20m) are unchanged — v1.9's 10m result is supporting evidence on a
3-region subset, not a replacement headline.
Test count: 412 collected, 403 passed (+8 new), 9 skipped (LEND +
Robbins data-gated).
Test plan
v1.5 single-pass within float32)
rebase-and-merge (NOT squash — `v1.9.0` tag already points
at `9f669ef` and squash would orphan it)
🤖 Generated with Claude Code