diff --git a/.agent/work-plans/issue-160/plan.md b/.agent/work-plans/issue-160/plan.md new file mode 100644 index 0000000..34aaac7 --- /dev/null +++ b/.agent/work-plans/issue-160/plan.md @@ -0,0 +1,296 @@ +# Plan: Consumer-side overview pyramid + bounded eviction for live sonar coverage + +## Issue + +https://github.com/rolker/camp/issues/160 + +## Context + +`SonarLiveCacheLayer::tiles_` is unbounded today — the only cap is reconciler +prune-on-absence. During a sustained live survey the map grows monotonically until +RAM/VRAM is exhausted (the salmon crash, #153). A naive fix (evict fine tiles) makes +zoomed-out views blank over evicted areas. The overview pyramid is the graceful-degradation +companion: before dropping a fine tile, decimate it into its coarse parent so coverage +degrades to lower resolution instead of disappearing. + +ADR-0006 makes this safe: live tiles are display-grade previews, always rebuildable from +the boat's durable stores. Building downsampled overviews is display-LOD generation with +no provenance burden. + +The work spans two repos: +- **unh_marine_autonomy** (GGGS `parent()`/`children()` helpers) — Phase A, prerequisite PR +- **camp** (eviction + overview pyramid) — Phase B, depends on Phase A + +**Sub-decisions resolved** (operator checkpoint 2, 2026-07-01 → applied here): +- Eviction trigger: **view-based LOD** — a configurable byte budget for VRAM accounting; + when over budget, evict the fine tiles **farthest from the current viewport centre** + (classic slippy-map/LOD behaviour — keep what's near the view, discard the rest), with + pure LRU as the fallback when no view is available (headless / tests). **No + vessel-position / tf dependency** (operator dropped distance-from-vessel as needless + complication). Self-account locally (camp knows every texture size); adopt + `ResourceMonitor` VRAM feed when #155 lands (the plan leaves a named integration seam). +- Overview chain depth: full chain from received fine level all the way to level 0 (one + whole-survey tile). Levels are 4× smaller per step; the full chain is a handful of + tiles — cheap enough to always keep resident. +- Fold timing: fold-on-evict only (cheapest). The overview is updated only when a fine + tile is actually dropped. The overview may lag the latest fine data until first eviction; + this is acceptable per ADR-0006 D1 (display-grade, rebuildable). +- Reconciler on evict: **REVISED as-built → keep `markHave`, do NOT `drop`** (see the + "Implementation notes" section). Dropping would make the next catalog reconcile + re-request the evicted tile → re-receive → re-evict = churn wasting bandwidth (#71). + `tiles_` tracks residency (memory); the reconciler tracks possession (disk). + +## Approach + +### Phase A — GGGS index math helpers (unh_marine_autonomy, separate PR) + +**Phase A lands as its own standalone `unh_marine_autonomy` issue + PR** (title ~"Add +GGGS `parent()`/`children()` index-math helpers"), tagged **`Part of rolker/camp#160`** in +the body, merged and the underlay rebuilt before camp Phase B can build against it. + +1. **Add `marine_autonomy/gggs/index_math.h`** — two free functions: + - `GridIndex parent(const GridIndex& child)` — computes the child cell centre from + `child.northLatitude()/southLatitude()/eastLongitude()/westLongitude()` (GridIndex has + **no** `center_lat`/`center_lon` accessor — verified `gggs/grid_index.h:87-102`), then + `Level(child.level()-1).gridIndex(center_lat, center_lon)` to handle polar + `latitudeScaleFactor` (1/3/9) column-scale changes at the 72°/80° latitude + boundaries correctly, without manual column arithmetic. Returns an invalid + `GridIndex` when `child.level() == 0` or the child is invalid. + - `std::vector children(const GridIndex& parent)` — returns the 2–4 + child GridIndex values at `parent.level()+1` whose geographic extent overlaps + the parent. In the normal latitude bands this is always 4 (2 rows × 2 cols); + at polar-scale-change boundaries the child rows may cross a scale-factor + transition, so the child column range is computed from the parent's west/east + longitudes rather than from a fixed ×2 multiplier. Returns empty when + `parent.level() == 20` or the parent is invalid. + +2. **Add `marine_autonomy/test/test_gggs_index_math.cpp`** — tests: + - `parent()` at equatorial, sub-polar (72°), and polar (80°) band boundaries. + - Round-trip: `parent(c)` equals `parent(sibling)` for two children of the same + parent. + - `parent()` at level 0 returns invalid. + - `children()` of a given parent contains exactly the tiles whose `parent()` maps + back to that parent. + - `children()` at level 20 returns empty. + +3. **Update `marine_autonomy/gggs.h`** — add `#include "gggs/index_math.h"` so + callers get both helpers with the existing single include. + +4. **Update `marine_autonomy/CMakeLists.txt`** — add `ament_add_gtest` for + `test_gggs_index_math` linked to `${PROJECT_NAME}`. + +### Phase B — Eviction + overview pyramid (camp, PR 2, depends on Phase A) + +5. **Add camp ADR-0010** (`docs/decisions/0010-bounded-eviction-overview-pyramid.md`) — + records the eviction + overview design decisions: trigger (distance + byte budget), + overview chain depth (full to level 0), fold timing (on-evict), reconciler interaction + (drop on evict), overview isolation from reconciler. Supersedes the TODO in ADR-0006 + consequences. Add an ADR-0006 addendum cross-reference per ADR-0012. + +6. **Add `OverviewEntry` struct and `overview_tiles_` map** to `SonarLiveCacheLayer`: + ```cpp + struct OverviewEntry { + SonarLiveTile tile; + std::unique_ptr texture; + bool texture_dirty = true; + }; + std::map overview_tiles_; + ``` + Overview tiles are indexed by their coarse `GridIndex` and are **never** entered in + `tiles_` or touched by `reconciler_`. The dtor tears down `overview_tiles_` textures + under the renderer context the same way it handles `tiles_`. + +7. **Add eviction budget state** to `SonarLiveCacheLayer`: + - `size_t vram_budget_bytes_` — loaded from + `QSettings("LiveTileCache/max_vram_bytes")` in the constructor; default 512 MiB. + - `uint64_t access_seq_` — monotonic counter; each `Entry` carries a + `uint64_t last_access_seq` bumped in `handleTile()` and `textureFor()` (the LRU + fallback ordering when no view is available). + - **No vessel-position state** — eviction ordering uses the current viewport centre + (below), obtained on demand from `scene()->views()`; no ROS position subscription. + - `size_t accountedBytes() const` — sum of `width * height * 4` (R32F) across all + `Entry` textures in `tiles_` that have a non-null texture. + +8. **Implement `evictIfOverBudget()`** (GUI thread): + - Compute `accountedBytes()`. If under `vram_budget_bytes_`, return immediately. + - Build a sorted eviction candidate list: entries in `tiles_` ordered by distance of + the tile's scene-space centre from the **current viewport centre** (farthest first). + Obtain the viewport centre in Web-Mercator scene coords via + `scene()->views().first()->mapToScene(viewport rect).boundingRect().center()`. When + no view is attached (headless / tests), fall back to LRU order by `last_access_seq` + (oldest first). + - For each candidate until under budget: + - Call `foldIntoParent(entry.tile)` (step 9). + - Call `scheduleWriteThrough(entry.tile)` (persist before drop, reusing the + existing coalesced write path). + - Free the GL texture under the renderer context (reuse the `hasContext()` / + `makeCurrent()` guard from `handleCatalog()`). + - Erase from `tiles_` (keep the reconciler `markHave` — revised as-built, see notes). + - Call `evictIfOverBudget()` at the end of `handleTile()` and after `warmLoad()`. + - Log (qWarning) when entering the shed-load path so the operator can see it in + the console. + +9. **Implement `foldIntoParent(const SonarLiveTile& fine)`** (GUI thread): + - Compute `parent_idx = gggs::parent(fine.index())`. If invalid (level 0), return. + - Look up or create an `OverviewEntry` in `overview_tiles_[parent_idx]`: + - On first creation: `SonarLiveTile(parent_idx, fine.width(), fine.height())` — the + parent tile **matches the fine tile's `width`/`height`** (operator decision: + standard pyramid where every tile is the same pixel size, so a parent W×H tile + covers the area of its 4 children at half linear resolution). + - Determine the child's **quadrant** within the parent (NW/NE/SW/SE) from the child's + row/col parity relative to `parent(child)`'s children (or from the child vs parent + geographic bounds). 2×2 mean-decimate the fine `W×H` band into the corresponding + `W/2 × H/2` quadrant of the parent band, skipping NoData cells (a 2×2 block that is + all NoData stays NoData). Mark the overview `texture_dirty = true`. + - Schedule write-through for the updated overview tile, writing to a sub-directory + `overviews/` within `cache_dir_` so warm-load of fine tiles cannot accidentally pick + up overview files. **Small API change (review-plan finding):** `scheduleWriteThrough` + / `startWriteThrough` currently hard-target `cache_dir_` root + (`sonar_live_cache_layer.h:151`); add an optional destination-subdir parameter (default + "" = root for fine tiles, "overviews" for parents) threaded through the write-state + bookkeeping so overview and fine writes stay serialized on distinct paths. + - Recurse: `foldIntoParent(overview_tiles_[parent_idx].tile)` to build the full + chain to level 0. + +10. **Bound `warmLoad()`**: + - Load overview tiles from `/overviews/__.tif` into + `overview_tiles_` before loading fine tiles (overviews are kept resident, never + evicted). + - After loading fine tiles into `tiles_`, call `evictIfOverBudget()` to trim to the + budget before activating the subscription. + +11. **LOD fallback in `items()`**: + - After collecting the normal `tiles_` entries, for each `OverviewEntry` in + `overview_tiles_`: emit a `RasterFieldItem` only if no finer `tiles_` entry + covers the same area (check by level comparison — if any `tiles_` entry with a + higher level number overlaps this overview's bounds, skip the overview). In + practice this means: include an overview only when at least one of its children + is absent from `tiles_`. + - `recomputeBounds()` must union extents from both `tiles_` and `overview_tiles_` + so the bounding rect stays correct after fine tiles are evicted. + - `foldAutoRange()` must also fold overview bands into `data_min_`/`data_max_` so + the colormap range tracks the full visible extent. + +12. **Update `updateDisplay()`**: + - Emit `"(live: %1 fine + %2 overview tiles)"` so the operator can see both counts. + +13. **Extend `test_sonar_live_cache.cpp`**: + - Eviction bounding: insert tiles up to 2× the byte budget; verify that + `tiles_.size()` stays within the budget and evicted tiles appear in + `overview_tiles_`. + - Fold-into-parent decimation: verify 2×2 mean of known values; verify NoData + propagation. + - Render fallback: after evicting a fine tile, verify that `items()` returns an + overview item covering its area. + - `warmLoad()` bounding: pre-populate a cache dir with more tiles than the budget; + verify warm-load trims to budget. + - Reconciler isolation: verify that `overview_tiles_` keys are never present in + `reconciler_`'s held set after fold. + +## Files to Change + +| File | Change | +|------|--------| +| `core_ws/.../marine_autonomy/include/marine_autonomy/gggs/index_math.h` | New: `parent()` and `children()` free functions | +| `core_ws/.../marine_autonomy/include/marine_autonomy/gggs.h` | Add `#include "gggs/index_math.h"` | +| `core_ws/.../marine_autonomy/test/test_gggs_index_math.cpp` | New: unit tests for both helpers | +| `core_ws/.../marine_autonomy/CMakeLists.txt` | Register `test_gggs_index_math` target | +| `ui_ws/src/camp/docs/decisions/0010-bounded-eviction-overview-pyramid.md` | New camp ADR-0010 | +| `ui_ws/src/camp/docs/decisions/0006-live-tile-cache-persistence.md` | Addendum cross-ref to ADR-0010 | +| `ui_ws/src/camp/src/camp_map/ros/live_coverage/sonar_live_cache_layer.h` | `OverviewEntry`, `overview_tiles_`, budget/position state | +| `ui_ws/src/camp/src/camp_map/ros/live_coverage/sonar_live_cache_layer.cpp` | `evictIfOverBudget()`, `foldIntoParent()`, bounded `warmLoad()`, LOD fallback in `items()` | +| `ui_ws/src/camp/test/test_sonar_live_cache.cpp` | Extended tests per step 13 | + +## Principles Self-Check + +| Principle | Consideration | +|---|---| +| Human control and transparency | Budgets via QSettings (ADR-0006 D2 / camp#117); status line shows fine + overview tile counts; qWarning logged on force-evict. | +| Capture decisions, not just implementations | ADR-0010 records the four open sub-decisions before implementation. ADR-0006 addendum per ADR-0012. | +| A change includes its consequences | `recomputeBounds()` and `foldAutoRange()` extended to cover overview tiles; existing test suite stays green; reconciler isolation tested explicitly. | +| Only what's needed | Fold-on-evict only (not fold-on-arrival). `children()` added to GGGS for completeness and future boat-side reuse, but camp only calls `parent()`. | +| Improve incrementally | Two-PR structure: GGGS helper is independently testable before the camp consumer lands. | +| Test what breaks | Tests target the crash scenario (unbounded growth) and the blank-zoom-out regression, not framework glue. | + +## ADR Compliance + +| ADR | Triggered | How addressed | +|---|---|---| +| camp ADR-0006 (live tile cache persistence) | Yes | Overview tiles use same `__.tif` format in `overviews/` sub-dir. Overview tiles stay out of reconciler (correctness constraint). Eviction reuses `scheduleWriteThrough` persist-then-drop path. | +| camp ADR-0006 D2 (no hardcoded defaults) | Yes | `max_vram_bytes` defaults to 512 MiB but overridable via QSettings. | +| camp ADR-0006 D4 (GUI-thread invariant) | Yes | Eviction, folding, and GL texture release all run on GUI thread. ROS position callback marshals to GUI thread. | +| camp ADR-0001 (threading) | Yes | No new thread-safety concerns; eviction callbacks are queued to GUI thread same as tile/catalog callbacks. | +| workspace ADR-0012 (cross-ref addendum) | Yes | ADR-0006 addendum is navigational (points to ADR-0010); substantive decisions go in ADR-0010. | + +## Consequences + +| If we change... | Also update... | Included in plan? | +|---|---|---| +| `tiles_` eviction | keep reconciler `markHave` (possession); NOT `drop()` — avoids re-request churn (revised as-built) | Yes (step 8) | +| Overview tile write (overviews/ sub-dir) | `warmLoad()` must load overviews from that sub-dir on re-enable | Yes (step 10) | +| `recomputeBounds()` now unions overview tiles | `foldAutoRange()` must also fold overview tile ranges | Yes (step 11) | +| Camp depends on `gggs::parent()` | unh_marine_autonomy Phase A must land and underlay be rebuilt before camp Phase B can build | Yes — two-PR structure; Phase A merges first | +| `#155 ResourceMonitor` lands | Replace `accountedBytes()` self-accounting with `ResourceMonitor` VRAM feed; the `vram_budget_bytes_` knob stays the same | Not in this PR — named integration seam (leave a `// TODO(#155)` comment) | + +## Open Questions (resolved at operator checkpoint 2, 2026-07-01) + +- [x] **Eviction ordering / vessel source** → **view-based LOD, no vessel source.** Evict + fine tiles farthest from the current viewport centre (via `scene()->views()`), pure LRU + fallback when headless. No ROS position subscription / tf dependency. (Operator: distance- + from-vessel needlessly complicates; use traditional LOD — discard those farthest from the + viewpoint.) +- [x] **Overview resolution** → **match the fine tile's `width`/`height`.** Standard pyramid: + every tile is the same pixel size; a parent W×H tile covers 4 children at half linear + resolution, each child decimated into its W/2×H/2 quadrant. + +## Estimated Scope + +Two PRs: +- **Phase A (unh_marine_autonomy)**: ~80 lines (header + tests + CMake) — small, independent. +- **Phase B (camp)**: ~500–700 lines across 4 files — medium; the bulk is + `evictIfOverBudget()`, `foldIntoParent()`, and the test extensions. + +## Implementation notes (as-built) + +Deviations from the approach above, made during implementation and kept in sync here: + +- **`Entry` reused for overviews** (not a separate `OverviewEntry`): the structs are + identical, so `overview_tiles_` uses `Entry` and `textureFor()` serves both — DRY. + `Entry` gained a `last_access_seq` (LRU fallback stamp). +- **`SonarLiveTile::foldChild(child)`** encapsulates the decimation (the tile owns + `bands_`/`refoldRange`). It **area-maps** each child cell to the parent cell + containing its geographic centre (not a fixed 2×2 quadrant), so the polar + `latitudeScaleFactor` and any non-2×2 subdivision are handled generally. +- **LOD fallback by draw order** (D5): `items()` emits overviews first (coarse→fine, + natural `std::map` order) then fine tiles on top — simpler and more robust than a + per-overview occlusion check; fine covers its parent, evicted areas show the parent. +- **Eviction keeps `reconciler_.markHave` (does NOT `drop`)**: dropping would make the + next catalog reconcile re-request the evicted tile → re-receive → re-evict = churn + wasting bandwidth (#71). `tiles_` tracks residency (memory); the reconciler tracks + possession (disk). Known limitation: an evicted fine tile is not re-loaded from disk + on pan-back within a session (no view-change→reload hook yet) — deferred follow-up; + the coarse overview covers it meanwhile. (Recorded in ADR-0010 D2.) +- **Incremental `warmLoad`** (not `SonarLiveTile::loadCacheDir`, which materializes the + whole cache in one vector): loads fine tiles one file at a time, skips + already-resident indices (memory ≥ disk, so no displaced-texture context dance), and + trims every 64 inserts — bounding the load *peak*, not just steady state (the salmon + crash was warm-load slurping the cache). Overviews warm-load from `overviews/` with a + per-file level probe (they span multiple coarse levels). +- **Tests**: `foldChild` decimation + NoData in `test_sonar_live_cache.cpp` (tile-level, + no new deps); layer-level warm-load-bounding + overview-build in a new + `test_sonar_live_eviction.cpp` (offscreen-QApplication + `Map` harness, GL/ROS-free). + camp suite: 158 tests, 0 failures. Render-fallback draw order and the distance-based + (viewed) eviction ordering are covered by design + the SIM-VERIFY gate that + live-coverage changes carry, not headless unit tests (no GL / no attached view). +- **Public introspection accessors** `residentTileCount()` / `overviewTileCount()` / + `reconcilerHeldCount()` added (tests + the future #158 status indicator). +- **Overview bounding (pre-push review must-fix)**: the overview pyramid is NOT + unbounded/resident-forever. `accountedBytes()` counts fine **and** overview tiles, and + `evictIfOverBudget()` is **two-phase** — evict fine tiles first (folding into parents), + then, if still over, evict the numerous near-fine overview tiles farthest-from-view, + always protecting the coarse apex (`level <= kApexProtectLevel`, =6) so whole-survey + zoom-out stays covered. Total residency is bounded (fine + near-view overviews to the + budget + O(small) apex). Eviction no longer writes (tiles are already persisted at + patch/fold time — fixes the warm-load write-back storm). See ADR-0010 D1/D2/D3 + the + deferred follow-ups (pan-back reload, prune-retraction overview cleanup, high-latitude + foldChild seam). diff --git a/.agent/work-plans/issue-160/progress.md b/.agent/work-plans/issue-160/progress.md new file mode 100644 index 0000000..9d12d12 --- /dev/null +++ b/.agent/work-plans/issue-160/progress.md @@ -0,0 +1,130 @@ +--- +issue: 160 +--- + +# Issue #160 — Consumer-side overview pyramid + bounded eviction for live sonar coverage + +## Issue Review +**Status**: complete +**When**: 2026-07-01 00:50 +00:00 +**By**: Claude Code Agent (Claude Sonnet) + +**Issue**: #160 +**Comment**: (best-effort post follows this entry; not recorded inline) +**Scope verdict**: well-scoped + +### Actions +- [ ] Record eviction policy + pyramid design decisions via camp ADR-0006 addendum (ADR-0012 carve-out) or new camp ADR — the eviction trigger metric (distance-from-vessel + byte budget vs. tile-count), fold timing (fold-on-evict vs. fold-on-arrival), and overview chain depth are open sub-decisions that ADR-0001 requires to be captured before or during implementation. +- [ ] Clarify reconciler interaction on LRU-evict: does the evicted tile stay in the reconciler's `tiles_` map (as disk-backed) or get `drop()`d? This affects whether camp re-requests tiles from the boat after LRU eviction — plan-task should settle this and record it. +- [ ] Plan multi-repo worktree: changes span `camp` (eviction + pyramid + LOD) and `unh_marine_autonomy` (GGGS `parent(GridIndex)` helper); use `--packages camp,unh_marine_autonomy` or sequence a GGGS PR first and a camp PR consuming it — plan-task should decide. +- [ ] Threading: verify eviction + overview fold happen on the GUI thread (ADR-0006 D4 invariant: reconciler and tile map are not thread-safe). + +### Orchestrator note (checkpoint 1 resolved by operator, 2026-07-01) +- **GGGS `parent()` helper: ADD IT NOW (cross-repo).** Operator chose the dedicated + `parent(GridIndex)`/`children()` primitive in `unh_marine_autonomy/gggs` (handling the + polar 1/3/9 column scaling) over a camp-local geographic round-trip, so the boat can reuse + it later for producer-side overviews. **plan-task**: plan the multi-repo coordination — + recommended sequencing is a small standalone GGGS-helper PR (own issue in + `unh_marine_autonomy`, `Part of rolker/camp#160`) merged + underlay rebuilt first, then the + camp PR consuming it. Settle the remaining open sub-decisions (eviction trigger metric, + fold timing, overview chain depth, reconciler-on-evict semantics) with the issue's stated + leanings (distance-from-vessel + byte budget; full overview chain) and record them in the + ADR-0006 addendum / new camp ADR. + +## Plan Authored +**Status**: complete +**When**: 2026-07-01 13:30 +00:00 +**By**: Claude Code Agent (Claude Sonnet) + +**Plan**: `.agent/work-plans/issue-160/plan.md` at `d523a1d` +**Branch**: feature/issue-160 at `d523a1d` +**Phases**: 2 (Phase A: unh_marine_autonomy GGGS helpers PR; Phase B: camp eviction + overview PR) + +### Open questions +- [ ] Vessel position source: which ROS topic does camp subscribe to for operator vessel position — if not yet wired to SonarLiveCacheLayer, fall back to view-center distance or pure LRU. +- [ ] Overview tile resolution: fixed 64×64 per overview tile (caps memory, simpler) vs. matching the fine tile's width/height? + +## Plan Review +**Status**: complete +**When**: 2026-07-01 01:26 +00:00 +**By**: Claude Code Agent (Claude Opus) + +**Plan**: `.agent/work-plans/issue-160/plan.md` at `d523a1d` +**PR**: PR-less (`--issue 160`, layer worktree `issue-camp-160`) +**Verdict**: approve-with-suggestions + +Independent review (fresh-context sub-agent, Opus; plan authored by a Sonnet +sub-agent). Not annotated as author self-review: the name-based detection is +degenerate here (all Claude agents share the name "Claude Code Agent"), and the +"in-context — author self-review" tag would be factually false for an +out-of-context, different-model review. + +### Findings +- [ ] (suggestion) `parent()` sketch calls `Level(child.level()-1).gridIndex(center_lat, center_lon)`, but `GridIndex` has no `center_lat`/`center_lon` accessors — compute the center from `northLatitude()/southLatitude()/eastLongitude()/westLongitude()` (verified in `gggs/grid_index.h:87-102`) — `plan.md:43` +- [ ] (suggestion) Phase A "separate PR" doesn't name a separate `unh_marine_autonomy` issue; the issue-review orchestrator note calls for a standalone GGGS-helper issue tagged `Part of rolker/camp#160` — state it explicitly — `plan.md:40`, `plan.md:227` +- [ ] (suggestion) `scheduleWriteThrough(const SonarLiveTile&)` currently writes to `cache_dir_` root (`sonar_live_cache_layer.h:151`); folding overviews into an `overviews/` sub-dir needs a destination/subdir parameter — spell out that small API change — `plan.md:129` +- [ ] (suggestion) Vessel-position source (open question) materially drives the eviction ordering in `evictIfOverBudget()`; resolve it or commit to the LRU/view-center fallback before implementing step 8, rather than mid-implementation — `plan.md:216` +- [ ] (suggestion) `children()` is added to GGGS but not consumed by camp ("Only what's needed" tension); operator-sanctioned per the checkpoint note and unit-tested, so acceptable — no change required, noted for the record — `plan.md:190` + +## Local Review (Pre-Push) +**Status**: complete +**When**: 2026-07-01 03:34 +00:00 +**By**: Claude Code Agent (Claude Opus) +**Verdict**: changes-requested + +**Branch**: feature/issue-160 at `479071a` +**Mode**: pre-push +**Depth**: Deep (reason: new ADR + ~700 LOC memory-lifecycle/concurrency-critical caching) +**Must-fix**: 1 | **Suggestions**: 9 +**Round**: 1 | **Ship**: continue — one genuine design/honesty concern (overview footprint is unbudgeted, contradicting the ADR's boundedness claim); address before pushing as "closes #153" + +Static analysis (ament_cpplint): no new actionable findings — only camp house-style +(Allman `else`, `CAMP_ROS_` guards, no copyright headers, C++-first includes, C-style +numeric casts) applied consistently with surrounding code, plus one trivial new +`kOverviewSubdir` `std::string`-vs-`char[]` nit. camp does not enforce stock ament_cpplint. +Cross-repo dep satisfied: GGGS `parent()`/`children()` (unh_marine_autonomy #249/#250) +merged into the underlay `jazzy` and present. Adversarial: 2 disjoint-lens Claude passes. +Concurrency / GL-texture lifetime / shutdown-join ordering all verified sound. + +### Findings +- [ ] (must-fix) Overview pyramid is excluded from the budget and never evicted, growing ≈O(survey area) at fine-tile byte size (recurses to level 0; `accountedBytes()` sums only `tiles_`) — contradicts ADR-0010's "no longer grow memory/VRAM without bound" / "closes #153"; cross-confirmed by both adversarial lenses + governance. Correct the ADR-0010 Consequences + ADR-0006 addendum and cap/evict overviews or open a tracked follow-up — `docs/decisions/0010-bounded-eviction-overview-pyramid.md:116`, `sonar_live_cache_layer.h:225` +- [ ] (suggestion) `handleCatalog` prune erases `tiles_`/disk/reconciler but never `overview_tiles_` (or `overviews/` on disk) — retracted regions leave stale/orphaned overview coverage + slow leak (compounds the must-fix) — `sonar_live_cache_layer.cpp:413` +- [ ] (suggestion) `accountedBytes()` recomputed per eviction-loop iteration + in the sort; bounded by trim-every-64 (so ~linear in N, not O(N²)), but a running incremental byte total removes the rescans — `sonar_live_cache_layer.cpp:562` +- [ ] (suggestion) `evictIfOverBudget()` after warm-load re-writes just-read, byte-identical fine tiles back to disk; guard persist-then-drop with a dirty flag to avoid the write-back storm on enabling a large cache — `sonar_live_cache_layer.cpp:332` +- [ ] (suggestion) `foldChild` assumes the parent geographically contains the child; across GGGS latitude-band boundaries (±72°/±80°) bounds guards prevent corruption but child cells whose centre falls outside the parent are silently dropped → possible overview gaps on high-latitude surveys. Add a boundary test / note it — `sonar_live_tile.cpp:141` +- [ ] (suggestion) Warm-load eviction has no viewport and all tiles share `last_access_seq=0`, so survivors are filesystem order, not most-relevant (acceptable — no history yet) — `sonar_live_cache_layer.cpp:297` +- [ ] (suggestion) Missing reconciler-isolation test (plan step 13): no assertion that `overview_tiles_` keys never enter the reconciler's held set (D4 constraint, currently design-only) — `test/test_sonar_live_eviction.cpp` +- [ ] (suggestion) Overview level-parse guard is `lvl < 256` rather than the real gggs level count (~21); `valid()` catches it downstream but tighten the bound — `sonar_live_cache_layer.cpp:313` +- [ ] (suggestion) Plan Consequences table still says eviction calls `reconciler_.drop()`, contradicting the as-built `markHave`-keep note — reconcile — `plan.md:228` +- [ ] (suggestion) ADR-0006 addendum's four descriptive lines edge past ADR-0012's navigational scope (borderline; passes the "misleading?" test) — trim to pointer + one-liner — `docs/decisions/0006-live-tile-cache-persistence.md:161` +- [ ] (noted, not a finding) Evicted fine tiles don't reload on pan-back within a session — captured in ADR-0010 D2 as a deferred follow-up; accepted limitation. + +## Local Review (Pre-Push) +**Status**: complete +**When**: 2026-07-01 04:10 +00:00 +**By**: Claude Code Agent (Claude Opus) +**Verdict**: approved + +**Branch**: feature/issue-160 at `56db193` +**Mode**: pre-push +**Depth**: Deep (reason: new ADR-0010 + ~700 LOC memory-lifecycle/concurrency-critical caching) +**Must-fix**: 0 | **Suggestions**: 3 +**Round**: 2 | **Ship**: recommended — round-1 must-fix (unbounded overview pyramid) resolved and cross-confirmed; no new must-fix, only cosmetic/defensive suggestions. + +Round-1 must-fix RESOLVED: `accountedBytes()` now sums fine + overview maps +(`sonar_live_cache_layer.cpp:519-524`); `evictIfOverBudget()` is two-phase with +apex protection (`:366-383`); ADR-0010 D1/Consequences corrected. Round-1 suggestions +cleared or tracked-deferred: overview level-parse guard → `gggs::levels.size()` (`:314`); +eviction no longer writes (write-back storm gone); reconciler-isolation test added +(D4); plan Consequences reconciled to `markHave`-keep; ADR-0006 addendum trimmed to a +navigational pointer; `handleCatalog` overview-prune + high-latitude `foldChild` seam +captured as deferred follow-ups in ADR-0010. Static analysis (ament_cpplint): 47 +findings, all consistent camp house-style (camp does not enforce stock cpplint) — no +new actionable findings. Adversarial: 2 disjoint-lens Claude passes (Deep horizon); +threading / GL-texture lifetime / write-worker path all verified sound. Base fetch +failed offline; reviewed against local `origin/jazzy`. + +### Findings +- [ ] (suggestion) `evictIfOverBudget()` can intentionally exit still-over-budget when only the protected apex (`level ≤ 6`) remains — documented in ADR-0010 Consequences but not noted in the function; add a one-line comment (not a boundedness bug: apex is bounded by survey extent) — `sonar_live_cache_layer.cpp:366` +- [ ] (suggestion) `writeTileToCache`: `fs::create_directories(dir, ec)` swallows `ec`; an early `if(ec) return;` avoids a wasted GDAL round-trip on an unwritable cache dir (safe today — `writeToGeoTiff` fails gracefully) — `sonar_live_cache_layer.cpp:75` +- [ ] (suggestion) `handleTile`'s new-`Entry{...}` omits the explicit `last_access_seq` initializer used at the other call sites; cosmetic only (in-class default 0, overwritten at `:362`) — `sonar_live_cache_layer.cpp:357` diff --git a/CMakeLists.txt b/CMakeLists.txt index 8893d80..7bbbf39 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -846,6 +846,32 @@ if(BUILD_TESTING) Qt5::Positioning ${GDAL_LIBRARY} ) + + # [camp#160] Bounded eviction + overview pyramid, layer level. Same offscreen + # QApplication + Map harness as test_range_persist (GL-free / ROS-free). + ament_add_gtest(test_sonar_live_eviction + test/test_sonar_live_eviction.cpp + ) + target_include_directories(test_sonar_live_eviction PRIVATE + ${CMAKE_CURRENT_SOURCE_DIR}/src/camp_map + ) + ament_target_dependencies(test_sonar_live_eviction + geometry_msgs + marine_autonomy + marine_colormap + marine_interfaces + marine_tiled_raster_store + rclcpp + tf2_ros + ) + target_link_libraries(test_sonar_live_eviction + camp_map_ros + Qt5::Core + Qt5::Gui + Qt5::Widgets + Qt5::Positioning + ${GDAL_LIBRARY} + ) endif() diff --git a/docs/decisions/0006-live-tile-cache-persistence.md b/docs/decisions/0006-live-tile-cache-persistence.md index fe77f2e..d90a1a0 100644 --- a/docs/decisions/0006-live-tile-cache-persistence.md +++ b/docs/decisions/0006-live-tile-cache-persistence.md @@ -158,6 +158,9 @@ discovered source from an active one. Selected band + colormap persist per sourc - No eviction beyond prune-on-absence. A long survey could accumulate many tiles; eviction-by-area is a deferred follow-up (`// TODO(camp): eviction-by-area`), not implemented here (operator secondary-question default). + **Addendum (camp#160): superseded by + [ADR-0010](0010-bounded-eviction-overview-pyramid.md)** — bounded view-based-LOD + eviction plus a consumer-side overview pyramid now cap the cache. ## Alternatives considered diff --git a/docs/decisions/0010-bounded-eviction-overview-pyramid.md b/docs/decisions/0010-bounded-eviction-overview-pyramid.md new file mode 100644 index 0000000..ceff548 --- /dev/null +++ b/docs/decisions/0010-bounded-eviction-overview-pyramid.md @@ -0,0 +1,170 @@ +# ADR-0010: Bounded live-tile eviction + consumer-side overview pyramid + +## Status + +Accepted + +Implements camp issue #160 (supersedes #153, bounded eviction alone). Part of the +#154 camp resource self-monitoring umbrella. Extends +[ADR-0006](0006-live-tile-cache-persistence.md) (live tile cache persistence): it +fills the "eviction-by-area follow-up" left as a TODO in ADR-0006's consequences, +adds warm-load bounding, and adds a coarse overview product plus an LOD render +fallback. Depends on the GGGS `parent()`/`children()` index-math helpers +(unh_marine_autonomy#249). + +## Context + +`SonarLiveCacheLayer::tiles_` was **unbounded** — the only cap was reconciler +prune-on-absence. During a sustained live survey the in-memory map (each tile a +Float32 CPU band buffer plus an R32F GL texture) grows monotonically until RAM/VRAM +is exhausted; camp crashed on the salmon operator station shortly after enabling +live data (#153). This is a design gap, not a leak — tiles are freed correctly, +just never capped. + +The naïve fix (evict fine tiles over a budget) fixes the crash but makes a +zoomed-out view go **blank** over evicted areas until the tiles are re-sent. The +operator's requirement is to *still see full coverage when some tiles are unloaded +at their highest resolution*. So eviction and a multi-resolution overview pyramid +are one design: an evicted fine tile is folded into its coarse parent **before** it +is dropped, so coverage degrades to a lower resolution instead of disappearing. + +ADR-0006 makes this cheap and safe: live tiles are a **display-grade preview, not +data of record** — lossy, quantized, always rebuildable from the boat's durable +stores. Building downsampled overviews is display-LOD generation with no +provenance/fidelity burden. + +Forces: + +1. **Bound the footprint** so long surveys can't OOM the operator station. +2. **Never lose data** — an evicted tile must be persisted before its buffers are + released (ADR-0006's disk cache is the durable backing). +3. **Keep the big picture** — a zoomed-out view must show coverage everywhere the + survey reached, even where fine tiles were evicted. +4. **Don't waste bandwidth** (#71) — eviction must not trigger a re-request/re-evict + churn on a constrained link. +5. **GUI-thread-only** mutation (ADR-0006 D4): the reconciler and tile maps are not + thread-safe. + +## Decision + +### D1 — View-based LOD eviction against a resident-footprint budget + +A configurable budget (`QSettings LiveTileCache/max_vram_bytes`, default 512 MiB; +`0` disables — the pre-#160 behaviour) caps the **total resident** footprint (CPU +band data + any uploaded GL texture) across **both** fine tiles **and** overview +tiles — the overviews grow with survey area (see D3), so they must count toward the +budget or they'd be an unbounded second cache. `evictIfOverBudget()` runs on the +GUI thread after every +applied patch and after warm-load, in **two phases**, always **farthest from the +current viewport centre first** (classic slippy-map LOD — keep what's near the view, +discard the rest; viewport centre from `scene()->views()`, LRU by a monotonic +`last_access_seq` fallback when headless): + +1. **Fine tiles** (the detail) are evicted first, each folded into its coarse parent + (D3) so its coverage survives at lower resolution. +2. If the pyramid *itself* is still over budget, the **numerous near-fine overview + tiles** are evicted next — but **never the coarse apex** (`level <= + kApexProtectLevel`, currently 6), so a whole-survey zoomed-out view always has + coverage. An evicted overview's data already lives in its coarser ancestors (built + by the same fold chain), so dropping it loses no coverage, only mid-zoom fidelity. + +The apex is inherently a small, bounded handful for any realistic survey extent +(level-6 tiles span ~0.125°), so total resident memory is bounded: fine tiles + +near-view overviews to the budget, plus the O(small) apex. No vessel-position / tf +dependency — the operator chose view-based LOD over distance-from-vessel as the +simpler, more natural model. + +### D2 — Persist-then-drop; keep possession in the reconciler (no churn) + +Tiles are persisted continuously — a fine tile on every applied patch +(`handleTile`), an overview on every fold (`foldIntoParent`), both via the existing +coalesced `scheduleWriteThrough`. So by eviction time the tile (or an in-flight write +holding a copy of it) is already on disk, and eviction just folds a fine tile into +its parent, frees the GL texture under the renderer's context, and erases the entry — +**no write at eviction time** (which avoids a write-back storm when warm-load trims a +large cache of byte-identical tiles). The reconciler's `markHave` for that index is +**kept, not dropped**: we +still *possess* the tile (it is on disk), so the next catalog reconcile will not +re-request it. `tiles_` tracks residency (memory); the reconciler tracks possession +(disk) — the two intentionally decouple under eviction. Dropping it would cause a +re-request → re-receive → re-evict churn that wastes the very bandwidth #71 guards. +A genuine later catalog prune still frees the disk copy via `handleCatalog`. + +**Consequence / known limitation:** an evicted fine tile is not re-loaded from disk +when the operator later pans back into its area within the same session (there is no +view-change → disk-reload hook yet); the coarse overview covers it until the layer +is re-enabled or the boat re-sends a newer version. On-demand disk reload on view +change is a deferred follow-up. + +### D3 — Overview pyramid: match-resolution, fold-on-evict, full chain to level 0 + +`foldIntoParent()` decimates an evicted fine tile into the GGGS `parent()` of its +index and recurses up to level 0, so a zoomed-out view always has coverage. Each +overview tile **matches the fine tile's width/height** (standard pyramid: every tile +is the same pixel size; a parent W×H tile covers its ~4 children at half linear +resolution). `SonarLiveTile::foldChild()` area-maps each child cell to the parent +cell containing its geographic centre and averages the finite, non-NoData samples — +area-mapping (rather than a fixed 2×2 quadrant) keeps the polar `latitudeScaleFactor` +column scaling correct. Overviews are folded **on eviction only** (cheapest; the +overview may lag the latest fine data until first eviction, acceptable per ADR-0006 +D1). They persist to an `overviews/` sub-directory of the cache dir (own +`__.tif` files) so warm-load of fine tiles can't pick them up. + +### D4 — Overviews are a local derived product, outside the reconciler + +Overview tiles are **never** entered into `reconciler_` / the anti-entropy `tiles_` +set. They are not in the boat's catalog, so if they were reconciled they would be +pruned on absence. They live in a separate resident `overview_tiles_` map, +warm-loaded from the `overviews/` sub-dir (each file's level recovered from its +stem, since overviews span multiple coarse levels), and are freed under the +renderer context in the destructor exactly like fine tiles. + +### D5 — LOD fallback by draw order + +`items()` emits overview tiles first (coarse→fine, the natural `std::map` order by +GGGS level) and the fine tiles last, so the renderer draws fine tiles on top. Where +a fine tile is present it fully covers its parent; where it was evicted, the coarse +parent shows through instead of a blank gap. `recomputeBounds()` and +`foldAutoRange()` union both maps so the extent and colormap range stay correct when +only overviews remain for a region. + +## Consequences + +- Long surveys no longer grow resident memory/VRAM without bound; the crash scenario + (#153) is closed. Total residency = fine tiles + near-view overviews to the budget, + plus the O(small) protected apex. A one-time `qWarning` logs when the shed path is + first entered. (Residual: the protected apex grows *minimally* — at the coarsest + resolution — with survey *extent*, unavoidable if whole-survey zoom-out must always + show coverage; negligible for the lake/harbour envelope.) +- Warm-load is bounded: it loads fine tiles incrementally and trims every 64 inserts, + so enabling a source with a large disk cache can't spike memory on enable (the + salmon accelerant) — not just a post-load trim. +- Evicted coverage degrades gracefully to a coarser resolution rather than vanishing. +- The `LiveTileCache/max_vram_bytes` budget is self-accounted today; when the #155 + `ResourceMonitor` lands, its VRAM feed replaces `accountedBytes()` self-accounting + behind the same knob (a named integration seam, not in this change). +- **Deferred follow-ups:** + - On-demand disk reload of an evicted fine/overview tile when the operator pans back + into its area (no view-change → disk-reload hook yet; D2 limitation). + - `handleCatalog` prune removes a retracted fine tile from `tiles_`/disk/reconciler + but does not invalidate the overview cells it was folded into, nor delete orphaned + `overviews/` files — a retracted region keeps stale coarse coverage and the + `overviews/` dir grows slowly. Bounded (overviews are display-grade + evictable), + but overview lifecycle-on-retraction is a tracked follow-up. + - `foldChild` silently drops a child cell whose geographic centre falls outside the + parent (only reachable across a ±72°/±80° GGGS latitude-band boundary) → a possible + overview seam on a high-latitude survey; add a boundary test / handling if such + surveys arise. + +## Alternatives considered + +- **Distance-from-vessel eviction** (keep tiles near the survey head): rejected by + the operator as needless complexity and a tf dependency; view-based LOD is the + natural model for a pan/zoom display. +- **Fixed 64×64 overview tiles** (decouple overview size from fine size): rejected in + favour of matching fine dimensions (standard pyramid, uniform tile size). +- **Fold-on-arrival** (keep overviews continuously fresh): rejected as needless + compute; fold-on-evict suffices for a display-grade preview. +- **Producer-side overviews** (boat emits a coarse level, serving slow-link + first-paint before any fine tiles are pulled, #71): a separate, deferred bandwidth + feature; this ADR is consumer-side only. diff --git a/src/camp_map/ros/live_coverage/sonar_live_cache_layer.cpp b/src/camp_map/ros/live_coverage/sonar_live_cache_layer.cpp index df5c01e..4d88b3c 100644 --- a/src/camp_map/ros/live_coverage/sonar_live_cache_layer.cpp +++ b/src/camp_map/ros/live_coverage/sonar_live_cache_layer.cpp @@ -10,6 +10,8 @@ #include #include #include +#include +#include #include #include #include @@ -48,6 +50,21 @@ QString sanitize(const std::string& ns) return QString::fromLatin1(QUrl::toPercentEncoding(QString::fromStdString(ns))); } +// [camp#160] Cache sub-directory for overview pyramid tiles, kept apart from the +// fine tiles at the cache-dir root so warm-load of fine tiles can't pick them up. +const std::string kOverviewSubdir = "overviews"; + +// [camp#160] Squared distance (Web-Mercator scene units) from a tile's centre to +// a scene point — the eviction key for view-based LOD (farthest tile evicts first). +double tileSceneDistanceSquared(const SonarLiveTile& tile, const QPointF& point) +{ + const QPointF centre = web_mercator::geoToMap(QGeoCoordinate( + 0.5 * (tile.minLat() + tile.maxLat()), 0.5 * (tile.minLon() + tile.maxLon()))); + const double dx = centre.x() - point.x(); + const double dy = centre.y() - point.y(); + return dx * dx + dy * dy; +} + // Off-thread write-through worker. Takes everything by value so it is fully // self-contained — safe even if the layer is destroyed before it finishes. // Atomic temp+rename for crash safety only (ADR-0006 D2). @@ -56,6 +73,8 @@ void writeTileToCache(SonarLiveTile tile, std::string dir) namespace fs = std::filesystem; std::error_code ec; fs::create_directories(dir, ec); + if(ec) + return; // unwritable cache dir — skip the GDAL round-trip (best-effort cache) const std::string stem = std::to_string(static_cast(tile.index().level())) + "_" + std::to_string(tile.index().row()) + "_" + std::to_string(tile.index().column()); @@ -87,6 +106,13 @@ SonarLiveCacheLayer::SonarLiveCacheLayer(MapItem* parent, Node* node, "/live_tile_cache").toString(); cache_dir_ = QDir(base_dir).filePath(sanitize(base_namespace_)).toStdString(); + // [camp#160] Resident fine-tile footprint budget for eviction. Default 512 MiB, + // operator-overridable (ADR-0006 D2 / #117: a default, never an un-changeable + // hardcode). 0 disables eviction (unbounded — the pre-#160 behaviour). + constexpr qulonglong kDefaultBudgetBytes = 512ull * 1024 * 1024; + vram_budget_bytes_ = static_cast( + settings.value("LiveTileCache/max_vram_bytes", kDefaultBudgetBytes).toULongLong()); + // Availability is cheap: subscribe to the catalog even while inactive so the // operator can see how much coverage the source holds. The tile stream stays // unsubscribed until enableLiveCoverage(). @@ -115,6 +141,8 @@ SonarLiveCacheLayer::~SonarLiveCacheLayer() { for(auto& item : tiles_) item.second.texture.reset(); + for(auto& item : overview_tiles_) // [camp#160] overview textures too + item.second.texture.reset(); renderer_.doneCurrent(); } } @@ -246,36 +274,64 @@ void SonarLiveCacheLayer::warmLoad() if(!level_) return; // nothing cached yet - // [camp#134] insert_or_assign below replaces any pre-existing Entry for an index - // (reachable on a disable→re-enable: disable leaves tiles_ — and their GL textures - // — intact). A displaced Entry's QOpenGLTexture must be freed under a current GL - // context, so make it current for the seed loop when tiles already hold textures. - // On the first (empty-map) warm load there is nothing to displace, so skip it. - // Gate on hasContext() (mirrors GggsTileLayer::applyBand): when no context exists - // yet no texture was ever uploaded, so skip makeCurrent() rather than forcing the - // offscreen context into existence to reset null textures. - const bool gl_current = - !tiles_.empty() && renderer_.hasContext() && renderer_.makeCurrent(); + // [camp#160] Load fine tiles INCREMENTALLY (not SonarLiveTile::loadCacheDir, which + // would materialize the whole disk cache in one vector) and trim to the budget as + // we go, so a large cache can't spike memory during warm-load — the salmon + // accelerant (#153). An index already resident is kept (memory is >= the disk + // copy), so a re-enable never displaces a live Entry — which also means no GL + // context is needed here (no displaced texture to free; evictIfOverBudget() manages + // its own context). const gggs::Level level(*level_); - for(auto& tile : SonarLiveTile::loadCacheDir(cache_dir_, level)) + QDir fine_dir(QString::fromStdString(cache_dir_)); + int since_trim = 0; + for(const QString& name : fine_dir.entryList(QStringList() << "*.tif", QDir::Files)) { - const gggs::GridIndex index = tile.index(); - if(!index.valid()) + auto tile = SonarLiveTile::loadFromGeoTiff(fine_dir.filePath(name).toStdString(), level); + if(!tile || !tile->index().valid()) continue; + const gggs::GridIndex index = tile->index(); + if(tiles_.count(index)) + continue; // keep the resident (>= disk) copy; skip re-load on re-enable // Seed the reconciler at version 0: "have something" — older than any real - // catalog version, so the next catalog re-requests it if the boat is newer, - // and the prune gate never deletes it spuriously (ADR-0006 D3). + // catalog version, so the next catalog re-requests it if the boat is newer, and + // the prune gate never deletes it spuriously (ADR-0006 D3). reconciler_.markHave(index, 0); - tiles_.insert_or_assign(index, Entry{std::move(tile), nullptr, true}); + tiles_.insert_or_assign(index, Entry{std::move(*tile), nullptr, true, 0}); + if(++since_trim >= 64) // cap the load peak at ~budget + 64 tiles + { + since_trim = 0; + evictIfOverBudget(); + } } - if(gl_current) - renderer_.doneCurrent(); + + // [camp#160] Warm-load the overview pyramid. Overview tiles span multiple coarse + // levels, so recover each file's level from its `__.tif` stem + // rather than assuming the fine level. Kept resident (never evicted). + QDir overview_dir( + QDir(QString::fromStdString(cache_dir_)).filePath(QString::fromStdString(kOverviewSubdir))); + for(const QString& name : overview_dir.entryList(QStringList() << "*.tif", QDir::Files)) + { + bool ok = false; + const int lvl = name.section('.', 0, 0).section('_', 0, 0).toInt(&ok); + if(!ok || lvl < 0 || lvl >= static_cast(gggs::levels.size())) + continue; + auto tile = SonarLiveTile::loadFromGeoTiff(overview_dir.filePath(name).toStdString(), + gggs::Level(static_cast(lvl))); + if(tile && tile->index().valid()) + overview_tiles_.insert_or_assign(tile->index(), + Entry{std::move(*tile), nullptr, true, 0}); + } + if(band_name_.empty()) band_name_ = defaultBand(); recomputeBounds(); resetAutoRange(); foldAutoRange(); updateDisplay(); + + // [camp#160] Trim the freshly warm-loaded fine tiles to the budget before the + // tile stream is subscribed, so a large disk cache can't spike memory on enable. + evictIfOverBudget(); } // ------------------------------ message handlers ----------------------------- @@ -300,17 +356,22 @@ void SonarLiveCacheLayer::handleTile(const marine_interfaces::msg::SonarVisualiz const bool is_new = (it == tiles_.end()); if(is_new) it = tiles_.emplace(index, - Entry{SonarLiveTile(index, msg.width, msg.height), nullptr, true}) + Entry{SonarLiveTile(index, msg.width, msg.height), nullptr, true, 0}) .first; Entry& entry = it->second; entry.tile.applyPatch(msg); entry.texture_dirty = true; + entry.last_access_seq = ++access_seq_; // [camp#160] freshest — LRU fallback reconciler_.markHave(index, entry.tile.version()); - // TODO(camp): eviction-by-area follow-up — only prune-on-absence bounds the - // cache today, so a long survey can accumulate many tiles (ADR-0006 consequences). scheduleWriteThrough(entry.tile); + // [camp#160] Bound the resident fine-tile cache: over budget, the farthest + // tiles from the viewport fold into their overview parent, persist, and drop. + // May erase entries other than this one (or this one if it is the farthest and + // we are over budget), so do not touch `entry` after this call. + evictIfOverBudget(); + if(band_name_.empty()) band_name_ = defaultBand(); if(is_new) @@ -385,7 +446,8 @@ void SonarLiveCacheLayer::handleCatalog(const marine_interfaces::msg::TileCatalo updateDisplay(); } -void SonarLiveCacheLayer::scheduleWriteThrough(const SonarLiveTile& tile) +void SonarLiveCacheLayer::scheduleWriteThrough(const SonarLiveTile& tile, + const std::string& subdir) { // GUI thread. Coalesce per tile: snapshot the latest state and launch a worker // only if this tile is idle. If a worker is already in flight for this tile, mark @@ -395,6 +457,7 @@ void SonarLiveCacheLayer::scheduleWriteThrough(const SonarLiveTile& tile) WriteState& ws = write_states_[tile.index()]; ws.pending = tile; // latest wins ws.dirty = true; + ws.subdir = subdir; // [camp#160] "" (fine, root) or "overviews" (pyramid) if(!ws.in_flight) startWriteThrough(tile.index()); } @@ -411,7 +474,11 @@ void SonarLiveCacheLayer::startWriteThrough(const gggs::GridIndex& index) write_watchers_.push_back(watcher); connect(watcher, &QFutureWatcher::finished, this, [this, index, watcher]() { onWriteThroughFinished(index, watcher); }); - watcher->setFuture(QtConcurrent::run(writeTileToCache, std::move(*ws.pending), cache_dir_)); + // [camp#160] Destination = cache_dir_[/subdir]; the worker creates it. + const std::string dir = ws.subdir.empty() + ? cache_dir_ + : (std::filesystem::path(cache_dir_) / ws.subdir).string(); + watcher->setFuture(QtConcurrent::run(writeTileToCache, std::move(*ws.pending), dir)); } void SonarLiveCacheLayer::onWriteThroughFinished(const gggs::GridIndex& index, @@ -434,6 +501,173 @@ void SonarLiveCacheLayer::onWriteThroughFinished(const gggs::GridIndex& index, write_states_.erase(it); // clean: drop the per-tile state } +// -------------------------- eviction / overview pyramid ---------------------- + +std::size_t SonarLiveCacheLayer::accountedBytes() const +{ + // Total resident footprint: CPU band data (always present) plus the selected + // band's GL texture when uploaded, over BOTH fine tiles and overview (pyramid) + // tiles — the overviews grow with survey area, so they must count toward the + // budget or they'd be an unbounded second cache (ADR-0010 D1). + auto tileBytes = [](const Entry& e) -> std::size_t + { + std::size_t bytes = 0; + for(const auto& name : e.tile.bandNames()) + if(const SonarLiveBand* band = e.tile.band(name)) + bytes += band->data.size() * sizeof(float); + if(e.texture) + bytes += static_cast(e.tile.width()) * e.tile.height() * sizeof(float); + return bytes; + }; + std::size_t bytes = 0; + for(const auto& item : tiles_) + bytes += tileBytes(item.second); + for(const auto& item : overview_tiles_) + bytes += tileBytes(item.second); + return bytes; +} + +std::optional SonarLiveCacheLayer::currentViewCentre() const +{ + // Viewport centre in Web-Mercator scene coords (the eviction reference point). + // Null when headless (no scene/view attached, e.g. the render tests). + const QGraphicsScene* graphics_scene = scene(); + if(!graphics_scene) + return std::nullopt; + const QList views = graphics_scene->views(); + if(views.isEmpty() || !views.first()) + return std::nullopt; + QGraphicsView* view = views.first(); + return view->mapToScene(view->viewport()->rect()).boundingRect().center(); +} + +void SonarLiveCacheLayer::foldIntoParent(const SonarLiveTile& fine) +{ + // Decimate a fine tile into its coarse parent and recurse up to level 0, so a + // zoomed-out view always has coverage even after the fine tiles are evicted. + // Overview tiles match the fine tile's width/height (standard pyramid) and are + // persisted under the `overviews/` cache sub-dir. They are never added to the + // reconciler (a local derived product; prune-on-absence must not touch them). + const gggs::GridIndex parent_index = gggs::parent(fine.index()); + if(!parent_index.valid()) + return; + auto it = overview_tiles_.find(parent_index); + if(it == overview_tiles_.end()) + it = overview_tiles_ + .emplace(parent_index, + Entry{SonarLiveTile(parent_index, fine.width(), fine.height()), + nullptr, true, 0}) + .first; + Entry& overview = it->second; + overview.tile.foldChild(fine); + overview.texture_dirty = true; + scheduleWriteThrough(overview.tile, kOverviewSubdir); + foldIntoParent(overview.tile); // build the full chain to level 0 +} + +void SonarLiveCacheLayer::evictIfOverBudget() +{ + if(vram_budget_bytes_ == 0 || accountedBytes() <= vram_budget_bytes_) + return; + + if(!eviction_warned_) + { + qWarning().noquote() << "[live coverage" << QString::fromStdString(base_namespace_) + << "] resident tile budget exceeded (" + << qulonglong(accountedBytes()) << ">" + << qulonglong(vram_budget_bytes_) + << "bytes) — shedding to the overview pyramid"; + eviction_warned_ = true; + } + + // Order candidate indices farthest-from-viewport-centre first (view-based LOD); + // LRU by last_access_seq (oldest first) when headless (no view attached). + const std::optional centre = currentViewCentre(); + auto orderByDistance = [&](const std::map& pool, + std::vector keys) + { + std::sort(keys.begin(), keys.end(), + [&](const gggs::GridIndex& a, const gggs::GridIndex& b) + { + const Entry& ea = pool.at(a); + const Entry& eb = pool.at(b); + if(centre) + return tileSceneDistanceSquared(ea.tile, *centre) > + tileSceneDistanceSquared(eb.tile, *centre); + return ea.last_access_seq < eb.last_access_seq; + }); + return keys; + }; + + // Fine tiles and overviews are already persisted (handleTile / foldIntoParent write + // through), so eviction just frees memory — the disk cache is the durable backing + // (ADR-0010 D2). Reconciler markHave is kept (possession on disk), NOT dropped, so + // eviction can't trigger a re-request/re-evict churn (#71): tiles_ = residency, the + // reconciler = possession. Free GL textures under the renderer's context. + const bool gl_current = renderer_.hasContext() && renderer_.makeCurrent(); + bool evicted = false; + + // Phase 1 — evict fine tiles (the detail) farthest-first, folding each into its + // coarse parent so its coverage survives at lower resolution. + { + std::vector keys; + keys.reserve(tiles_.size()); + for(const auto& item : tiles_) + keys.push_back(item.first); + for(const gggs::GridIndex& index : orderByDistance(tiles_, std::move(keys))) + { + if(accountedBytes() <= vram_budget_bytes_) + break; + auto it = tiles_.find(index); + if(it == tiles_.end()) + continue; + foldIntoParent(it->second.tile); // degrade to coarse parent (persists it) + it->second.texture.reset(); + tiles_.erase(it); + evicted = true; + } + } + + // Phase 2 — if the pyramid itself is still over budget, evict the numerous + // near-fine overview tiles farthest-first, but NEVER the coarse apex + // (level <= kApexProtectLevel) so a whole-survey zoom-out always has coverage. An + // evicted overview's data already lives in its coarser ancestors (built by the same + // fold chain), so dropping it loses no coverage — just some mid-zoom fidelity. + if(accountedBytes() > vram_budget_bytes_) + { + std::vector keys; + for(const auto& item : overview_tiles_) + if(item.first.level() > kApexProtectLevel) + keys.push_back(item.first); + for(const gggs::GridIndex& index : orderByDistance(overview_tiles_, std::move(keys))) + { + if(accountedBytes() <= vram_budget_bytes_) + break; + auto it = overview_tiles_.find(index); + if(it == overview_tiles_.end()) + continue; + it->second.texture.reset(); + overview_tiles_.erase(it); + evicted = true; + } + } + // May intentionally return still-over-budget when only the protected apex remains + // (apex bytes > budget) — that is the O(small) floor that guarantees zoom-out + // coverage, not a leak (ADR-0010 D1 / Consequences). + + if(gl_current) + renderer_.doneCurrent(); + + if(evicted) + { + recomputeBounds(); + resetAutoRange(); + foldAutoRange(); + cached_image_ = QImage(); + update(boundingRect()); + } +} + // ------------------------------ extent / range ------------------------------- void SonarLiveCacheLayer::recomputeBounds() @@ -441,15 +675,20 @@ void SonarLiveCacheLayer::recomputeBounds() prepareGeometryChange(); QRectF bounds; bool first = true; - for(const auto& entry : tiles_) + auto expand = [&](const SonarLiveTile& tile) { - const SonarLiveTile& tile = entry.second.tile; const QPointF lo = web_mercator::geoToMap(QGeoCoordinate(tile.minLat(), tile.minLon())); const QPointF hi = web_mercator::geoToMap(QGeoCoordinate(tile.maxLat(), tile.maxLon())); const QRectF rect = QRectF(lo, hi).normalized(); bounds = first ? rect : bounds.united(rect); first = false; - } + }; + for(const auto& entry : tiles_) + expand(entry.second.tile); + // [camp#160] Include overviews so the extent stays correct after fine tiles are + // evicted (an all-evicted region is still covered by its resident overview). + for(const auto& entry : overview_tiles_) + expand(entry.second.tile); scene_bounds_ = bounds; if(!scene_bounds_.isNull()) { @@ -469,15 +708,21 @@ void SonarLiveCacheLayer::resetAutoRange() void SonarLiveCacheLayer::foldAutoRange() { bool first = (data_min_ > data_max_); - for(const auto& entry : tiles_) + auto fold = [&](const Entry& e) { - const SonarLiveBand* band = entry.second.tile.band(band_name_); + const SonarLiveBand* band = e.tile.band(band_name_); if(!band || band->data_min > band->data_max) - continue; + return; if(first || band->data_min < data_min_) data_min_ = band->data_min; if(first || band->data_max > data_max_) data_max_ = band->data_max; first = false; - } + }; + for(const auto& entry : tiles_) + fold(entry.second); + // [camp#160] Fold overview ranges too so the colormap tracks the full visible + // extent when only overviews remain for a region. + for(const auto& entry : overview_tiles_) + fold(entry.second); // [camp#142] Keep the Auto resolved range current with the freshly-folded extents // (a no-op while the operator holds a Manual override). camp#138 coordination: // this update_auto() call site is where #138's fold changes meet this PR — the @@ -504,7 +749,9 @@ std::string SonarLiveCacheLayer::defaultBand() const void SonarLiveCacheLayer::updateDisplay() { if(enabled_) - setStatus(QString("(live: %1 tiles)").arg(qulonglong(tiles_.size()))); + setStatus(QString("(live: %1 fine + %2 overview tiles)") + .arg(qulonglong(tiles_.size())) + .arg(qulonglong(overview_tiles_.size()))); else setStatus("(available)"); } @@ -518,6 +765,9 @@ QRectF SonarLiveCacheLayer::boundingRect() const QOpenGLTexture* SonarLiveCacheLayer::textureFor(Entry& entry) { + // [camp#160] Rendering a tile marks it recently used (the LRU eviction fallback + // approximates "in view"). Harmless for overview entries (never evicted). + entry.last_access_seq = ++access_seq_; const SonarLiveBand* band = entry.tile.band(band_name_); if(!band || band->data.empty()) return nullptr; @@ -581,16 +831,15 @@ QList SonarLiveCacheLayer::items() // textureFor() may lazily (re)upload here. The geo->Web-Mercator warp lives in // the renderer; this only forwards each tile's lat/lon extent + NoData sentinel. QList result; - result.reserve(int(tiles_.size())); - for(auto& item : tiles_) + result.reserve(int(overview_tiles_.size() + tiles_.size())); + auto append = [&](Entry& entry) { - Entry& entry = item.second; const SonarLiveBand* band = entry.tile.band(band_name_); if(!band) - continue; + return; QOpenGLTexture* texture = textureFor(entry); if(!texture) - continue; + return; raster::RasterFieldItem fi; fi.texture = texture; fi.format = raster::RasterFieldItem::Format::Scalar; @@ -602,13 +851,22 @@ QList SonarLiveCacheLayer::items() fi.has_nodata = band->has_nodata; fi.nodata = band->has_nodata ? band->nodata : 0.0f; result.push_back(fi); - } + }; + // [camp#160] LOD fallback by draw order: overviews first (coarse->fine, the + // std::map orders by GGGS level), then the fine tiles on top. Where a fine tile + // is present it fully covers its parent; where it was evicted, the coarse parent + // shows through instead of a blank gap. + for(auto& item : overview_tiles_) + append(item.second); + for(auto& item : tiles_) + append(item.second); return result; } QImage SonarLiveCacheLayer::renderImage(const QSize& size) { - if(tiles_.empty() || data_min_ > data_max_ || size.isEmpty()) + if((tiles_.empty() && overview_tiles_.empty()) || data_min_ > data_max_ || + size.isEmpty()) return QImage(); // [camp#134] Make the renderer's context current, collect the held tiles // (uploading textures under it), then delegate the warp + draw. @@ -625,7 +883,7 @@ QImage SonarLiveCacheLayer::renderImage(const QSize& size) void SonarLiveCacheLayer::paint(QPainter* painter, const QStyleOptionGraphicsItem*, QWidget*) { - if(tiles_.empty() || data_min_ > data_max_) + if((tiles_.empty() && overview_tiles_.empty()) || data_min_ > data_max_) return; const QRectF dev = painter->worldTransform().mapRect(boundingRect()); diff --git a/src/camp_map/ros/live_coverage/sonar_live_cache_layer.h b/src/camp_map/ros/live_coverage/sonar_live_cache_layer.h index f4eb6c3..de37fc6 100644 --- a/src/camp_map/ros/live_coverage/sonar_live_cache_layer.h +++ b/src/camp_map/ros/live_coverage/sonar_live_cache_layer.h @@ -16,6 +16,7 @@ #include #include +#include #include #include #include @@ -79,6 +80,14 @@ class SonarLiveCacheLayer: public Layer, public raster::RasterFieldSource /// Whether the tile stream is currently subscribed (operator-enabled). bool isEnabled() const { return enabled_; } + /// [camp#160] Resident fine-tile count and coarse overview (pyramid) tile count. + /// Introspection for tests and the future status indicator (#158). + std::size_t residentTileCount() const { return tiles_.size(); } + std::size_t overviewTileCount() const { return overview_tiles_.size(); } + /// [camp#160] Number of indices the reconciler holds — used by tests to assert + /// overview tiles never enter the anti-entropy set (ADR-0010 D4). + std::size_t reconcilerHeldCount() const { return reconciler_.size(); } + /// [camp#121] Render the in-memory tiles into an offscreen image of @p size /// spanning the layer extent. Null image if there is no data / GL is /// unavailable. Exposed for a headless render check (skips with no GL). @@ -134,6 +143,9 @@ private slots: SonarLiveTile tile; std::unique_ptr texture; bool texture_dirty = true; + // [camp#160] Monotonic access stamp for the LRU eviction fallback (bumped on + // patch + on render). Unused for overview entries (they are never evicted). + std::uint64_t last_access_seq = 0; }; void subscribeCatalog(); @@ -148,11 +160,28 @@ private slots: // onWriteThroughFinished() (GUI thread) re-launches once more if a newer patch // arrived during the write — so writes for a tile are serialized and never race // on the shared .tif.tmp path. - void scheduleWriteThrough(const SonarLiveTile& tile); + // [camp#160] @p subdir is a cache sub-directory ("" = fine tiles at cache_dir_ + // root, "overviews" = pyramid parents) so fine and overview writes never share + // a `.tif.tmp` path. Fine and overview indices are at different GGGS + // levels, so the per-index write-state map keys never collide. + void scheduleWriteThrough(const SonarLiveTile& tile, const std::string& subdir = ""); void startWriteThrough(const gggs::GridIndex& index); void onWriteThroughFinished(const gggs::GridIndex& index, QFutureWatcher* watcher); + // [camp#160] Bounded eviction + overview pyramid (view-based LOD). + // accountedBytes(): resident fine-tile footprint (CPU band data + any uploaded + // GL texture). evictIfOverBudget(): while over vram_budget_bytes_, fold the + // fine tile farthest from the viewport centre into its coarse parent, persist + // it, free its texture, and drop it (LRU by last_access_seq when no view is + // attached). foldIntoParent(): 2x2-decimate a fine tile up the full overview + // chain to level 0. currentViewCentre(): viewport centre in Web-Mercator scene + // coords, or nullopt when headless. + std::size_t accountedBytes() const; + void evictIfOverBudget(); + void foldIntoParent(const SonarLiveTile& fine); + std::optional currentViewCentre() const; + void recomputeBounds(); void resetAutoRange(); void foldAutoRange(); @@ -170,6 +199,13 @@ private slots: static constexpr int kMaxImageEdge = 4096; + // [camp#160] Overview tiles at level <= this coarse "apex" are never evicted, so a + // whole-survey zoomed-out view always has coverage. The apex is inherently a small, + // bounded handful for any realistic survey extent (level-6 tiles span ~0.125 deg), + // while the numerous near-fine overview levels ARE evicted by view like fine tiles — + // so total resident memory stays bounded (ADR-0010 D1/D3). + static constexpr std::uint8_t kApexProtectLevel = 6; + std::string base_namespace_; // e.g. "/cube_bathymetry" std::string cache_dir_; // / bool enabled_ = false; // tile stream subscribed (persisted) @@ -190,6 +226,22 @@ private slots: marine_tiled_raster_store::TileCatalogReconciler reconciler_; std::map tiles_; + // [camp#160] Coarse overview (pyramid) tiles keyed by their own GGGS index at a + // level below tiles_'s. Built by folding evicted fine tiles into their parents, + // kept resident (never evicted — a handful of tiles), and rendered *under* the + // fine tiles so an evicted area degrades to a coarser resolution instead of + // going blank. These are a LOCAL derived product: they are NEVER entered into + // `reconciler_`, so anti-entropy prune-on-absence can't delete them. + std::map overview_tiles_; + + // [camp#160] Monotonic access counter feeding Entry::last_access_seq (LRU + // eviction fallback), and the resident-footprint budget for eviction + // (QSettings "LiveTileCache/max_vram_bytes", default 512 MiB — ADR-0006 D2 / + // #117 no-hardcoded-defaults). + std::uint64_t access_seq_ = 0; + std::size_t vram_budget_bytes_ = 0; + bool eviction_warned_ = false; // log the shed-load path once, not per evict + QRectF scene_bounds_; double data_min_ = 1.0; // auto-range over the selected band (crossed => none) double data_max_ = 0.0; @@ -214,6 +266,7 @@ private slots: bool in_flight = false; bool dirty = false; std::optional pending; // SonarLiveTile has no default ctor + std::string subdir; // [camp#160] "" (root) or "overviews" }; std::map write_states_; diff --git a/src/camp_map/ros/live_coverage/sonar_live_tile.cpp b/src/camp_map/ros/live_coverage/sonar_live_tile.cpp index 9433778..593e8d5 100644 --- a/src/camp_map/ros/live_coverage/sonar_live_tile.cpp +++ b/src/camp_map/ros/live_coverage/sonar_live_tile.cpp @@ -127,6 +127,73 @@ void SonarLiveTile::applyPatch(const marine_interfaces::msg::SonarVisualizationT version_ = std::max(version_, toNanoseconds(msg.header)); } +void SonarLiveTile::foldChild(const SonarLiveTile& child) +{ + // [camp#160] Decimate a finer child into this coarser parent's matching + // sub-window for the overview pyramid. Both tiles are the same width x height + // (a parent cell therefore spans ~2x2 child cells). We area-map each child + // cell to the parent cell containing its geographic centre and average the + // finite, non-NoData samples landing in each parent cell — north-up, so row 0 + // is the northernmost row. Only cells the child covers are written, so folding + // each of a parent's children in turn accumulates the full parent. + if(width_ <= 0 || height_ <= 0 || child.width_ <= 0 || child.height_ <= 0) + return; + const double parent_lon0 = minLon(); + const double parent_lat1 = maxLat(); + const double parent_lon_span = maxLon() - parent_lon0; + const double parent_lat_span = parent_lat1 - minLat(); + if(parent_lon_span <= 0.0 || parent_lat_span <= 0.0) + return; + const double child_lon0 = child.minLon(); + const double child_lat1 = child.maxLat(); + const double child_lon_span = child.maxLon() - child_lon0; + const double child_lat_span = child_lat1 - child.minLat(); + + const std::size_t cells = static_cast(width_) * height_; + for(const auto& [name, cband] : child.bands_) + { + SonarLiveBand& pband = bands_[name]; + if(pband.data.empty()) + { + // Uncovered parent cells stay transparent: the child's NoData sentinel if + // it has one, else NaN (both are discarded by the renderer, camp#134). + const float fill = cband.has_nodata ? cband.nodata + : std::numeric_limits::quiet_NaN(); + pband.name = name; + pband.data.assign(cells, fill); + pband.has_nodata = cband.has_nodata; + pband.nodata = cband.nodata; + } + + std::vector sum(cells, 0.0); + std::vector count(cells, 0); + for(int cr = 0; cr < child.height_; ++cr) + { + const double lat = child_lat1 - ((cr + 0.5) / child.height_) * child_lat_span; + const int pr = static_cast((parent_lat1 - lat) / parent_lat_span * height_); + if(pr < 0 || pr >= height_) + continue; + for(int cc = 0; cc < child.width_; ++cc) + { + const float v = cband.data[static_cast(cr) * child.width_ + cc]; + if(!std::isfinite(v) || (cband.has_nodata && v == cband.nodata)) + continue; + const double lon = child_lon0 + ((cc + 0.5) / child.width_) * child_lon_span; + const int pc = static_cast((lon - parent_lon0) / parent_lon_span * width_); + if(pc < 0 || pc >= width_) + continue; + const std::size_t idx = static_cast(pr) * width_ + pc; + sum[idx] += v; + count[idx] += 1; + } + } + for(std::size_t i = 0; i < cells; ++i) + if(count[i] > 0) + pband.data[i] = static_cast(sum[i] / count[i]); + refoldRange(pband); + } +} + void SonarLiveTile::refoldRange(SonarLiveBand& band) { // A patch can overwrite a former extreme cell, so re-fold the whole band rather diff --git a/src/camp_map/ros/live_coverage/sonar_live_tile.h b/src/camp_map/ros/live_coverage/sonar_live_tile.h index 2e84133..4e1af9b 100644 --- a/src/camp_map/ros/live_coverage/sonar_live_tile.h +++ b/src/camp_map/ros/live_coverage/sonar_live_tile.h @@ -63,6 +63,17 @@ class SonarLiveTile /// skipped rather than throwing. void applyPatch(const marine_interfaces::msg::SonarVisualizationTile& msg); + /// [camp#160] Decimate a finer child tile into this coarser (parent) tile for + /// the live-coverage overview pyramid. `parent(child.index())` must equal this + /// tile's index and both tiles must be the same width x height. The child + /// covers ~1/4 of this tile's geographic extent (a quadrant in temperate + /// bands); its cells are area-mapped by geographic centre and averaged + /// (skipping NoData / non-finite) into the corresponding parent cells, leaving + /// the rest of the parent untouched. Area-mapping (rather than a fixed 2x2 + /// quadrant) keeps the polar `latitudeScaleFactor` column scaling correct. + /// Creates missing parent bands (NoData/NaN-filled) and re-folds their ranges. + void foldChild(const SonarLiveTile& child); + /// [camp#121] Warm-load one cached Float32 GeoTIFF (all bands, band names from /// band descriptions, per-band NoData, GridIndex recovered from the /// geotransform at @p level — the marine_tiled_raster_store::loadTile diff --git a/test/test_sonar_live_cache.cpp b/test/test_sonar_live_cache.cpp index 6f8d8dd..ad24905 100644 --- a/test/test_sonar_live_cache.cpp +++ b/test/test_sonar_live_cache.cpp @@ -261,3 +261,77 @@ TEST(SonarLiveCache, PruneTimestampGate) EXPECT_TRUE(result.to_prune.empty()); EXPECT_TRUE(result.to_request.empty()); // A,B held at current version } + +// 5. [camp#160] foldChild decimates a fine tile into its coarse parent. A uniform +// child covers ~1/4 of the (same-sized) parent, so exactly (kEdge/2)^2 parent cells +// take the child value and the rest stay NoData (untouched by this one child). +TEST(SonarLiveCache, FoldChildDecimatesIntoParentQuadrant) +{ + const gggs::Level fine_level(kLevel); + const gggs::GridIndex fine_idx = fine_level.gridIndex(43.07, -70.76); + ASSERT_TRUE(fine_idx.valid()); + const gggs::GridIndex parent_idx = gggs::parent(fine_idx); + ASSERT_TRUE(parent_idx.valid()); + + // Uniform fine tile: every cell = 3.0 (raw 300, scale 0.01). + SonarLiveTile fine(fine_idx, kEdge, kEdge); + fine.applyPatch(makeDepthPatch(fine_idx, 300, 100)); + + // Parent matches the fine tile's dimensions (standard pyramid). + SonarLiveTile parent(parent_idx, kEdge, kEdge); + parent.foldChild(fine); + + const auto* pb = parent.band("depth"); + ASSERT_NE(pb, nullptr); + ASSERT_EQ(pb->data.size(), static_cast(kEdge) * kEdge); + + int written = 0; + for(float v : pb->data) + if(v == 3.0f) + ++written; + EXPECT_EQ(written, (kEdge / 2) * (kEdge / 2)); // one quadrant, 2x2-decimated + EXPECT_FLOAT_EQ(pb->data_min, 3.0f); // uniform child -> uniform mean + EXPECT_FLOAT_EQ(pb->data_max, 3.0f); + EXPECT_TRUE(pb->has_nodata); +} + +// 6. [camp#160] foldChild NoData handling: a 2x2 child block that is entirely NoData +// leaves its parent cell a NoData hole; a partial block averages only the finite +// samples. +TEST(SonarLiveCache, FoldChildPropagatesNoData) +{ + const gggs::Level fine_level(kLevel); + const gggs::GridIndex fine_idx = fine_level.gridIndex(43.07, -70.76); + const gggs::GridIndex parent_idx = gggs::parent(fine_idx); + ASSERT_TRUE(parent_idx.valid()); + + // Base 400 (= 4.0). One 2x2 child block (gggs rows 6-7, cols 0-1) entirely NoData + // -> a single parent NoData hole. One extra NoData cell (7,2) makes an adjacent + // parent cell a partial block averaging the 3 finite samples to 4.0. + SonarLiveTile fine(fine_idx, kEdge, kEdge); + fine.applyPatch(makeDepthPatch(fine_idx, 400, 100, + {{7, 0, -32768}, {7, 1, -32768}, + {6, 0, -32768}, {6, 1, -32768}, + {7, 2, -32768}})); + + SonarLiveTile parent(parent_idx, kEdge, kEdge); + parent.foldChild(fine); + const auto* pb = parent.band("depth"); + ASSERT_NE(pb, nullptr); + + int written = 0; + int holes = 0; + for(float v : pb->data) + { + if(v == 4.0f) + ++written; + else if(pb->has_nodata && v == pb->nodata) + ++holes; + } + // (kEdge/2)^2 parent cells are in the child's quadrant; the all-NoData block is a + // hole, the rest (incl. the partial block averaged to 4.0) are written. + EXPECT_EQ(written, (kEdge / 2) * (kEdge / 2) - 1); + EXPECT_GE(holes, 1); + EXPECT_FLOAT_EQ(pb->data_min, 4.0f); + EXPECT_FLOAT_EQ(pb->data_max, 4.0f); +} diff --git a/test/test_sonar_live_eviction.cpp b/test/test_sonar_live_eviction.cpp new file mode 100644 index 0000000..3f324f7 --- /dev/null +++ b/test/test_sonar_live_eviction.cpp @@ -0,0 +1,141 @@ +// [camp#160] Headless layer-level test for bounded eviction + the overview pyramid. +// Pre-populate the on-disk cache with more fine tiles than the resident budget, +// enable the layer (null ROS node, no GL), and assert the resident fine-tile set is +// trimmed to the budget while the evicted tiles are folded into resident overview +// tiles (so coverage is never lost). Reuses the offscreen-QApplication + Map harness +// from test_range_persist; GL-free (renderImage() is never called) and ROS-free +// (null Node -> no subscription). + +#include + +#include + +#include +#include +#include +#include +#include + +#include "marine_autonomy/gggs.h" +#include "marine_interfaces/msg/sonar_visualization_tile.hpp" +#include "marine_interfaces/msg/visualization_band.hpp" + +#include "map/map.h" +#include "map/layer_list.h" +#include "ros/live_coverage/sonar_live_cache_layer.h" +#include "ros/live_coverage/sonar_live_tile.h" + +namespace mi = marine_interfaces::msg; +using camp::map::Map; +using camp::ros::live_coverage::SonarLiveCacheLayer; +using camp::ros::live_coverage::SonarLiveTile; +using camp::ros::live_coverage::tileIndexFromGridIndex; + +namespace +{ + +constexpr int kLevel = 10; +constexpr int kEdge = 8; + +// Percent-encode a namespace exactly as SonarLiveCacheLayer::sanitize() does, so the +// test writes tiles into the same cache sub-directory the layer will read. +QString sanitizeNamespace(const QString& ns) +{ + return QString::fromLatin1(QUrl::toPercentEncoding(ns)); +} + +// A uniform full-tile INT16 "depth" patch (value = base * 0.01) for @p grid. +mi::SonarVisualizationTile makeUniformDepth(const gggs::GridIndex& grid, std::int16_t base) +{ + mi::SonarVisualizationTile msg; + msg.header.stamp.sec = 100; + msg.header.frame_id = "gggs"; + msg.index = tileIndexFromGridIndex(grid); + msg.width = kEdge; + msg.height = kEdge; + msg.window_col = 0; + msg.window_row = 0; + msg.window_width = kEdge; + msg.window_height = kEdge; + + mi::VisualizationBand band; + band.name = "depth"; + band.dtype = mi::VisualizationBand::INT16; + band.scale = 0.01; + band.offset = 0.0; + band.nodata = -32768.0; + for(int i = 0; i < kEdge * kEdge; ++i) + { + band.data.push_back(static_cast(base & 0xff)); + band.data.push_back(static_cast((base >> 8) & 0xff)); + } + msg.bands.push_back(band); + return msg; +} + +} // namespace + +TEST(SonarLiveEviction, WarmLoadTrimsToBudgetAndBuildsOverviews) +{ + QSettings().clear(); + QTemporaryDir cache; + ASSERT_TRUE(cache.isValid()); + + const QString ns = "/evict_test"; + constexpr std::size_t kTiles = 24; + // Each fine tile is kEdge*kEdge floats (one band) resident. Budget = 4 tiles' worth + // so warm-load must evict most of them. + const std::size_t per_tile = static_cast(kEdge) * kEdge * sizeof(float); + const std::size_t budget = 4 * per_tile; + + QSettings().setValue("LiveTileCache/cache_dir", cache.path()); + QSettings().setValue("LiveTileCache/max_vram_bytes", qulonglong(budget)); + + // Write kTiles distinct fine tiles into the layer's (sanitized) cache directory. + const QString tile_dir = QDir(cache.path()).filePath(sanitizeNamespace(ns)); + ASSERT_TRUE(QDir().mkpath(tile_dir)); + const gggs::Level level(kLevel); + std::size_t written = 0; + for(std::size_t i = 0; i < kTiles; ++i) + { + const gggs::GridIndex grid = level.gridIndex(43.0 + 0.02 * i, -70.5); + ASSERT_TRUE(grid.valid()); + SonarLiveTile tile(grid, kEdge, kEdge); + tile.applyPatch(makeUniformDepth(grid, static_cast(300 + i))); + const QString path = + QDir(tile_dir).filePath(QString("%1_%2_%3.tif") + .arg(static_cast(grid.level())) + .arg(grid.row()) + .arg(grid.column())); + ASSERT_TRUE(tile.writeToGeoTiff(path.toStdString())); + ++written; + } + ASSERT_EQ(written, kTiles); + + Map map; + camp::map::LayerList* layers = map.topLevelLayers(); + ASSERT_NE(layers, nullptr); + auto* layer = new SonarLiveCacheLayer(layers, nullptr, ns); + + layer->enableLiveCoverage(); // incremental warm-load + two-phase evict-to-budget + + // Resident fine tiles are bounded (evicted well below the 24 loaded)... + EXPECT_LT(layer->residentTileCount(), kTiles); + EXPECT_LE(layer->residentTileCount(), std::size_t(4)); + // ...evicted tiles were folded into resident overview parents, so a zoomed-out + // view still has coverage (the coarse apex is protected, never fully evicted)... + EXPECT_GT(layer->overviewTileCount(), std::size_t(0)); + // ...and the overviews are a LOCAL derived product: every fine tile's possession is + // kept (markHave, never dropped) and NO overview leaks into the reconciler, so the + // held count is exactly the fine tiles loaded — asserts both the no-drop / no-churn + // policy (D2) and the reconciler-isolation invariant (D4). + EXPECT_EQ(layer->reconcilerHeldCount(), kTiles); +} + +int main(int argc, char** argv) +{ + qputenv("QT_QPA_PLATFORM", "offscreen"); + QApplication app(argc, argv); + ::testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); +}