fix: recover 351 failures from the eight-container valid_updated batch

[kgerb]

Recover 351 failures from the eight-container valid_updated batch

Summary

The current eight-container ForestStructure batch has accumulated 351 failed datasets. The failures are fully accounted for by seven disjoint root-cause classes. Only 54 failures are memory-related; 297 are deterministic implementation/validation defects or one transient storage failure.

This issue is the umbrella tracker for fixing the unresolved classes, rebuilding canonical ordered COPCs where required, and retrying only the failed cohort against valid_updated as the scientific oracle.

Batch context

• Running image: 3dtrees-foreststructure:julia-faithful-singlefile-fixeddatasets-memory70-20260817
• Parallel jobs: 8
• Per container: 12 CPUs, 37 GiB hard memory limit
• Internal analysis guard: 32 GiB
• LAScatalog workers: 2
• Snapshot: 2026-08-31 12:46 CEST
• Snapshot counts: 718 success, 351 failed, 8 running, 236 queued
• Failure policy: failed datasets are recorded once and are not retried automatically

The classification below was derived from the batch failure ledger, container OOM state, stage markers, and the terminal error signature in every failed container log. The classes sum to exactly 351.

Root cause	Count	Share	Current status
Fine-coordinate DTM integer conversion	234	66.7%	Patched by #1; failed cohort still needs ordered-COPC rerun
Memory exhaustion or 32 GiB guard rejection	54	15.4%	COPC path materially reduces memory, but cohort is not yet proven
AOI conversion false negative	41	11.7%	Unresolved
Conflicting textual CRS metadata in DTM chunks	11	3.1%	Unresolved
DTM mosaic excludes empty boundary chunks	3	0.9%	Unresolved
Extra GDAL .aux.xml sidecars	7	2.0%	Batch comparator defect; unresolved
Storage staging outage	1	0.3%	Transient; retry required

1. Fine-coordinate DTM integer conversion — 234 datasets

Symptom

Internal error in C_interpolate_delaunay: xy coordinates were not converted to integer. Scale factors are likely to be invalid.

Cause

lidR's Delaunay implementation converts absolute XY coordinates to signed 32-bit integers. Inputs combining large absolute coordinates with very fine LAS scales (for example dataset 2160 at 1e-7 m) overflow that representation during DTM triangulation.

Resolution status

Patched in #1. The new path catches this exact error and rescales only the in-memory DTM copy to the smallest safe decimal scale. Source LAZ/COPC coordinates are not rewritten. The implementation and regression test are referenced in the follow-up comment.

Acceptance

• Rebuild canonical COPCs with complete unique OriginalPointIndex.
• Rerun all 234 datasets with the ordered-COPC image.
• Require successful completion and exact comparison against each available valid_updated oracle.

234 affected dataset IDs

1210, 1211, 1213, 2011, 2013, 2102, 2103, 2104, 2105, 2108, 2109, 2110, 2111, 2143, 2144, 2145, 2146, 2148, 2149, 2150, 2151, 2152, 2153, 2154, 2155, 2156, 2157, 2159, 2160, 2162, 2163, 2164, 2165, 2166, 2167, 2168, 2169, 2170, 2171, 2172, 2173, 2174, 2175, 2176, 2177, 2178, 2179, 2180, 2181, 2182, 2183, 2184, 2185, 2186, 2187, 2189, 2190, 2192, 2193, 2195, 2196, 2197, 2198, 2199, 2201, 2202, 2203, 2204, 2205, 2206, 2207, 2209, 2210, 2213, 2214, 2215, 2217, 2218, 2219, 2220, 2221, 2222, 2223, 2224, 2225, 2226, 2227, 2228, 2229, 2230, 2231, 2232, 2233, 2234, 2235, 2236, 2237, 2238, 2239, 2240, 2241, 2242, 2243, 2244, 2246, 2248, 2249, 2250, 2252, 2255, 2256, 2260, 2261, 2262, 2264, 2274, 2275, 2276, 2277, 2278, 2279, 2280, 2281, 2283, 2285, 2286, 2287, 2288, 2289, 2290, 2291, 2292, 2293, 2294, 2295, 2296, 2297, 2352, 2362, 2363, 2364, 2365, 2367, 2368, 2372, 2375, 2377, 2378, 2380, 2381, 2384, 2386, 2387, 2389, 2390, 2391, 2392, 2394, 2397, 2398, 2399, 2402, 2403, 2404, 2408, 2410, 2411, 2412, 2414, 2415, 2418, 2419, 2423, 2425, 2427, 2428, 2429, 2430, 2431, 2432, 2452, 2453, 2457, 2503, 2505, 2709, 2710, 2711, 2712, 2713, 2714, 2715, 2716, 2717, 2718, 2719, 2720, 2721, 2722, 2723, 2724, 2725, 2726, 2727, 2728, 2729, 2730, 2731, 2732, 2733, 2734, 2735, 2736, 2737, 2738, 2739, 2740, 2741, 2742, 2743, 2744, 2745, 2746, 2747, 2748, 2749, 2750, 2751, 2752, 2753, 2754, 2755, 2756, 2757

2. Memory exhaustion and resource-guard rejection — 54 datasets

Symptoms

Two subgroups are currently reported as failures:

1. 36 hard OOM kills: the 37 GiB container limit was reached. These comprise 22 DTM, 11 internal-validation, and 3 segmentation failures.
2. 18 completed analyses rejected by the 32 GiB guard: output generation reached the internal validation step, but cgroup_memory_peak_mib >= 32768. Host-side sampling sometimes appeared below 32 GiB because it missed the true cgroup peak; the performance report captured it. Examples include dataset 97 at 37,888.0 MiB, 136 at 36,512.4 MiB, 267 at 33,875.9 MiB, and 980 at 35,204.2 MiB.

The second subgroup is not corrupt output: it is deliberate enforcement of the resource contract.

Proposed resolution

• Use the ordered-COPC spatial-streaming path so DTM, CHM, segmentation, and tile reads spatially prune the input instead of repeatedly reading the original LAZ.
• Retry with the validated profile: 10 CPUs, 30 GiB container limit, 25 GiB internal guard, one catalog worker.
• Keep COPC conversion and analysis sequential so their memory peaks cannot overlap.
• If a dataset still crosses the limit, record the failing stage and peak, then reduce chunk width or add a bounded spill strategy. Do not silently raise the guard.

The large dataset-107 benchmark reduced cgroup peak from 29,023.309 MiB to 17,767.770 MiB, but this is evidence of improvement—not proof that all 54 datasets are fixed.

36 OOM-killed dataset IDs

148, 388, 404, 516, 531, 651, 657, 659, 660, 662, 664, 665, 820, 821, 823, 827, 834, 850, 893, 915, 934, 962, 981, 982, 989, 991, 992, 1013, 1090, 1101, 2003, 2355, 2358, 2359, 2361, 2800

18 completed-but-over-budget dataset IDs

97, 136, 267, 652, 654, 661, 663, 666, 771, 980, 983, 1829, 1843, 1855, 1890, 2002, 2004, 2062

3. AOI conversion false negative — 41 datasets

Symptom

GeoJSON to GeoPackage conversion changed normalized geometry, area, or bounds

Reproduction and evidence

Dataset 259 deterministically fails the AOI-only conversion in both the currently running image and the ordered-COPC image. A geometry-only differential was then run across all 41 affected AOIs.

For every affected AOI:

• GEOS topological equality: true
• exact bounds: identical
• exact order-neutral XY vertex inventory: identical
• only the floating-point area accumulation differs

The largest absolute area difference was 6.332994e-08; the largest ratio to the current tolerance was 6.177076. Dataset 259 differed by 1.117587e-08 while the current tolerance is 5.889852e-09.

This is a validation false positive caused by shoelace summation order after GDAL rotates polygon rings. Exact vertices plus topology and bounds already prove that no coordinate was added, removed, or moved.

Proposed resolution

• Make topology + exact bounds + exact order-neutral vertex inventory authoritative.
• Remove the redundant area-equality gate, or increase only its numerical tolerance with a regression fixture covering the observed worst case.
• Preserve rejection tests for moved vertices and added collinear vertices.
• Run the AOI-only differential across all 41 inputs before the full retry.

41 affected dataset IDs

225, 259, 396, 461, 467, 489, 513, 582, 584, 655, 677, 774, 802, 1085, 1095, 1247, 1254, 1260, 1277, 1279, 1287, 1288, 1294, 1308, 1317, 1318, 1324, 1325, 1735, 1746, 1747, 1748, 1749, 1769, 2010, 2059, 2072, 2073, 2706, 2707, 2782

4. Conflicting CRS metadata among DTM chunks — 11 datasets

Symptom

DTM chunk rasters contain conflicting CRS metadata

Header inspection shows that 10 source point clouds have no declared CRS; dataset 2788 declares WGS 84. The current implementation compares non-empty chunk WKT strings with raw string uniqueness. Equivalent CRS representations can therefore be treated as conflicting even when they describe the same CRS. The failed runs remove their temporary chunk rasters, so the exact conflicting WKT pairs are not retained in the current ledger.

Proposed resolution

• First retain/log the normalized CRS identity and original WKT for every distinct candidate so the 11 failures become directly inspectable.
• Compare CRS definitions semantically, not by raw WKT string identity.
• For a CRS-less source, canonicalize all chunk outputs to one explicit local/undefined CRS representation before VRT construction.
• Continue to reject genuinely incompatible declared CRSs.
• Add tests for equivalent WKT variants, missing CRS mixed with canonical undefined CRS, and a real conflicting CRS pair.

Affected datasets: 244, 389, 390, 392, 504, 505, 977, 979, 985, 1765, 2788.

5. DTM mosaic excludes empty boundary chunks — 3 datasets

Symptom

global DTM mosaic does not contain the complete pinned-lidR publication extent

Datasets 228, 422, and 423 contain many boundary chunks without enough non-collinear PTD ground points. Empty chunks are correctly emitted, but when at least one populated chunk exists the VRT builder keeps only populated paths. The subsequent coverage assertion then rejects the reduced extent before the covering VRT can extend it with NoData.

Proposed resolution

• Preserve empty boundary chunk rasters in DTM VRT construction, or build the final VRT directly with the pinned lidR target extent and NoData outside populated sources.
• Keep the exact Julia/lidR grid origin and resolution.
• Compare the resulting DTM and CHM grids and values against valid_updated for all three datasets.

Affected datasets: 228, 422, 423.

6. Extra GDAL sidecars rejected by outer validation — 7 datasets

Symptom

The image successfully validated and atomically promoted the expected scientific artifacts. The batch-level artifact comparator then rejected four additional GDAL sidecars:

<id>_chm.tif.aux.xml
<id>_chm.tif.partial.tif.aux.xml
<id>_dtm.tif.aux.xml
<id>_dtm.tif.partial.tif.aux.xml

Proposed resolution

• Compare the required artifact manifest rather than requiring directory-set equality.
• Explicitly ignore known GDAL sidecars and internal partial-file metadata.
• Still reject missing required outputs and unexpected primary scientific artifacts.
• Reclassify these seven attempts as successful after confirming their required artifacts against valid_updated.

Affected datasets: 1996, 1998, 1999, 2546, 2547, 2549, 2551.

7. Storage staging outage — dataset 588

Symptom

The container completed and promoted nine artifacts, but staging failed with:

[Errno 112] Host is down

Proposed resolution

• Retry staging after the shared filesystem is healthy.
• Make final staging retryable and idempotent without rerunning ForestStructure when the validated local attempt is complete.
• Distinguish analysis_failed from staging_failed in batch status.

Implementation checklist

• Patch AOI validation and add a 41-input AOI differential regression.
• Canonicalize/semantically compare chunk CRS metadata; validate all 11 datasets.
• Preserve the pinned DTM extent across empty boundary chunks; validate datasets 228, 422, and 423.
• Ignore GDAL sidecars in the outer required-artifact comparison and reclassify seven successful attempts.
• Add idempotent staging retry and retry dataset 588.
• Build canonical ordered COPCs with OriginalPointIndex for the failed cohort.
• Retry the 234 coordinate-scale datasets with the ordered-COPC image.
• Retry the 54 memory-related datasets under 10 CPU / 30 GiB / 25 GiB internal / one catalog worker and record actual peaks.
• Compare all available outputs against valid_updated, including primary, SAT, and FM dimensions where present.
• Publish a final machine-readable result table with dataset ID, old cause, retry status, peak memory, runtime, and oracle comparison.

Release gate

This issue is complete only when:

1. every one of the 351 attempts has a new terminal classification;
2. deterministic implementation/validation failures no longer reproduce;
3. every successful retry matches valid_updated at the established zero-tolerance scientific comparison;
4. any remaining memory failures have measured evidence and a separately tracked bounded-memory design;
5. no result is accepted by merely weakening scientific-output validation.

[kgerb]

What the ordered-COPC image from #1 already solves

PR #1 merged the ordered-COPC Julia-parity pipeline. The following mapping is intentionally conservative: it distinguishes a direct fix from an expected resource improvement and from issues that remain unresolved.

Directly patched: 234 fine-coordinate DTM failures

• tin_compatible_xy_scale() calculates an integer-safe decimal XY scale for the in-memory DTM copy.
• classified_ground_surface() catches the exact lidR integer-conversion error and retries the TIN without modifying the source file.
• The regression test exercises large absolute coordinates with 1e-7 m XY scales.
• The changelog records the dataset-2160 production validation.

This resolves the known mechanism, but the 234 failed batch entries still require a cohort rerun and valid_updated comparison before closure.

Implemented: scientific point-order preservation with COPC spatial streaming

• The runtime requires OriginalPointIndex for COPC inputs and rejects non-canonical COPCs.
• Selective COPC reads restore original order per chunk, retaining spatial pruning without exposing COPC storage order to the scientific algorithms.
• The validation harness pins the 10-CPU / 30-GiB / 25-GiB / one-worker profile and compares science plus rasters against valid_updated.
• Documented parity evidence: 36 distinct real datasets matched primary valid_updated results exactly. The total is the 15-dataset cohort (104, 144, 549, 806, 1337, 1338, 1404, 1416, 1418, 1422, 1425, 1431, 1432, 1790, 2074), the additional 20-dataset cohort (229, 1135, 1137, 1149, 1158, 1163, 1164, 1168, 1169, 1269, 1352, 1408, 1409, 1413, 1415, 1417, 1420, 1421, 1424, 1427), and large benchmark dataset 107 (15 + 20 + 1 = 36). The first 35 were validated with the immediate predecessor julia-memory-safe-copc-stream-ordinal-20260828; dataset 107 used the final julia-memory-safe-copc-all-instances-dtmfix-20260829 image now aliased as 3dtrees-foreststructure:copc. Thus, 36 is cumulative ordered-COPC implementation-lineage evidence, not 36 executions of the exact final Docker digest. Dataset 2160 is separate additional SAT/FM differential evidence, and the synthetic three-dimension fixture produced 15 byte-identical artifacts.

Likely improved, not yet proven: 54 memory-related failures

The ordered-COPC path avoids repeated whole-file LAZ access and defaults the controlled validation path to one catalog worker. The dataset-107 benchmark measured:

• cgroup peak: 29,023.309 MiB → 17,767.770 MiB (-38.8%)
• analysis runtime: 6,217.955 s → 1,996.875 s (-67.9%)
• exact primary scientific comparison on 156,184,097 points

This strongly supports retrying the 54 memory cases with the new profile, but it does not prove that all 54 remain below 30 GiB. They stay open until measured.

Not solved by #1

The ordered-COPC image still reproduces the AOI failure for dataset 259. Its source currently retains the too-tight area accumulation check.

The following also remain open:

• 41 AOI floating-point false negatives;
• 11 CRS metadata conflicts—the current code still uses raw WKT string uniqueness;
• 3 empty-boundary DTM extent failures—the current coverage precondition still rejects the reduced populated-chunk mosaic;
• 7 batch-level .aux.xml comparison failures;
• dataset 588's transient shared-storage staging failure.

Relevant merged work: PR #1, especially 7ea0302 for ordered COPC and the DTM scale fallback, and 580ff6f for the refreshed benchmark evidence.