diff --git a/.gitignore b/.gitignore index 5a88fe2..3300da8 100644 --- a/.gitignore +++ b/.gitignore @@ -4,5 +4,6 @@ __pycache__/* out/* .serena/* *md +!CONTEXT.md run_* scripts/* diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..d148078 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,194 @@ +# SmartTile Agent Context + +This file is for coding agents working on SmartTile. It is intentionally more +implementation-facing than `README.md`; use the README for user-facing behavior +and the repository root `CONTEXT.md` for shared 3Dtrees terminology. + +## Product Contract + +- Preserve uploaded point clouds as closely as the selected output format allows. +- User-facing analysis products are `original_with_predictions/` and + prod-merged files created from those originals. +- SmartTile has two distinct user-facing merge goals: + 1. Create a final merged point-cloud product next to the enriched original + files after prediction/remap. + 2. Merge multiple uploaded source files into one point-cloud product without + losing CRS, source dimensions, or metadata that remains true for the merged + product. +- COM/processed merged files are intermediate or diagnostic products. Do not use + center-of-mass geometry as the authoritative merged product for further + analysis. +- Instance labels use the simple contract: `0` is background/no tree, positive + values are tree instances, and negative labels are invalid. +- Keep prediction dimension names exactly as supplied. Multi-collection remap + must fail on duplicate output dimension names instead of auto-renaming. +- Use `uint16` for prediction labels unless a positive instance value exceeds + `65535`; then use `uint32`. + +## Task Modes + +- `tile`: converts uploaded LAZ/LAS/COPC inputs into spatial COPC tiles, then + creates subsampled products. The default first resolution is 1cm COPC LAZ; the + default second resolution is 10cm regular LAZ. +- `merge`: merges segmented tile predictions and can create Original-with- + predictions outputs through the remap path. +- `filter`: removes duplicate buffer-zone instances from segmented/remapped + tile files before downstream merge/remap workflows. +- `remap`: transfers prediction dimensions back to original source points. It + supports multiple segmented prediction collections when their dimension names + are already unique. The explicit production interface is + `--original-laz-input-dir` for uploaded raw LAZ/LAS files. Optional + `--original-copc-input-dir` may provide matching original COPCs for validation + and source context, but remap does not create enriched COPC + originals. Legacy `--original-input-dir` remains accepted as a LAZ/LAS source. +- `create_merged_file`: creates user-facing prod-merged outputs from + Original-with-predictions files. It stages LAZ/LAS inputs to COPC, reuses + existing staged COPCs when valid, and supports `copc.laz`, `laz`, and `ply`. + +## Two Merge Goals + +SmartTile's product merge behavior must support two related but different +workflows: + +1. Final product after processing: after tiling, segmentation, filtering, and + remap, users should receive `original_with_predictions/` plus one or more + prod-merged files. These prod-merged files are built from the enriched + originals so they use original uploaded points as product geometry. +2. Source-file union: users may also need multiple uploaded point-cloud files + merged into one product even when the main objective is not segmentation + cleanup. This path must keep CRS, scales, offsets, compatible source + dimensions, and truthful metadata as far as the output format allows. + +Do not confuse these goals with the processed/COM merged intermediate. A COM +merged file can be useful for diagnostics or model input/output inspection, but +it is not the analysis-grade source union and not the default final download +product. + +## Metadata And CRS Invariants + +- Original-with-predictions files represent one uploaded source file and should + preserve that source file's header, CRS/projection VLRs, scales, offsets, point + format, and non-prediction extra dimensions as far as LAS/COPC allows. +- Keep the product-enrichment source explicit: `--original-laz-input-dir` must + use uploaded non-COPC LAZ/LAS files as the metadata source for faithful + downloadable enriched originals. Optional `--original-copc-input-dir` must + only describe matching original COPCs for validation or source context. +- In remap, validate matching COPC/LAZ source pairs when the optional COPC lane + is configured, then enrich the uploaded LAZ/LAS originals directly from the + prediction collections. The COPC lane must not produce an intermediate + COPC-enriched original. +- For merged-COPC-to-original remap, stream uploaded LAZ/LAS originals in chunks + and query the merged COPC by each chunk's spatial bounds before building a + local KDTree. Do not load a full uploaded original or full merged COPC when a + bounded spatial query can produce the same enriched original output. +- Create prod-merged `copc.laz`, `laz`, and `ply` outputs from the enriched LAZ + originals rather than from COPC-original processing outputs. +- COPC output must preserve CRS metadata, including GeoTIFF/GeoKeyDirectory and + WKT projection records. Do not only preserve one projection VLR record type. +- Do not promise byte-identical raw VLR preservation after LAZ -> COPC + conversion. A COPC can preserve CRS semantically as WKT even when the uploaded + raw LAZ represented CRS with GeoKeyDirectory/GeoAscii VLRs. Use the raw lane + when the user-facing downloadable file should preserve the uploaded metadata + representation as closely as possible. A LAZ written from COPC is not + guaranteed to be identical to a LAZ enriched directly from the uploaded raw + file. +- `--standardization-json` restores the v2.1 schema guard. It reads + `collection.reference_attribute_names` from tool_standard + `collection_summary.json`, maps R/LAS names to laspy names, ignores constant + dims when global stats mark them as zero-variance, and validates that staged + Original-with-predictions COPCs and LAS/COPC prod-merged outputs still expose + those expected source dimensions. +- Multi-source prod-merged files should preserve CRS and run-true metadata, but + must not pretend one source file's source-specific metadata describes the whole + product. +- PLY is allowed as an output format, but PLY does not carry LAS/COPC VLR + metadata. Do not claim CRS/VLR preservation for PLY products. +- SmartTile assumes upstream tools ensure CRS consistency across input files. + SmartTile should preserve CRS, not perform semantic CRS reconciliation. + +## Subsampling Contract + +- `center-of-mass` is the default subsampling method. It averages only XYZ inside + each populated voxel. +- Non-coordinate attributes must not be averaged. When attributes need to remain + on subsampled points, copy them from a real nearest source point. +- `nearest-to-centroid` preserves the previous PDAL voxel nearest-neighbor + behavior. +- `--num-spatial-chunks` controls spatial parallelism for both subsampling + strategies, COPC-original remap windows, and bounded prod-merged COPC reads. +- Keep large runs memory bounded: prefer chunked COPC reads/writes, avoid one + giant in-memory point cloud, and stream batches into final products whenever + practical. + +## Module Map + +- `src/run.py`: CLI entry point and task routing. +- `src/parameters.py`: Pydantic settings, CLI parameters, and validators. +- `src/main_tile.py`: tile task orchestration. +- `src/tile_copc.py`, `src/tile_tindex.py`, `src/tile_spatial.py`, + `src/tile_bounds_graph.py`: tiling helpers. +- `src/main_subsample.py`: subsampling orchestration. +- `src/subsample_com.py`, `src/subsample_chunk_worker.py`, + `src/subsample_methods.py`, `src/subsample_outputs.py`: subsampling helpers. +- `src/main_merge.py`, `src/merge_tiles.py`, `src/merge_tiles_cli.py`: merge + task orchestration and compatibility entry points. +- `src/merge_*`: merge internals for overlap handling, instance matching, + global IDs, orphan recovery, tile loading, and original dimension handling. +- `src/main_remap.py`, `src/prediction_collection_remap.py`, + `src/output_remap.py`, `src/dimension_transfer.py`: remapping and dimension + transfer. +- `src/main_create_merged_file.py`: prod-merged product creation. +- `src/copc_metadata.py`, `src/copc_staging.py`, `src/point_cloud_metadata.py`, + `src/point_cloud_outputs.py`: metadata preservation, COPC staging, and output + writing. +- `src/instance_labels.py`, `src/worker_budget.py`, `src/union_find.py`: shared + contracts/utilities. + +## Change Safety Checklist + +Before changing product behavior, check: + +- Does the change preserve source metadata for Original-with-predictions? +- Does it preserve CRS VLRs for LAS/COPC, including WKT projection records? +- Does it keep prediction dimensions and original extra dimensions? +- If `--standardization-json` is supplied, does the output still contain the + expected standardized source dimensions? +- Does it keep 0/background and positive-instance semantics? +- Does it avoid introducing center-of-mass geometry into prod-merged products? +- Does it keep large files chunked or streamed enough for production memory + limits? +- Are README user examples and this context file still aligned? + +## Validation + +Fast local validation: + +```bash +python -m py_compile src/*.py +python -m unittest discover -s tests +git diff --check +``` + +Important test areas: + +- output format validation and `create_merged_file` products +- metadata/header/CRS preservation helpers +- COM and nearest-to-centroid subsampling selection +- scientific-notation bounds parsing +- one-pass multi-collection remap behavior +- prediction label dtype rules +- scale/offset preservation during remap + +For production-like checks, use small real datasets first, then run a multi-file +dataset through tiling, segmentation, filter/remap, detailview/remap, and +`create_merged_file`. Compare output headers and dimensions against the original +source files. + +## Current Watch Items + +- `main_subsample.py` and `main_create_merged_file.py` are still large. Prefer + extracting focused helpers instead of adding new modes inline. +- COPC finalization can be scratch-disk heavy even when memory is bounded. + Preserve the current warnings and staged-COPC reuse behavior. +- The README is the user contract. This file is the agent/developer orientation; + keep both short enough to stay useful. diff --git a/Dockerfile b/Dockerfile index 14b7f54..00155ca 100644 --- a/Dockerfile +++ b/Dockerfile @@ -51,7 +51,7 @@ COPY src/ /src/ # Create a non-root user for running the application # Create data directories with proper permissions (owned by appuser) RUN mkdir -p /in /out /src/out -RUN chmod -R 755 /in /out /src/out +RUN chmod -R a+rX /src && chmod -R 755 /in /out /src/out # Set environment variables ENV PYTHONUNBUFFERED=1 diff --git a/README.md b/README.md index 597c53a..5600ca4 100644 --- a/README.md +++ b/README.md @@ -1,142 +1,1199 @@ -# 3DTrees SmartTile +# 3DTrees Smart Tiling Pipeline -Point cloud tiling, border-instance filtering, and prediction remapping for the 3DTrees pipeline. +**A high-performance point cloud processing pipeline for 3D tree segmentation: intelligent tiling, multi-resolution subsampling, prediction remapping, and cross-tile instance merging with species ID preservation.** -## Tasks +--- -| Task | Purpose | -|------|---------| -| **tile** | COPC-normalize input files, create overlapping spatial tiles, write two subsampled output collections | -| **filter** | Remove duplicate instances in tile overlap zones; optionally remap filtered results onto target files | -| **remap** | Transfer prediction dimensions from segmented tiles back onto original (and/or subsampled) files | +## Table of Contents + +1. [Overview](#overview) +2. [Key Features](#key-features) +3. [Pipeline Architecture](#pipeline-architecture) +4. [Installation](#installation) +5. [Quick Start](#quick-start) +6. [Detailed Usage](#detailed-usage) +7. [Pipeline Stages](#pipeline-stages) +8. [Parameters Reference](#parameters-reference) +9. [Input/Output Formats](#inputoutput-formats) +10. [Advanced Configuration](#advanced-configuration) +11. [Docker & Automation](#docker--automation) +12. [Troubleshooting](#troubleshooting) +13. [Project Structure](#project-structure) +14. [Dependencies](#dependencies) +15. [License](#license) + +--- + +## Overview + +The **3DTrees Smart Tiling Pipeline** is a production-ready system designed to process large-scale LiDAR point clouds for individual tree segmentation. It addresses the fundamental challenge of processing massive datasets that exceed memory limits by intelligently dividing point clouds into manageable tiles, processing them independently, and then seamlessly merging the results. + +### The Problem + +Modern airborne and terrestrial LiDAR surveys can produce datasets with billions of points covering entire forests. Deep learning-based tree segmentation models typically operate on limited spatial extents due to memory constraints. Processing such data requires: + +1. **Spatial partitioning** - Dividing large datasets into manageable tiles +2. **Buffer zones** - Handling tree instances that span tile boundaries +3. **Multi-resolution processing** - Subsampling for efficient neural network inference +4. **Prediction upscaling** - Remapping low-resolution predictions back to high-resolution data +5. **Instance merging** - Reconnecting tree instances split across tiles + +### The Solution + +This pipeline provides an end-to-end solution with five user-facing task modes: +`tile`, `merge`, `filter`, `remap`, and `create_merged_file`. + +``` +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ TILE TASK │ +│ │ +│ Input LAZ/LAS Files │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Spatial │ │ Tile Grid │ │ Two-Phase │ │ +│ │ Index │────▶│ Calculation │────▶│ Tiling │ │ +│ │ (tindex) │ │ (bounds) │ │ (laspy+COPC) │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────┐ │ +│ │ Multi-Resolution │ │ +│ │ Subsampling │ │ +│ │ (1cm + 10cm) │ │ +│ └───────────────────┘ │ +│ │ │ +│ ▼ │ +│ Outputs: tiles_100m/ │ +│ ├─ c00_r00.copc.laz │ +│ ├─ subsampled_1cm/ │ +│ └─ subsampled_10cm/ │ +└─────────────────────────────────────────────────────────────────────────────────┘ + │ + ▼ + [External Segmentation] + (e.g., ForAINet, SegmentAnyTree) + │ + ▼ +┌─────────────────────────────────────────────────────────────────────────────────┐ +│ MERGE TASK │ +│ │ +│ Segmented 10cm Tiles (with PredInstance attribute) │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ +│ │ Prediction │ │ Buffer │ │ Cross-tile │ │ +│ │ Remapping │────▶│ Filtering │────▶│ Instance │ │ +│ │ (10cm→1cm) │ │ │ │ Matching │ │ +│ └──────────────┘ └──────────────┘ └──────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────┐ │ +│ │ Deduplication + Small Volume Merge │ │ +│ └───────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌───────────────────────────────────┐ │ +│ │ Remap to Original Input Files │ │ +│ └───────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ Unified Point Cloud │ +│ (Consistent Instance IDs Across Tiles) │ +└─────────────────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Key Features + +### High Performance +- **Parallel processing** using Python's `ProcessPoolExecutor` for multi-core utilization +- **COPC format output** via untwine (with automatic PDAL fallback) for efficient spatial queries and streaming access +- **Memory-efficient chunking** - tiles are processed independently to minimize memory footprint +- **Parallel subsampling** - each tile is spatially divided into chunks processed concurrently + +### Intelligent Tiling +- **Configurable tile size** - default 100m × 100m, adjustable for different use cases +- **Buffer zones** - overlapping regions (default 20m) ensure trees at boundaries are fully captured +- **Spatial indexing** - uses PDAL tindex for efficient data retrieval +- **Data-aligned grid** - tiles start from actual data extent, minimizing empty tiles +- **Smart tiling threshold** - optional single-file bypass for small datasets + +### Multi-Resolution Processing +- **Dual subsampling** - generates both 1cm and 10cm resolution outputs (configurable) +- **Selectable voxel subsampling** - defaults to SmartTile `center-of-mass` XYZ averaging; `nearest-to-centroid` preserves the previous PDAL voxel centroid nearest-neighbor behavior +- **Explicit dimension policy** - intermediate COPC conversion strips extra point attributes by default; prod-merged creation preserves enriched dimensions + +### Smart Instance Merging +- **Centroid-based filtering** - removes duplicate instances in buffer zones +- **Overlap ratio matching** - identifies same trees across tile boundaries using point correspondence +- **Union-Find algorithm** - efficiently groups matched instances into unified trees +- **Species ID preservation** - always preserves species from the larger instance fragment +- **Small volume merging** - reassigns orphaned tree fragments to nearby larger instances +- **Original file remapping** - maps predictions back to original input files + +--- + +## Pipeline Architecture -## Quick start +### Stage-by-Stage Breakdown + +#### TILE TASK: Data Preparation + +| Stage | Component | Description | +|-------|-----------|-------------| +| 1 | **Spatial Index** | Creates a GeoPackage tindex using `pdal tindex` for efficient spatial queries across all input files. | +| 2 | **Tile Bounds** | Calculates optimal tile grid based on data extent, tile size (default: 100m), and buffer (default: 20m) parameters. | +| 3a | **Phase 1: Distribute** | Reads each source LAZ/LAS file once (in memory-efficient chunks via laspy), distributes points to overlapping tiles as intermediate part files. Per-tile offsets prevent int32 overflow. | +| 3b | **Phase 2: COPC Conversion** | Merges part files and converts each tile to COPC format using untwine (fast, automatic fallback to PDAL). | +| 4 | **Subsampling R1** | Downsamples COPC tiles to resolution 1 (default: 1cm) using parallel spatial chunk processing. | +| 5 | **Subsampling R2** | Further downsamples to resolution 2 (default: 10cm) for neural network inference. | + +#### MERGE TASK: Result Integration + +| Stage | Component | Description | +|-------|-----------|-------------| +| 0 | **Prediction Remapping** | Transfers PredInstance labels from 10cm predictions to 1cm resolution using KDTree nearest-neighbor lookup. | +| 1 | **Load and Filter** | Loads tiles, applies centroid-based buffer zone filtering to remove duplicate instances. | +| 2 | **Global ID Assignment** | Creates unique instance IDs across all tiles using tile-specific offsets. | +| 3 | **Cross-tile Matching** | Identifies matching instances in tile overlaps using overlap ratio (Union-Find grouping). | +| 3b | **Orphan Recovery** | Recovers filtered instances that no neighbor "covers", so no trees are lost. | +| 4 | **Merge and Deduplicate** | Combines all tiles, removes duplicate points from overlapping buffer regions. | +| 5 | **Small Volume Merge** | Reassigns tree fragments with volume < 4m³ to nearest large instance. | +| 6 | **Retiling** | Maps final instance IDs back to original tile boundaries for per-tile output. | +| 7 | **Original Remap** | Maps final instance IDs back to original input LAZ files (pre-tiling, optional). | + +--- + +## Installation + +### Conda Environment ```bash -# Show all parameters +# Create conda environment +mamba create -n 3dtrees -c conda-forge \ + python=3.10 \ + pdal=2.6 \ + untwine \ + gdal \ + laspy \ + lazrs-python \ + numpy \ + scipy \ + matplotlib-base \ + fiona \ + pyproj \ + geopandas \ + pydantic \ + pydantic-settings + +# Activate environment +conda activate 3dtrees + +# Verify installation python src/run.py --show-params +``` + +### System Requirements -# Tile +- **Operating System**: Linux (tested on Ubuntu 20.04+), macOS, Windows with WSL2 +- **Memory**: Minimum 8GB RAM, 16GB+ recommended for large datasets +- **CPU**: Multi-core processor recommended (parallel processing scales with cores) +- **Storage**: SSD recommended for I/O-intensive operations +- **PDAL**: Version 2.5 or higher +- **GDAL**: Version 3.0 or higher + +--- + +## Quick Start + +### Basic Tile Task + +Process a directory of LAZ files into tiled, multi-resolution outputs: + +```bash python src/run.py --task tile \ - --input-dir /data/input --output-dir /data/output \ - --tile-length 300 --tile-buffer 20 \ - --resolution-1 0.01 --resolution-2 0.1 \ - --output-copc-res1 True \ - --output-copc-res2 False - -# Filter (tile_bounds_json is optional — neighbors are auto-detected from cXX_rYY filenames) -# Segmented inputs may be LAZ/LAS/COPC. + --input-dir /path/to/input \ + --output-dir /path/to/output +``` + +### Basic Merge Task + +Merge segmented tiles and create processed per-tile outputs plus the current processed merged LAZ +(requires **tile_bounds_tindex.json** from the Tile task): + +```bash +python src/run.py --task merge \ + --subsampled-segmented-folder /path/to/subsampled_10cm \ + --subsampled-target-folder /path/to/subsampled_1cm \ + --tile_bounds_json /path/to/tile_bounds_tindex.json \ + --output-folder /path/to/out \ + --output-merged-laz /path/to/out/merged.laz +``` + +Optional: add `--original-input-dir /path/to/original` to also write `original_with_predictions/`. + +### Basic Filter Task + +Filter segmented or remapped tiles by removing instances whose centroids sit in +overlap buffers facing neighboring tiles: + +```bash python src/run.py --task filter \ - --segmented-folders /data/segmented \ - --tile-bounds-json /data/tile_bounds_tindex.json \ + --input-dir /path/to/segmented_remapped \ + --output-dir /path/to/filtered_tiles \ + --buffer 10.0 \ + --instance-dimension PredInstance +``` + +The filter task writes one output file per input file and preserves all point +dimensions in the kept points. Use `--filter-suffix` to change the default +`_filtered` filename suffix. Use `--filter-output-extension .laz` when mixed +LAZ/LAS inputs should be collected as LAZ outputs. + +### Create Prod-Merged Products + +Create user-facing prod-merged files from `original_with_predictions/`. This mode serves two +production goals: + +1. Write a final merged point-cloud product next to the enriched original files after remap. +2. Merge multiple uploaded source files into one point-cloud product while preserving CRS, + source dimensions, scales, offsets, and truthful LAS/COPC metadata as far as the output + format allows. + +Inputs are first staged through preservation-mode COPC, then real original points are +selected with nearest-to-centroid product downsampling: + +```bash +python src/run.py --task create_merged_file \ + --original-with-predictions-dir /path/to/original_with_predictions \ + --output-dir /path/to/products \ + --staged-copc-dir /path/to/already_converted_original_with_predictions_copc \ + --standardization-json /path/to/collection_summary.json \ + --merged-resolutions res1,res2 \ + --merged-output-formats copc.laz +``` + +By default this writes `prod_merged_1cm.copc.laz` and `prod_merged_10cm.copc.laz`. +Use `--merged-output-formats laz,copc.laz,ply` to write multiple formats for each selected resolution. +The intermediate COPC files are written under `original_with_predictions_copc/` in the output directory. +If matching `.copc.laz` files are already present for an Original-with-predictions source, they are reused and the matching raw LAZ/LAS file is not staged a second time. +Use `--staged-copc-dir` to reuse an explicit COPC cache from a previous product or validation run. SmartTile checks that each staged COPC has a readable header before using it, so interrupted partial conversions are ignored and rebuilt in the current output directory. +Use `--standardization-json` with the tool_standard `collection_summary.json` to validate that staged Original-with-predictions COPCs and LAS/COPC prod-merged outputs still expose the expected standardized source dimensions. This restores the v2.1 schema guard; it does not filter prediction dimensions. + +### Basic Remap Task (merged file → original files) + +Add 3Dtrees prediction dimensions from a merged LAZ file to the original files, +then create prod-merged products from those Original-with-predictions files: + +```bash +python src/run.py --task remap \ + --merged-laz /path/to/merged.laz \ + --original-input-dir /path/to/original/files \ + --output-dir /path/to/original_with_predictions \ + --merged-resolutions res1,res2 \ + --merged-output-formats laz,copc.laz +``` + +When `--merged-laz` points to a `.copc.laz` file, SmartTile uses a bounded +streaming remap path: uploaded original LAZ/LAS files are read in +`--chunk-size` point chunks, each original chunk queries the merged COPC by its +XY bounds plus remap buffer, a local KDTree is built only for that window, and +the enriched original chunk is written immediately. This keeps the uploaded +LAZ/LAS file as the metadata and geometry source while avoiding a full-original +or full-merged KDTree in memory. Plain merged `.laz` inputs fall back to the +legacy loaded-merged-cloud path. + +### Multi-Collection Remap Task (prediction collections → original files) + +Finalized prediction collections can be remapped together onto the original +files. This is intended for model outputs that have already been filtered and +merged independently. SmartTile preserves prediction dimension names exactly as +provided; model-specific names such as `PredInstance_SAT`, +`PredInstance_ForestMamba`, `species_id_sat`, and `species_prob_foma` must be +present before this step. + +```bash +python src/run.py --task remap \ + --segmented-folders /path/to/sat_predictions,/path/to/foma_predictions,/path/to/species_predictions \ + --original-copc-input-dir /path/to/original_copc_files \ + --original-laz-input-dir /path/to/uploaded/raw_laz_files \ + --original-laz-output-dir /path/to/original_with_predictions_raw \ + --remap-dims PredInstance_SAT,PredSemantic_SAT,PredInstance_ForestMamba,species_id_sat,species_prob_sat,species_id_foma,species_prob_foma \ + --chunk-size 10000000 \ + --merged-resolutions 1cm \ + --merged-output-formats laz +``` + +If `--remap-dims` is omitted, all extra dimensions from every prediction +collection are transferred. Duplicate extra-dimension names across prediction +collections fail early; SmartTile does not auto-rename them to `_2`, `_3`, or +add late model suffixes during final remap. + +Multi-collection remap writes each Original-with-predictions file in one pass: +for every original-point chunk it loads the overlapping prediction points from +each finalized collection, transfers all selected dimensions, and writes the +enriched original chunk once. This avoids temporary per-collection LAZ rewrites. +Use `--chunk-size` to tune the memory/speed tradeoff. Larger chunks reduce +repeated prediction-window scans but increase peak memory; for large two-file +datasets under a 30GB container cap, `10000000` points per chunk has been a +useful validation setting. + +Prediction collections stored as COPC are loaded with bounded COPC spatial +queries when remap needs a local prediction window. + +For production downloads, pass the uploaded LAZ/LAS files to +`--original-laz-input-dir`; this is the writer and metadata source for +`original_with_predictions_raw/`. `--original-input-dir` remains accepted as a +legacy alias for the same LAZ/LAS source. If original COPCs are available, pass +them to `--original-copc-input-dir` only as a matching/validation lane. SmartTile +validates that COPC and LAZ sources match, but it does not write an enriched +COPC-original intermediate. Selected prediction dimensions are written directly +onto the uploaded LAZ/LAS files, so headers, GeoTIFF/GeoKey VLRs, scales, +offsets, point format, and non-prediction dimensions come from the uploaded file +itself. Prod-merged `copc.laz`, `laz`, and `ply` outputs are then created from +the enriched uploaded LAZ/LAS files. COPC-derived products preserve CRS +semantically, but a LAZ -> COPC conversion may represent the same CRS as WKT VLR +rather than the original GeoKey VLRs. Therefore SmartTile does not promise that a +COPC-derived LAZ is byte-identical to enriching the raw uploaded LAZ directly. + +### View Current Parameters + +```bash +python src/run.py --show-params +``` + +--- + +## Detailed Usage + +### Tile Task Options + +```bash +python src/run.py --task tile \ + --input-dir /path/to/input \ # Required: Directory with LAZ/LAS files + --output-dir /path/to/output \ # Required: Output directory + --tile-length 100 \ # Tile size in meters (default: 100) + --tile-buffer 20 \ # Buffer overlap in meters (default: 20) + --resolution-1 0.01 \ # First resolution (default: 1cm) + --resolution-2 0.1 \ # Second resolution (default: 10cm) + --output-copc-res1 True \ # 1cm output as COPC LAZ (default: True) + --output-copc-res2 False \ # 10cm output as regular LAZ (default: False) + --workers 8 \ # Parallel workers (default: 4) + --threads 10 # Threads per COPC writer (default: 10) +``` + +### Create Merged File Task Options + +```bash +python src/run.py --task create_merged_file \ + --original-with-predictions-dir /path/to/original_with_predictions \ + --output-dir /path/to/products \ + --resolution-1 0.01 \ + --resolution-2 0.1 \ + --merged-resolutions res1,res2 \ + --merged-output-formats laz,copc.laz,ply \ + --staged-copc-dir /path/to/products/original_with_predictions_copc \ + --standardization-json /path/to/collection_summary.json \ + --num-spatial-chunks 10 +``` + +`--merged-resolutions` accepts `res1`, `res2`, numeric meter values such as `0.05`, or centimeter labels such as `1cm,10cm`. +`--merged-output-formats` accepts `laz`, `copc.laz`, and `ply`; it can contain one or several comma-separated formats. +The task stages LAZ/LAS inputs to COPC in preservation mode with untwine when available, falling back to PDAL `writers.copc`, before merging and product downsampling. Existing matching `.copc.laz` files are reused so one source is not merged twice. +`--staged-copc-dir` points to a reusable cache of already converted Original-with-predictions COPCs. This is recommended for repeat validation runs and production reruns where the enriched originals have not changed. +`--standardization-json` points to the standardization `collection_summary.json` and validates that expected non-constant source dimensions survived into the staged COPCs and final LAS/COPC prod-merged products. +LAZ and COPC outputs use LAS/COPC metadata forwarding. PLY outputs carry point dimensions as PLY properties, but do not preserve LAS/COPC VLR metadata such as CRS records. +`--num-spatial-chunks` controls bounded COPC reads for prod-merged creation. For large UTM datasets, prefer setting it to the available CPU budget (for example `10`) instead of using a single global merge. +For COPC output, SmartTile first writes bounded LAZ chunks and then prefers direct `untwine` chunk-to-COPC finalization. This keeps RAM bounded and avoids a giant merged temporary LAZ, but it still needs scratch disk for the chunk files and untwine hierarchy/output staging. Direct untwine output is accepted only when its point count exactly matches the source chunk total; otherwise SmartTile falls back to the PDAL merge/conversion path. +When multiple product formats are selected for the same resolution, SmartTile generates one canonical set of nearest-to-centroid chunk LAZ files and writes all selected formats from those same chunks. This avoids repeated chunk computation and keeps LAZ, COPC LAZ, and PLY point counts aligned for a given resolution. +Chunked nearest-to-centroid product generation is designed to preserve product metadata, CRS, scales, offsets, and dimensions. It should not be treated as a bit-for-bit reproducible operation: different chunking, PDAL/untwine versions, or parallel execution details may change the selected representative point at voxel boundaries while preserving the same spatial extent and metadata contract. + +### Local Validation + +Run the unit suite from the tool directory: + +```bash +python -m unittest discover -s tests -p 'test_*.py' +``` + +The suite covers output format validation, metadata/header preservation helpers, COM subsampling method selection, scientific-notation bounds parsing, one-pass multi-collection remap behavior, and instance-label dtype rules. + +### Merge Task Options + +**Required:** `--subsampled-segmented-folder`, `--subsampled-target-folder`, `--tile_bounds_json` (from Tile task). + +```bash +python src/run.py --task merge \ + --subsampled-segmented-folder /path/to/10cm \ # Segmented 10cm tiles + --subsampled-target-folder /path/to/1cm \ # Subsampled 1cm tiles (remap target) + --tile_bounds_json /path/to/tile_bounds_tindex.json \ + --output-folder /path/to/out \ # Optional; default: parent of segmented + --output-merged-laz /path/to/out/merged.laz \ # Optional; merged LAZ path + --original-copc-input-dir /path/to/original_copc \ # Optional; matching/validation source + --original-laz-input-dir /path/to/raw_original \ # Optional; uploaded LAZ/LAS source to enrich + --merged-resolutions res1,res2 \ # Prod-merged outputs from original_with_predictions + --buffer 10.0 \ # Buffer zone distance (default: 10m) + --overlap-threshold 0.3 \ # Instance matching (default: 0.3) + --max-centroid-distance 3.0 \ # Max centroid distance (default: 3m) + --workers 8 \ # Parallel workers (default: 4) + --disable-matching # Disable cross-tile matching +``` + +--- + +## Pipeline Stages + +### Stage 1: Spatial Indexing + +**Purpose**: Create a spatial index (tindex) for efficient querying across all input files. + +**Technology**: Uses `pdal tindex` to create a GeoPackage with file boundaries. + +**Output**: `tindex_100m.gpkg` containing polygons representing each input file's extent. + +### Stage 2: Tile Grid Calculation + +**Purpose**: Compute optimal tile boundaries based on data extent and parameters. + +**Algorithm**: +1. Load extent from tindex +2. Apply grid offset to starting coordinates +3. Create tiles of `tile_length` × `tile_length` meters +4. Add `tile_buffer` meters to each side +5. Generate job list with projected and geographic bounds + +**Output Files**: +- `tile_bounds_tindex.json`: Complete tile metadata +- `tile_jobs_100m.txt`: Per-tile processing instructions +- `overview_copc_tiles.png`: Visualization of tiles and input files + +### Stage 3: Tile Creation (Two-Phase) + +**Purpose**: Distribute points from source LAZ/LAS files into spatially partitioned tiles in COPC format. + +#### Phase 1: Distribute + +Each source file is read once using laspy (in memory-efficient chunks controlled by `--chunk-size`). Points are distributed to all overlapping tiles as intermediate `.las` part files. Per-tile offsets are computed from tile bounds to prevent int32 overflow in scaled coordinates. + +**Parallelization**: One source file at a time, but tile writing is batched. + +#### Phase 2: COPC Conversion + +All part files for each tile are merged and converted to COPC format. Untwine is preferred for COPC writing. Intermediate COPC conversion strips extra point attributes while forwarding header metadata and CRS records. SmartTile first asks Untwine for an empty extra-dimension keep-list with `--dims ""`. If the installed Untwine rejects that form or the output still contains extra dimensions, SmartTile writes a temporary standard-dimension LAZ and then converts that file with Untwine. Prod-merged creation is the explicit exception and preserves enriched dimensions for final products. + +The SmartTile Docker image has been validated against Untwine 1.5.1: direct `--dims ""` is rejected, `--dims Classification` does not strip all extra dimensions on real inputs, and long explicit standard-dimension lists such as `--dims Red,Green,Blue` are not used because they can crash this build. Keep the output-header validation and fallback in place when changing this path. + +**Parallelization**: Multiple tiles converted concurrently (controlled by `--workers`). + +**Output**: `c{col}_r{row}.copc.laz` files in `tiles_{tile_length}m/`. + +**Options**: +- **Default**: Keep standard LAS dimensions and strip extra point attributes from intermediate COPC outputs. +- **Preserved LAZ intermediates**: Set `--skip-dimension-reduction true` only when non-COPC intermediate files must keep extra dimensions. +- **Prod-merged exception**: `create_merged_file` preserves extra dimensions in staged COPCs and final LAZ/COPC products. + +### Stage 4-5: Multi-Resolution Subsampling + +**Purpose**: Create downsampled versions for efficient neural network processing. + +**Algorithm**: Voxel-based downsampling using the selected `--subsampling-method`. + +- `center-of-mass` (default): averages only X/Y/Z within each populated voxel. Non-coordinate dimensions are copied from the real point nearest to that averaged XYZ when dimensions are preserved; they are never averaged. +- `nearest-to-centroid`: uses PDAL `filters.voxelcentroidnearestneighbor`, preserving the previous SmartTile behavior. + +**Process**: +1. COPC `center-of-mass` runs voxel-aligned COPC windows in parallel, controlled by `--num-spatial-chunks` +2. Other subsampling paths split each tile spatially into chunks along X-axis +3. Chunks/windows are processed in parallel +4. Results are merged back into single file + +**Outputs**: +- `subsampled_res1/`: Resolution_1 files, default 1cm COPC LAZ (`*.copc.laz`) +- `subsampled_res2/`: Resolution_2 files, default 10cm regular LAZ (`*.laz`) + +### The Merge Process in Detail + +The merge process takes segmented tiles (each with local `PredInstance` IDs) and produces a single unified point cloud where every tree has a globally unique ID, even trees that were split across tile boundaries. It also writes per-tile outputs and optionally maps predictions back to the original input files. + +The diagram below shows the spatial layout. Tiles overlap by a configurable buffer (default 10 m). Trees near tile edges appear in both tiles with **different** local instance IDs. The merge must figure out which IDs in adjacent tiles refer to the same physical tree and unify them. + +``` +Tile A Tile B +┌──────────────────────┐ ┌──────────────────────┐ +│ │ │ │ +│ Tree 42 │ │ Tree 17 │ +│ ╲ │ │ ╱ │ +│ ╲ buffer │ │ buffer ╱ │ +│ ╲ zone ──────┤ ├──── zone ╱ │ +│ ╲ ▓▓▓▓▓▓▓▓▓│ │▓▓▓▓▓▓▓▓▓ ╱ │ +│ ╲ ▓overlap▓│ │▓overlap▓ ╱ │ +│ ╲ ▓▓▓▓▓▓▓▓▓│ │▓▓▓▓▓▓▓▓▓ ╱ │ +│ ╲──────────┤ ├──────────╱ │ +│ │ │ │ +│ Tree 42 and Tree 17 are the SAME physical │ +│ tree — the merge must unify them. │ +└──────────────────────┘ └──────────────────────┘ +``` + +Below is each stage explained. + +--- + +### Stage 0: Prediction Remapping (10 cm to 1 cm) + +**What it does**: The segmentation model ran on 10 cm subsampled tiles. This stage transfers the `PredInstance` labels from the coarse 10 cm points to the finer 1 cm subsampled points using nearest-neighbor lookup, so subsequent stages work at 1 cm resolution. + +**How it works**: +1. For each tile, load the 10 cm segmented file and the corresponding 1 cm subsampled file. +2. Build a cKDTree from the 10 cm points. +3. For every 1 cm point, find the closest 10 cm point and copy its `PredInstance` (and `species_id` if present). +4. Save the result as `{tile_id}_segmented_remapped.laz`. + +**Why**: Segmentation at 10 cm is fast but loses detail. Remapping to 1 cm gives the merge much denser point clouds to work with, which improves overlap detection and deduplication accuracy. + +--- + +### Stage 1: Load and Filter (Centroid-Based Buffer Filtering) + +**What it does**: Loads all tiles and removes instances whose centroid is in the buffer zone on any side that has a neighbor tile. This eliminates most duplicates before the expensive matching stage. + +**How it works**: + +1. Load each LAZ tile; read XYZ coordinates, `PredInstance`, and any extra dimensions (e.g. `species_id`). +2. Use the `tile_bounds_tindex.json` file (from the Tile task) to determine which tiles are neighbors (east/west/north/south). +3. For each tile, compute the centroid of every instance. +4. An instance is **filtered** (removed) if its centroid falls in the buffer zone on a side that has a neighbor. The rule is: the tile with the lower column index (west) or lower row index (south) "owns" instances in the overlap. +5. Instances **not** in any buffer zone are **kept**. + +**Result**: Each tile now has a set of "kept" instances and a set of "filtered" instances. Filtered instances are candidates for orphan recovery later. + +``` +Tile boundary: +┌──────────────────────────────────────┐ +│ buffer │ │ buffer│ +│ zone │ CORE AREA │ zone │ +│ (west) │ (instances here │(east) │ +│ │ are always kept) │ │ +│ ▓▓▓▓▓▓ │ │▓▓▓▓▓▓│ +│ filtered│ │filtered│ +│ if west │ │if east│ +│ neighbor│ │neighbor│ +└──────────────────────────────────────┘ +``` + +--- + +### Stage 2: Global ID Assignment + +**What it does**: Assigns globally unique IDs to every kept instance across all tiles. + +**How it works**: +- Each tile gets an offset: `tile_idx * 100000`. +- A local instance ID `42` in tile 3 becomes global ID `300042`. +- A Union-Find data structure is initialized with one entry per kept instance, tracking its point count (used later to preserve the species ID from the larger fragment). + +**Why**: Local IDs are only unique within one tile. Subsequent stages need IDs that are unique across the entire dataset. + +--- + +### Stage 3: Border Region Instance Matching + +**What it does**: Finds instances that represent the **same physical tree** across tile boundaries and groups them using Union-Find, so they end up with the same final ID. + +**How it works**: + +1. **Identify border instances**: For each tile, find kept instances whose centroid lies in the "border region" (the strip from `buffer` to `buffer + border_zone_width` meters from the tile edge, on sides with a neighbor). Only these instances can possibly match a counterpart in a neighbor tile. + +2. **For each neighbor pair** (e.g. tile A east <-> tile B west): compare border instances from tile A facing east with border instances from tile B facing west: + - **Bounding box check** (fast filter): Skip pairs whose 2D bounding boxes don't overlap or come within 10 cm. This eliminates most non-matching pairs cheaply. + - **FF3D overlap ratio** (expensive check): For surviving pairs, compute point-to-point correspondence using hash-based grid matching (O(n) per pair). The overlap ratio is `max(intersection/size_a, intersection/size_b)` -- this handles asymmetric cases where one fragment is much larger than the other. + - **Match decision**: If the overlap ratio exceeds `overlap_threshold` (default 0.3 = 30%), the pair is matched. Union-Find merges their global IDs into one group. + +3. **Species ID preservation**: When instances are grouped, the species ID is always taken from the **larger** instance (by point count). + +**Result**: A mapping from every global ID to a "merged ID". Matched instances share the same merged ID. + +``` +Tile A (east border) Tile B (west border) +┌──────────┐ ┌──────────┐ +│ │ border zone │ │ +│ inst 42 ─┼─────────────────┼─ inst 17 │ overlap ratio = 0.65 +│ (500 pts)│ │ (480 pts)│ > threshold 0.3 → MATCH +│ │ │ │ +│ inst 99 ─┼─────────────────┼─ (none) │ no counterpart → no match +│ │ │ │ +└──────────┘ └──────────┘ +``` + +--- + +### Stage 3b: Orphan Recovery + +**What it does**: Recovers filtered instances that would otherwise be lost because no neighbor tile "owns" them (e.g. a tree whose centroid landed in the buffer zone of **both** tiles due to slightly different segmentation results). + +**How it works**: + +1. Compute bounding boxes **only** for filtered instances and kept instances in the border region (not all instances -- this is fast). +2. Build a cKDTree of the centers of all border-region kept instances. +3. For each filtered (orphan) instance: + - Query the tree for kept instances within 30 m. + - For each nearby kept instance, check if their points overlap (>50% of the orphan's points are within 1 m of the neighbor's points, using a per-neighbor cKDTree that is cached so each neighbor tree is built at most once). + - If a covering instance is found, the orphan is skipped (it already exists in a neighbor tile). + - If **no** covering instance is found, the orphan is **recovered**: it gets a new unique merged ID and is added back to the kept set. + +**Why**: Without this step, trees at tile corners (where buffer zones of multiple tiles overlap) can be lost entirely. + +--- + +### Stage 4: Merge and Deduplicate + +**What it does**: Concatenates all kept points from all tiles into a single point cloud and removes duplicate points that exist in overlapping buffer regions. + +**How it works**: + +1. For each tile, remap local instance IDs to their final merged IDs (from Stage 3). Points belonging to filtered instances that were not recovered get ID `-1` and are discarded. +2. Concatenate all tile points, instance arrays, and extra dimension arrays. +3. **Deduplication**: Uses grid-based spatial hashing (not KDTree) for O(n) performance: + - Divide the point cloud into 50 m grid cells. + - Within each cell, hash each point's coordinates at 1 cm resolution. + - Points with the same hash in the same cell are duplicates; keep the one with the higher instance ID. + +**Result**: A single merged point cloud with no duplicate points and consistent instance IDs. + +--- + +### Stage 5: Small Volume Instance Merging + +**What it does**: Reassigns tiny tree fragments (orphaned clusters) to the nearest large instance. + +**How it works**: + +1. For each instance with a positive ID, compute its 3D convex hull volume (in parallel using ProcessPoolExecutor). +2. Classify instances as "small" if: volume < `max_volume_for_merge` (default 4 m3) **or** point count < `min_cluster_size` (default 300). +3. Build a cKDTree from the centroids of all "large" instances. +4. For each small instance, find the nearest large instance by centroid distance. +5. Reassign all points of the small instance to the nearest large instance (vectorized lookup table, no per-point loop). + +**Species preservation**: The species ID of the receiving (larger) instance is kept unchanged. Small fragments do not overwrite species. + +--- + +### Stage 6: Retiling to Original Tile Files + +**What it does**: Maps the final merged instance IDs back onto each original tile's point cloud, so you get per-tile output files with globally consistent IDs. + +**How it works**: + +1. For each original tile file: + - Read the tile's bounding box from its header. + - Filter the merged point cloud to points within `tile bounds + spatial buffer`. + - Build a cKDTree from those filtered merged points. + - Read the original tile's points and query the tree for the nearest merged point for each original point. + - Write the tile with the matched `PredInstance` (and any extra dims like `species_id`). + +**Parallelism**: Can process multiple tiles concurrently using `--workers`. + +--- + +### Stage 7: Uploaded Original Remapping (Optional) + +**What it does**: If `--original-laz-input-dir` is provided, maps the final merged prediction dimensions back to the uploaded original LAZ/LAS files before tiling. `--original-input-dir` remains a legacy alias for this raw-LAZ lane. If matching original COPCs are available, pass them via `--original-copc-input-dir` for source matching/validation; SmartTile still writes enriched originals from the uploaded LAZ/LAS files so the original metadata/VLRs remain the writer source. + +**How it works**: For a plain merged LAZ source, SmartTile uses spatial filtering plus a local cKDTree. For a merged COPC source, uploaded originals are streamed in chunks; each chunk queries only the overlapping merged-COPC window and writes enriched original chunks immediately. Multi-collection remap follows the same original-writer contract while transferring already model-named prediction attributes. + +--- + +## Parameters Reference + +### Tile Task Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `--tile-length` | 100 | Tile size in meters | +| `--tile-buffer` | 20 | Buffer overlap in meters | +| `--threads` | 10 | Threads per COPC writer | +| `--workers` | 4 | Parallel file/tile processing | +| `--num-spatial-chunks` | `--workers` | Per-file subsampling parallelism for COM windows or stripe chunks | +| `--resolution-1` | 0.01 | First subsampling resolution (1cm) | +| `--resolution-2` | 0.1 | Second subsampling resolution (10cm) | +| `--output-copc-res1` | True | Write first-resolution subsampled outputs as COPC LAZ (`*.copc.laz`) | +| `--output-copc-res2` | False | Write second-resolution subsampled outputs as COPC LAZ; default keeps 10cm as regular LAZ | +| `--subsampling-method` | center-of-mass | Subsampling method: `center-of-mass` or `nearest-to-centroid` | +| `--skip-dimension-reduction` | False | Keep extra dimensions in LAZ intermediates; intermediate COPC conversion still strips extra attributes by default | +| `--chunk-size` | 20000000 | Points per chunk when reading LAZ/LAS in tiling Phase 1, multi-collection remap, and merged-COPC-to-original remap (smaller = less peak RAM; larger = fewer scans) | +| `--tiling-threshold` | None | File size threshold in MB for skipping tiling on single small files | + +### Merge Task Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `--tile_bounds_json` | **Required** | Path to tile_bounds_tindex.json from Tile task | +| `--buffer` | 10.0 | Buffer distance for filtering (meters) | +| `--border-zone-width` | 10.0 | Width of border zone beyond buffer for instance matching (meters) | +| `--overlap-threshold` | 0.3 | Overlap ratio for instance matching (30%) | +| `--max-centroid-distance` | 3.0 | Max centroid distance to merge (meters) | +| `--max-volume-for-merge` | 4.0 | Max volume for small instance merge (m³) | +| `--min-cluster-size` | 300 | Minimum cluster size in points for reassignment | +| `--original-laz-input-dir` | None | Optional: uploaded original LAZ/LAS files to enrich; `--original-input-dir` is a legacy alias | +| `--original-laz-output-dir` | None | Optional output directory for enriched uploaded originals | +| `--original-copc-input-dir` | None | Optional matching original COPC LAZ files for source matching/validation | +| `--merged-resolutions` | res1,res2 | Prod-merged output resolutions when original inputs are available | +| `--merged-output-formats` | copc.laz | Prod-merged output formats: `laz`, `copc.laz`, `ply` | +| `--skip-merged-file` | False | Skip creating the processed merged LAZ intermediate (prod-merged outputs still come from Original-with-predictions when enabled) | +| `--disable-matching` | False | Disable cross-tile instance matching | +| `--disable-volume-merge` | False | Disable small volume instance merging | +| `--workers` | 4 | Parallel processing (tile loading, KDTree queries) | +| `--tolerance` | 5.0 | Max difference in meters for bounds matching (remap task) | + +*Retile buffer is fixed internally at 2.0 m; correspondence tolerance is no longer a user parameter.* + +### Filter Task Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `--input-dir` | **Required** | Directory with segmented/remapped LAZ/LAS tile files | +| `--output-dir` | **Required** | Directory for filtered tile outputs | +| `--buffer` | 10.0 | Buffer distance in meters | +| `--instance-dimension` | PredInstance | Instance ID dimension to filter; falls back to `treeID` when absent | +| `--filter-suffix` | _filtered | Suffix added to output filenames | +| `--filter-output-extension` | None | Optional extension override such as `.laz`; by default the input extension is preserved | + +### Understanding `--workers`, `--threads`, and `--num-spatial-chunks` + +These parameters control different aspects of parallelism: + +#### `--workers` (Global Parallelism) + +Controls how many files/tasks run simultaneously using Python's `ProcessPoolExecutor`: + +| Task | What `--workers` Controls | +|------|---------------------------| +| **Tile Task** | Parallel COPC conversions, parallel tile creation | +| **Merge Task** | Parallel tile loading, parallel convex hull computation, KDTree queries | +| **Remap Task** | Parallel files/tiles; KDTree query workers are divided across those outer workers | + +**Memory impact**: Higher values = more files in memory simultaneously. Remap keeps the total KDTree CPU budget bounded by sharing `--workers` across outer remap workers and inner SciPy query workers. + +#### `--threads` (COPC Writer Threads) + +Controls tiling/COPC writer threading. It does not control subsampling parallelism. + +#### `--num-spatial-chunks` (Subsampling Parallelism) + +Controls per-file or per-product spatial chunking: + +- COPC `center-of-mass` uses `--num-spatial-chunks` voxel-aligned COPC window workers +- Other paths split each tile into `--num-spatial-chunks` spatial chunks along the X-axis +- `create_merged_file` uses `--num-spatial-chunks` for bounded COPC product reads before final COPC/LAZ/PLY writing +- Remap uses `--num-spatial-chunks` as the number of native COPC spatial-query windows when original files are COPC +- Subsampling chunks/windows are processed in parallel using `ProcessPoolExecutor` +- Product chunks are currently processed sequentially to keep memory and disk pressure predictable + +``` +Example with --num-spatial-chunks=5: + tile.laz → [chunk/window 0..4] → parallel subsample → merge + original_with_predictions/*.copc.laz → [bounds 0..4] → product chunks → prod_merged_*.copc.laz +``` + +**Memory and disk impact**: Higher values increase subsampling worker concurrency and reduce per-product read windows. Tune down if memory or storage I/O becomes the bottleneck; tune up when bounded product chunks are still too large. + +If not specified, this defaults to `--workers`. + + + +--- + +## Input/Output Formats + +### Input Requirements + +#### Tile Task +- **File formats**: LAZ (compressed) or LAS (uncompressed) +- **Coordinate system**: Should be in a projected CRS (e.g., UTM) +- **Directory structure**: Flat directory with LAZ/LAS files + +#### Merge Task +- **Required attribute**: `PredInstance` (integer instance IDs) +- **Optional attributes**: `PredSemantic`, `species_id` +- **File naming**: `c{col}_r{row}*.laz` pattern + +### Output Structure + +``` +output_dir/ +├── tiles_100m/ # Tiled point clouds (100m default) +│ ├── c00_r00.copc.laz # COPC tiles (Phase 2 output) +│ ├── c00_r01.copc.laz +│ └── c01_r00.copc.laz +│ +├── subsampled_res1/ # Resolution 1 subsamples (1cm COPC LAZ by default) +│ ├── output_100m_c00_r00_subsampled_1cm.copc.laz +│ └── ... +│ +├── subsampled_res2/ # Resolution 2 subsamples (10cm regular LAZ by default) +│ ├── output_100m_c00_r00_subsampled_10cm.laz +│ └── ... +│ +├── segmented_remapped/ # Remapped predictions (merge task) +│ ├── c00_r00_segmented_remapped.laz +│ └── ... +│ +├── output_tiles/ # Final per-tile outputs with merged IDs +│ ├── c00_r00.copc.laz +│ └── ... +│ +├── original_with_predictions/ # Original files with PredInstance (merge task) +│ ├── input_file_1.laz +│ └── input_file_2.laz +│ +├── products/ +│ ├── original_with_predictions_copc/ +│ │ ├── input_file_1.copc.laz +│ │ └── input_file_2.copc.laz +│ ├── prod_merged_1cm.copc.laz # Default prod-merged product +│ └── prod_merged_10cm.copc.laz +│ +├── tindex_100m.gpkg # Spatial index +├── tile_bounds_tindex.json # Tile metadata +├── tile_jobs_100m.txt # Processing jobs +├── overview_copc_tiles.png # Visualization +│ +└── logs/ # Processing logs +``` + +### Point Cloud Attributes + +#### Tile Task Output +- `X`, `Y`, `Z`: 3D coordinates +- Intermediate COPC conversion strips extra dimensions by default. +- `create_merged_file` preserves enriched dimensions for prod-merged outputs. + +#### Merge Task Output +- `X`, `Y`, `Z`: 3D coordinates +- `PredInstance`: Global tree instance ID (consistent across tiles) +- `PredSemantic`: Semantic class (if present in input) +- `species_id`: Tree species ID (if present in input) + +--- + +## Advanced Configuration + +### Large Dataset Processing + +For datasets exceeding 100GB with larger tiles: + +```bash +python src/run.py --task tile \ + --input-dir /data/input \ --output-dir /data/output \ - --instance-dimension PredInstance_SAT + --tile-length 500 \ + --tile-buffer 30 \ + --workers 32 \ + --threads 10 \ + --num-spatial-chunks 10 +``` -# Filter + remap in one step -python src/run.py --task filter \ - --segmented-folders /data/segmented \ +### High-Precision Processing + +For research applications requiring maximum fidelity (already default): + +```bash +python src/run.py --task tile \ + --input-dir /data/input \ --output-dir /data/output \ - --instance-dimension PredInstance_SAT \ - --remap-merge True \ - --original-input-dir /data/originals \ - --subsampled-target-folder /data/subsampled \ - --transfer-original-dims-to-merged True - -# Standalone remap (multiple segmentation sources) -# Source collections and remap targets may be LAZ/LAS/COPC. -python src/run.py --task remap \ - --segmented-folders /data/sat_tiles,/data/rct_tiles \ - --original-input-dir /data/originals \ + --tile-length 300 \ + --tile-buffer 20 \ + --resolution-1 0.01 \ + --resolution-2 0.05 +``` + +### Memory-Constrained Systems + +For systems with limited RAM (smaller tiles, lower resolution): + +```bash +python src/run.py --task tile \ + --input-dir /data/input \ --output-dir /data/output \ - --produce-merged-file True \ - --transfer-original-dims-to-merged True + --tile-length 100 \ + --tile-buffer 10 \ + --workers 2 \ + --threads 2 \ + --num-spatial-chunks 2 \ + --resolution-1 0.02 \ + --resolution-2 0.15 ``` -## Docker +### Single Small File Processing + +For processing single files without tiling: ```bash -docker build -t 3dtrees-smart-tile . +python src/run.py --task tile \ + --input-dir /data/input \ + --output-dir /data/output \ + --tiling-threshold 1000 # Skip tiling if single file < 1000 MB +``` -docker run -v /path/to/data:/data 3dtrees-smart-tile \ - --task filter --segmented-folders /data/segmented --output-dir /data/output +--- + +## Docker & Automation + +The pipeline includes Docker support for containerized execution and automated workflows with resource monitoring. + +### Docker Setup + +Build the Docker image: + +```bash +docker build -t 3dtrees_smart_tile . ``` -## Key behaviors +Run the pipeline in Docker: -**Tile task** -- Converts all inputs to COPC via untwine (preserves all dimensions). -- Creates overlapping tiles on a `cXX_rYY` grid with configurable buffer. -- Writes two subsampled collections (default 1 cm and 10 cm); both res1 and res2 can be emitted as COPC. -- Outputs `tile_bounds_tindex.json` for downstream neighbor resolution. +```bash +./run_docker.sh # Tile task: input → tiled + subsampled (1cm, 10cm) +./run_docker_merge.sh # Merge task: segmented 10cm + 1cm + tile_bounds JSON → merged.laz +./run_docker_remap.sh # Remap task: merged.laz + original files → originals with all merged dimensions +``` -**Filter task** -- `tile_bounds_json` is **optional**. When omitted, neighbors and border zone width are derived from `cXX_rYY` filename coordinates and spatial overlap of header bounds. -- Accepts segmented inputs as `.las`, `.laz`, or `.copc.laz`. -- Removes instances whose anchor (centroid by default) falls in the overlap zone of a neighbor tile. -- Omits filtered tile files that contain only buffer-region points and records that state in `filtered_tile_manifest.json` and, when available, an annotated output copy of `tile_bounds_tindex.json`. -- Small-cluster reassignment merges tiny fragments into nearby large instances. -- With `--remap-merge`, the filter tail runs a full remap onto originals and/or subsampled targets. +Edit the path variables at the top of each script to match your data. Merge requires `tile_bounds_tindex.json` from the Tile task output. -**Remap task** -- Supports multiple segmentation sources (comma-separated `--segmented-folders`). -- Accepts COPC collections and COPC remap targets in addition to plain LAZ/LAS. -- Transfers prediction dims to original files via KDTree nearest-neighbor matching. -- Enrichment step adds original-file attributes (intensity, RGB, etc.) to the merged output. -- RGB promotion: when `red`, `green`, `blue` are present, outputs use LAS point format 7 (standard RGB fields) instead of format 6. +### Automated Pipeline -## Output structure +For fully automated execution with resource monitoring, see [AUTOMATION_README.md](AUTOMATION_README.md). +Quick start: + +```bash +./run_automated_pipeline.sh 1 ``` -output_dir/ - # tile task - original_copc/ COPC-normalized sources - subsampled_res1/ 1 cm subsampled tiles - subsampled_res2/ 10 cm subsampled tiles - tile_bounds_tindex.json tile layout metadata - overview_copc_tiles.png tile grid preview - - # filter task - filtered_tiles/ filtered LAZ tiles - filtered_tiles_copc/ filtered COPC tiles (when prepared / reused) - filtered_trees/ filtered tree TXT sidecars - filtered_tile_manifest.json tile creation/omission state - tile_bounds_tindex.json annotated tile layout with per-tile filter_output status - - # remap / filter+remap - original_with_predictions/ per-original-file outputs - subsampled_with_predictions/ per-subsampled-file outputs - merged_with_all_dims.laz merged output (prediction dims only) - merged_with_originals.laz merged output enriched with original attributes - tiles_with_original_dimensions/ enriched per-tile files -``` - -## Parameters - -Run `python src/run.py --show-params` for the full list. Key parameters: - -| Parameter | Default | Notes | -|-----------|---------|-------| -| `--task` | — | `tile`, `filter`, or `remap` | -| `--tile-length` | 300 | tile edge in meters | -| `--tile-buffer` | 20 | overlap buffer in meters | -| `--resolution-1` / `--resolution-2` | 0.01 / 0.1 | subsampling resolutions | -| `--output-copc-res1` / `--output-copc-res2` | True / False | emit res1/res2 outputs as `.copc.laz` instead of `.laz` | -| `--tile-bounds-json` | optional | tile layout JSON; filter auto-detects from filenames when absent | -| `--instance-dimension` | `PredInstance` | name of the instance-ID dimension | -| `--border-zone-width` | auto | derived from JSON, spatial overlap, or explicit override | -| `--segmented-folders` | — | comma-separated paths to segmented tile folders | -| `--original-input-dir` | — | original pre-tiling files for remap | -| `--subsampled-target-folder` | — | subsampled files to remap onto | -| `--produce-merged-file` | True | write a single merged LAZ | -| `--transfer-original-dims-to-merged` | True | enrich merged file with original attributes | -| `--standardization-json` | — | limits which original dims are transferred | -| `--chunk-size` | 20M | streaming chunk size (points) | -| `--workers` | 4 | parallel workers | + +This automated workflow: +- Downloads data from S3 +- Runs tile task (COPC conversion, tiling, subsampling) +- Adds dummy PredInstance dimension (for testing) +- Runs remap_merge task with auto-detection of single vs multi-file workflows +- Tracks CPU and RAM usage throughout the process +- Generates resource usage logs + +### Additional Documentation + +- [AUTOMATION_README.md](AUTOMATION_README.md) - Detailed automation and Docker workflow guide +- [CLAUDE.md](CLAUDE.md) - Quick reference for AI assistants and developers +- [Dockerfile](Dockerfile) - Container configuration + +--- + +## Troubleshooting + +### Common Issues + +#### "No LAZ/LAS files found" +- Ensure input files have `.laz` or `.las` extension (lowercase) +- Check that `input_dir` points to the correct directory +- Verify file permissions + +#### "pdal: command not found" +- Install PDAL: `conda install -c conda-forge pdal` +- Verify installation: `pdal --version` +- Check PATH environment variable + +#### "untwine: command not found" +- The pipeline automatically falls back to PDAL's `writers.copc` if untwine is not installed or if the preferred untwine path fails validation, so this is not an error — just slower COPC conversion. +- To install for better performance: `conda install -c conda-forge untwine` +- Verify: `untwine --help` + +#### "Memory allocation failed" +- Reduce `--tile-length` for smaller tiles +- Decrease `--workers` to limit concurrent memory usage +- Decrease `--num-spatial-chunks` to limit per-file subsampling workers +- Use `--resolution-1` and `--resolution-2` with larger values + +#### "No space left on device" during `create_merged_file` +- Use an output directory on local scratch storage, not network storage +- Increase free scratch space; direct untwine COPC finalization can temporarily need many times the final COPC size +- Reduce simultaneous jobs writing to the same disk +- Keep `--num-spatial-chunks` enabled so SmartTile avoids one giant temporary merged LAZ + +#### "CRS mismatch" or "Coordinates appear projected" +- Ensure all input files are in the same coordinate reference system +- Use projected CRS (e.g., UTM) not geographic (WGS84) + +#### "No PredInstance attribute found" +- Verify segmentation output includes `PredInstance` dimension +- Check attribute names (case-sensitive): `PredInstance`, not `predinstance` + +### Debugging + +Enable verbose output: + +```bash +python src/run.py --task merge --verbose ... +``` + +Check processing logs in `output_dir/logs/`: + +```bash +ls -la output_dir/logs/ +cat output_dir/logs/c00_r00_convert.log +``` + +### Performance Tuning + +#### Optimize for SSD +```bash +# Use more workers when I/O is fast +python src/run.py --task tile --workers 16 --threads 10 --num-spatial-chunks 10 ... +``` + +#### Optimize for HDD +```bash +# Reduce parallel I/O +python src/run.py --task tile --workers 2 --threads 2 --num-spatial-chunks 2 ... +``` + +#### Monitor resource usage +```bash +# Run with htop in another terminal +htop -p $(pgrep -f "python src/run.py") +``` + +--- + +## Project Structure + +``` +3dtrees_smart_tile/ +├── src/ # Python source code +│ ├── run.py # Main CLI orchestrator +│ ├── parameters.py # Parameter configuration (Pydantic) +│ ├── main_tile.py # Tiling pipeline +│ ├── main_subsample.py # Subsampling pipeline +│ ├── main_remap.py # Prediction remapping +│ ├── main_create_merged_file.py # Prod-merged product creation +│ ├── main_merge.py # Merge wrapper +│ ├── merge_tiles.py # Merge compatibility/core entry points +│ ├── merge_*.py # Merge internals +│ ├── tile_*.py # Tiling internals +│ ├── subsample_*.py # Subsampling internals +│ ├── copc_*.py # COPC metadata and staging helpers +│ ├── filter_buffer_instances.py # Buffer zone filtering +│ ├── prepare_tile_jobs.py # Tile job generation +│ ├── get_bounds_from_tindex.py # Extent calculation +│ └── plot_tiles_and_copc.py # Visualization +│ +├── README.md # This documentation +├── CONTEXT.md # Agent/developer context and invariants +├── AUTOMATION_README.md # Automation and Docker guide +├── CLAUDE.md # Quick reference for developers +├── Dockerfile # Container configuration +├── run_automated_pipeline.sh # Automated workflow orchestrator +├── run_docker.sh # Docker: Tile task +├── run_docker_merge.sh # Docker: Merge task (requires tile_bounds JSON) +├── run_docker_remap.sh # Docker: Remap task (merged file → original files) +└── .gitignore # Git ignore rules +``` + +### Module Descriptions + +| Module | Purpose | +|--------|---------| +| `run.py` | CLI entry point, task routing, parameter handling | +| `parameters.py` | Pydantic-based parameter definitions with CLI support | +| `main_tile.py` | Two-phase tiling (distribute + COPC conversion), tindex creation | +| `main_subsample.py` | Parallel voxel-based subsampling | +| `main_remap.py` | KDTree-based prediction remapping | +| `main_create_merged_file.py` | Prod-merged product creation from Original-with-predictions files | +| `main_merge.py` | Merge task orchestration | +| `merge_tiles.py`, `merge_*.py` | Merge orchestration and internals | +| `tile_*.py`, `subsample_*.py` | Extracted tiling and subsampling helpers | +| `copc_*.py`, `point_cloud_*.py` | COPC staging, metadata preservation, and output helpers | +| `filter_buffer_instances.py` | Centroid-based buffer zone filtering | +| `prepare_tile_jobs.py` | Tile grid calculation and job list generation | +| `get_bounds_from_tindex.py` | Extent extraction from spatial index | +| `plot_tiles_and_copc.py` | Matplotlib visualization of tiles | + +--- ## Dependencies -Python 3.10+, laspy, lazrs, numpy, scipy, pydantic, pydantic-settings, matplotlib, fiona, pyproj, PDAL, untwine. +### Core Dependencies + +| Package | Version | Purpose | +|---------|---------|---------| +| Python | ≥3.10 | Runtime | +| PDAL | ≥2.5 | Point cloud processing, subsampling | +| untwine | Latest | Fast COPC conversion (auto-fallback to PDAL if unavailable) | +| laspy | Latest | LAZ/LAS file I/O | +| lazrs-python | Latest | LAZ compression | +| NumPy | Latest | Array operations | +| SciPy | Latest | KDTree spatial queries | +| pydantic | ≥2.0 | Parameter validation | +| pydantic-settings | Latest | CLI and env var support | + +### Optional Dependencies + +| Package | Purpose | +|---------|---------| +| matplotlib | Visualization | +| fiona | Vector file handling | +| pyproj | CRS transformations | +| geopandas | Geospatial operations | + +### External Tools + +| Tool | Purpose | +|------|---------| +| [untwine](https://github.com/hobuinc/untwine) | COPC conversion (preferred, auto-fallback to PDAL) | + +--- ## License -MIT +MIT License + +Copyright (c) 2026 + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +--- + +## Contributing + +Contributions are welcome! Please open an issue or pull request on the project repository. + +## Citation + +If you use this pipeline in your research, please cite: + +```bibtex +@software{3dtrees_smart_tile, + title = {3DTrees Smart Tiling Pipeline}, + author = {}, + year = {2026}, + url = {https://github.com/your-org/3dtrees_smart_tile} +} +``` + +--- + +**Questions or issues?** Open an issue on GitHub or contact the maintainers. diff --git a/src/copc_metadata.py b/src/copc_metadata.py new file mode 100644 index 0000000..b8a0653 --- /dev/null +++ b/src/copc_metadata.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +"""COPC CRS and GeoTIFF metadata preservation helpers. + +These helpers are shared by tiling, subsampling conversion, and prod-merged +creation. They keep CRS/GeoTIFF handling out of task orchestration modules and +make the metadata contract explicit in one place. +""" + +from __future__ import annotations + +import os +import struct +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + + +PROJECTION_VLR_USER_ID = "LASF_Projection" +PROJECTION_VLR_RECORD_IDS = {2111, 2112, 2113, 34735, 34736, 34737} +GEOTIFF_PROJECTION_RECORD_IDS = {34735, 34736, 34737} +LAS14_START_OF_FIRST_EVLR_OFFSET = 235 +LAS14_NUMBER_OF_EVLRS_OFFSET = 243 + + +def laspy_laz_backend(): + """Return the best available laspy LAZ backend.""" + try: + import laspy + + if hasattr(laspy.LazBackend, "LazrsParallel"): + return laspy.LazBackend.LazrsParallel + if hasattr(laspy.LazBackend, "Lazrs"): + return laspy.LazBackend.Lazrs + except Exception: + pass + return None + + +def projection_vlr_keys(header) -> set[tuple[str, int]]: + """Return CRS/projection VLR keys that should survive format conversion.""" + keys = set() + for vlr in getattr(header, "vlrs", []): + user_id = getattr(vlr, "user_id", "") + record_id = getattr(vlr, "record_id", None) + if user_id == PROJECTION_VLR_USER_ID or record_id in PROJECTION_VLR_RECORD_IDS: + keys.add((user_id, record_id)) + return keys + + +def vlr_record_bytes(vlr) -> bytes: + """Return raw record bytes for regular laspy VLR objects.""" + if hasattr(vlr, "record_data_bytes"): + try: + return bytes(vlr.record_data_bytes()) + except Exception: + pass + record_data = getattr(vlr, "record_data", None) + if record_data is None: + return b"" + try: + return bytes(record_data) + except Exception: + return str(record_data).encode("utf-8", errors="replace") + + +def projection_records(header, record_ids: set[int]) -> Dict[tuple[str, int], bytes]: + """Return projection record payloads from header VLRs/EVLRs.""" + records = {} + collections = [getattr(header, "vlrs", [])] + evlrs = getattr(header, "evlrs", None) + if evlrs: + collections.append(evlrs) + + for collection in collections: + for vlr in collection: + user_id = getattr(vlr, "user_id", "") + record_id = getattr(vlr, "record_id", None) + if user_id == PROJECTION_VLR_USER_ID and record_id in record_ids: + records[(user_id, record_id)] = vlr_record_bytes(vlr) + return records + + +def geotiff_projection_records(header) -> Dict[tuple[str, int], bytes]: + """Return GeoTIFF CRS projection records, including GeoKeyDirectoryVlr.""" + return projection_records(header, GEOTIFF_PROJECTION_RECORD_IDS) + + +def projection_vlr_fingerprints(header) -> Dict[tuple[str, int], bytes]: + """Return CRS/projection VLR record bytes keyed by LAS VLR identity.""" + return projection_records(header, PROJECTION_VLR_RECORD_IDS) + + +def parse_crs(header) -> Optional[Any]: + try: + return header.parse_crs() + except Exception: + return None + + +def crs_text(header) -> Optional[str]: + """Best-effort parseable CRS text; returns None when CRS metadata is unavailable.""" + crs = parse_crs(header) + if crs is None: + return None + try: + return crs.to_wkt() + except Exception: + return str(crs) + + +def crs_authority_from_crs(crs) -> Optional[str]: + if crs is None: + return None + + try: + authority = crs.to_authority() + except Exception: + authority = None + if authority and authority[0] and authority[1]: + return f"{authority[0].upper()}:{authority[1]}" + + try: + epsg = crs.to_epsg() + except Exception: + epsg = None + if epsg: + return f"EPSG:{epsg}" + return None + + +def crs_authority_string(header) -> Optional[str]: + """Return a compact CRS authority string such as EPSG:32632 when available.""" + return crs_authority_from_crs(parse_crs(header)) + + +def crs_equivalent(source_crs, output_crs) -> bool: + """Return True when two parsed CRS objects describe the same CRS.""" + if source_crs is None or output_crs is None: + return False + + source_authority = crs_authority_from_crs(source_crs) + output_authority = crs_authority_from_crs(output_crs) + if source_authority and source_authority == output_authority: + return True + + try: + if source_crs.equals(output_crs, ignore_axis_order=True): + return True + except Exception: + pass + + try: + return source_crs.to_wkt() == output_crs.to_wkt() + except Exception: + return str(source_crs) == str(output_crs) + + +def srs_assignment_from_file(path: Path) -> Optional[str]: + """Read an input header and return an Untwine --a_srs value when possible.""" + import laspy + + try: + with laspy.open(str(path), laz_backend=laspy_laz_backend()) as src: + return crs_authority_string(src.header) + except Exception: + return None + + +def first_srs_assignment(paths: List[Path]) -> Optional[str]: + crs_source = first_crs_source(paths) + if crs_source is None: + return None + return srs_assignment_from_file(crs_source) + + +def evlr_record_bytes(user_id: str, record_id: int, description: str, data: bytes) -> bytes: + user = user_id.encode("ascii", errors="replace")[:16].ljust(16, b"\0") + desc = description.encode("ascii", errors="replace")[:32].ljust(32, b"\0") + return struct.pack(" List[Tuple[str, int, str, bytes]]: + """Return source projection VLR payloads for preservation. + + The function name is kept for compatibility with existing callers. It now + preserves all LASF_Projection CRS records, including WKT and GeoTIFF keys, + because COPC writers can normalize WKT while still carrying an equivalent + CRS. Appending the source records as EVLRs keeps the original metadata + available without rewriting COPC hierarchy bytes. + """ + import laspy + + records = [] + with laspy.open(str(source_file), laz_backend=laspy_laz_backend()) as src: + collections = [getattr(src.header, "vlrs", [])] + evlrs = getattr(src.header, "evlrs", None) + if evlrs: + collections.append(evlrs) + for collection in collections: + for vlr in collection: + user_id = getattr(vlr, "user_id", "") + record_id = getattr(vlr, "record_id", None) + if user_id == PROJECTION_VLR_USER_ID and record_id in PROJECTION_VLR_RECORD_IDS: + records.append( + ( + user_id, + int(record_id), + getattr(vlr, "description", "") or "", + vlr_record_bytes(vlr), + ) + ) + return records + + +def append_source_geotiff_projection_evlrs(source_file: Path, copc_file: Path) -> Tuple[bool, str]: + """Append original projection VLRs as EVLRs without moving COPC chunks.""" + import laspy + + try: + source_records = source_geotiff_projection_vlrs(source_file) + if not source_records: + return (True, "source has no projection VLRs") + + with laspy.open(str(copc_file), laz_backend=laspy_laz_backend()) as out: + output_header = out.header + output_records = projection_vlr_fingerprints(output_header) + existing_count = int(getattr(output_header, "number_of_evlrs", 0) or 0) + existing_start = int(getattr(output_header, "start_of_first_evlr", 0) or 0) + if str(output_header.version) != "1.4": + return (False, "GeoTIFF EVLR preservation requires LAS 1.4 output") + + missing = [ + (user_id, record_id, description, data) + for user_id, record_id, description, data in source_records + if output_records.get((user_id, record_id)) != data + ] + if not missing: + return (True, "projection VLRs already preserved") + + with open(copc_file, "r+b") as f: + f.seek(0, os.SEEK_END) + append_start = f.tell() + for user_id, record_id, description, data in missing: + f.write(evlr_record_bytes(user_id, record_id, description, data)) + f.seek(LAS14_START_OF_FIRST_EVLR_OFFSET) + f.write(struct.pack(" bool: + return bool(projection_vlr_keys(header) or crs_text(header)) + + +def first_crs_source(paths: List[Path]) -> Optional[Path]: + """Return the first source file with CRS metadata, else the first source file.""" + if not paths: + return None + import laspy + + fallback = paths[0] + for path in paths: + try: + with laspy.open(str(path), laz_backend=laspy_laz_backend()) as src: + if crs_metadata_present(src.header): + return path + except Exception: + continue + return fallback + + +def copc_preserves_source_crs(source_file: Path, copc_file: Path) -> Tuple[bool, str]: + """Validate that a COPC output still carries the source CRS/projection metadata.""" + import laspy + + try: + with laspy.open(str(source_file), laz_backend=laspy_laz_backend()) as src: + source_header = src.header + source_crs = parse_crs(source_header) + source_projection = projection_vlr_fingerprints(source_header) + source_geotiff_projection = geotiff_projection_records(source_header) + + if not source_crs and not source_projection: + return (True, "source has no CRS metadata") + + with laspy.open(str(copc_file), laz_backend=laspy_laz_backend()) as out: + output_header = out.header + output_crs = parse_crs(output_header) + output_projection = projection_vlr_fingerprints(output_header) + output_geotiff_projection = geotiff_projection_records(output_header) + except Exception as e: + return (False, f"could not validate CRS metadata: {e}") + + if source_geotiff_projection and not all( + output_geotiff_projection.get(key) == value + for key, value in source_geotiff_projection.items() + ): + return ( + False, + "source GeoTIFF projection VLRs missing or changed in COPC output", + ) + changed_projection_keys = [ + key + for key, value in source_projection.items() + if key in output_projection and output_projection[key] != value + ] + if changed_projection_keys: + return ( + False, + "source CRS/projection metadata missing or changed in COPC output", + ) + if crs_equivalent(source_crs, output_crs): + return (True, "CRS metadata preserved") + if source_projection and all( + output_projection.get(key) == value for key, value in source_projection.items() + ): + return (True, "projection VLRs preserved") + if source_crs or source_projection: + return ( + False, + "source CRS/projection metadata missing or changed in COPC output", + ) + return (True, "source has no CRS metadata") diff --git a/src/copc_staging.py b/src/copc_staging.py new file mode 100644 index 0000000..586d729 --- /dev/null +++ b/src/copc_staging.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +"""COPC staging-cache freshness checks for prod-merged creation.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Dict, Optional + +import numpy as np + +from copc_metadata import laspy_laz_backend +from point_cloud_metadata import point_cloud_source_key + + +MANIFEST_FILENAME = ".smarttile_copc_manifest.json" + + +def _extra_dimension_signature(dim) -> Dict[str, object]: + """Return the structural extra-dimension schema used for cache freshness. + + Descriptions are intentionally excluded. Some LAZ->COPC conversions drop or + normalize descriptive text while preserving the actual point record schema. + """ + return { + "name": dim.name, + "dtype": str(np.dtype(dim.dtype)), + } + + +def point_cloud_header_signature(path: Path) -> Dict[str, object]: + """Return cheap header metadata that should match across LAZ->COPC staging.""" + import laspy + + with laspy.open(str(path), laz_backend=laspy_laz_backend()) as reader: + header = reader.header + return { + "point_count": int(header.point_count), + "bounds": [ + float(header.x_min), + float(header.x_max), + float(header.y_min), + float(header.y_max), + float(header.z_min), + float(header.z_max), + ], + "scales": [float(value) for value in header.scales], + "offsets": [float(value) for value in header.offsets], + "point_format": int(header.point_format.id), + "version": str(header.version), + "extra_dimensions": [ + _extra_dimension_signature(dim) + for dim in header.point_format.extra_dimensions + ], + } + + +def source_file_fingerprint(source_file: Path) -> Dict[str, object]: + """Return the source identity used to decide whether a staged COPC is fresh.""" + stat = source_file.stat() + return { + "name": source_file.name, + "size": int(stat.st_size), + "mtime_ns": int(stat.st_mtime_ns), + "header": point_cloud_header_signature(source_file), + } + + +def _manifest_path(copc_dir: Path) -> Path: + return copc_dir / MANIFEST_FILENAME + + +def read_copc_stage_manifest(copc_dir: Path) -> Dict[str, object]: + manifest_file = _manifest_path(copc_dir) + if not manifest_file.exists(): + return {"version": 1, "sources": {}} + try: + with manifest_file.open() as handle: + manifest = json.load(handle) + except Exception: + return {"version": 1, "sources": {}} + if not isinstance(manifest, dict): + return {"version": 1, "sources": {}} + manifest.setdefault("version", 1) + manifest.setdefault("sources", {}) + return manifest + + +def write_copc_stage_manifest_entry(source_file: Path, copc_file: Path) -> None: + """Record which source file produced a staged COPC.""" + copc_file.parent.mkdir(parents=True, exist_ok=True) + manifest = read_copc_stage_manifest(copc_file.parent) + sources = manifest.setdefault("sources", {}) + key = point_cloud_source_key(source_file) + sources[key] = { + "source": source_file_fingerprint(source_file), + "copc_file": copc_file.name, + "copc_header": point_cloud_header_signature(copc_file), + } + with _manifest_path(copc_file.parent).open("w") as handle: + json.dump(manifest, handle, indent=2, sort_keys=True) + + +def staged_copc_matches_source(copc_file: Path, source_file: Optional[Path]) -> bool: + """Return True when a staged COPC is compatible with source_file. + + A SmartTile manifest is preferred when available. Older validation runs may + have reusable COPCs without a manifest, so fall back to comparing structural + header metadata. + """ + if source_file is None: + return True + + manifest = read_copc_stage_manifest(copc_file.parent) + entry = manifest.get("sources", {}).get(point_cloud_source_key(source_file)) + try: + source_header = point_cloud_header_signature(source_file) + copc_header = point_cloud_header_signature(copc_file) + if not entry: + return source_header == copc_header + if entry.get("copc_file") != copc_file.name: + return False + return ( + entry.get("source") == source_file_fingerprint(source_file) + and entry.get("copc_header") == copc_header + ) + except Exception: + return False diff --git a/src/dimension_transfer.py b/src/dimension_transfer.py new file mode 100644 index 0000000..3a194c3 --- /dev/null +++ b/src/dimension_transfer.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +"""Dimension collision and transfer policy for SmartTile point-cloud products.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Callable, Dict, Optional, Set, Tuple + +import numpy as np + + +CORE_COORDINATE_DIMS = {"X", "Y", "Z"} + + +@dataclass(frozen=True) +class ResolvedDimensionTransferPlan: + """Fully resolved dimension transfer plan with output dtypes.""" + + add_new: Dict[str, np.dtype] + overwrite: Dict[str, np.dtype] + renamed: Dict[str, str] + output_dtypes: Dict[str, np.dtype] + output_to_source: Dict[str, str] + + @property + def has_transfers(self) -> bool: + return bool(self.output_dtypes) + + +def next_available_suffix(base: str, used: Set[str]) -> str: + """Return base_1, base_2, ... first not in used.""" + for i in range(1, 10000): + candidate = f"{base}_{i}" + if candidate not in used: + return candidate + return f"{base}_9999" + + +def suffixes_for_collision(base: str, used: Set[str]) -> Tuple[str, str]: + """Return collision-safe source/target suffixes for one base dimension name.""" + first = f"{base}_1" + second = f"{base}_2" + out_first = first if first not in used else next_available_suffix(base, used) + used.add(out_first) + out_second = second if second not in used else next_available_suffix(base, used) + used.add(out_second) + return out_first, out_second + + +def _target_dimension_is_empty_or_constant(values) -> bool: + arr = np.asarray(values) + if arr.size == 0: + return True + return np.min(arr) == np.max(arr) or np.count_nonzero(arr) == 0 + + +def plan_dimension_transfer( + source_dims: Dict[str, np.dtype], + target_dim_names: Set[str], + get_target_array: Callable[[str], object], + skip: Optional[Set[str]] = None, +) -> ResolvedDimensionTransferPlan: + """Plan how source dimensions should be transferred into a target cloud. + + Policy: + - skip core coordinate dimensions by default + - add source-only dimensions under their original name + - overwrite target dimensions only when the target array is empty/constant + - preserve non-empty target/source collisions by adding the source as name_1 + """ + skipped = skip or CORE_COORDINATE_DIMS + source_names = set(source_dims.keys()) - skipped + target_names = set(target_dim_names) - skipped + + add_new = {name: source_dims[name] for name in source_names - target_names} + overwrite: Dict[str, np.dtype] = {} + for name in target_names & source_names: + arr = get_target_array(name) + if arr is None: + continue + if _target_dimension_is_empty_or_constant(arr): + overwrite[name] = source_dims[name] + + used_names = set(target_dim_names) | set(add_new.keys()) | set(overwrite.keys()) + collision = (source_names & target_names) - set(overwrite.keys()) + renamed: Dict[str, str] = {} + for name in sorted(collision): + candidate = f"{name}_1" + output_name = candidate if candidate not in used_names else next_available_suffix(name, used_names) + renamed[name] = output_name + used_names.add(output_name) + + output_dtypes = {**add_new, **overwrite} + output_dtypes.update({out_name: source_dims[source_name] for source_name, out_name in renamed.items()}) + output_to_source = {name: name for name in add_new} + output_to_source.update({name: name for name in overwrite}) + output_to_source.update({out_name: source_name for source_name, out_name in renamed.items()}) + + return ResolvedDimensionTransferPlan( + add_new=add_new, + overwrite=overwrite, + renamed=renamed, + output_dtypes=output_dtypes, + output_to_source=output_to_source, + ) diff --git a/src/filter_buffer_instances.py b/src/filter_buffer_instances.py index fb630a9..930e4a3 100644 --- a/src/filter_buffer_instances.py +++ b/src/filter_buffer_instances.py @@ -15,28 +15,35 @@ import argparse import numpy as np import laspy +from laspy.vlrs.vlrlist import VLRList from pathlib import Path from typing import Dict, List, Tuple, Set, Optional +from instance_labels import validate_prediction_instance_labels + def get_tile_neighbors(tile_name: str, all_tile_names: List[str]) -> Dict[str, bool]: """ Determine which edges of a tile have neighbors. Returns dict with 'east', 'west', 'north', 'south' boolean values. """ - # Parse tile name format: c{col}_r{row} - parts = tile_name.split('_') - col_str = parts[0][1:] # Extract number after 'c' - row_str = parts[1][1:] # Extract number after 'r' - col = int(col_str) - row = int(row_str) - + # Parse tile name format: c{col}_r{row}. Non-grid filenames can appear + # when the filter task is used on a single tile; treat them as edge tiles. + try: + parts = tile_name.split('_') + col_str = parts[0][1:] # Extract number after 'c' + row_str = parts[1][1:] # Extract number after 'r' + col = int(col_str) + row = int(row_str) + except (IndexError, ValueError): + return {'east': False, 'west': False, 'north': False, 'south': False} + col_padding = len(col_str) row_padding = len(row_str) - + def format_tile_name(c, r): return f"c{str(c).zfill(col_padding)}_r{str(r).zfill(row_padding)}" - + return { 'east': format_tile_name(col+1, row) in all_tile_names, 'west': col > 0 and format_tile_name(col-1, row) in all_tile_names, @@ -45,6 +52,60 @@ def format_tile_name(c, r): } +def _tile_base_name(path: Path) -> str: + """Return the tile grid id from a LAZ/LAS filename.""" + name = path.stem + if name.endswith(".copc"): + name = name[:-5] + for suffix in ['_segmented_remapped', '_segmented', '_remapped', '_filtered']: + name = name.replace(suffix, '') + return name + + +def _input_point_clouds(input_dir: Path) -> List[Path]: + """Return all filterable point-cloud files in stable order.""" + files = [] + seen = set() + for pattern in ("*.laz", "*.las", "*.LAZ", "*.LAS"): + for path in sorted(input_dir.glob(pattern)): + key = path.resolve() + if key in seen: + continue + seen.add(key) + files.append(path) + return sorted(files, key=lambda p: p.name.lower()) + + +def _copy_filtered_output_header(source_header: laspy.LasHeader) -> laspy.LasHeader: + """Copy source metadata for a filtered LAZ output.""" + header = source_header.copy() + header.point_count = 0 + + def is_stale_copc_vlr(vlr) -> bool: + return getattr(vlr, "user_id", "") == "copc" and getattr(vlr, "record_id", None) in (1, 2) + + header.vlrs = VLRList([vlr for vlr in header.vlrs if not is_stale_copc_vlr(vlr)]) + if getattr(header, "evlrs", None) is not None: + header.evlrs = VLRList([vlr for vlr in header.evlrs if not is_stale_copc_vlr(vlr)]) + return header + + +def _write_las_points( + source_las: laspy.LasData, + output_file: Path, + keep_mask: Optional[np.ndarray] = None, +) -> None: + """Write selected source points with stale COPC VLRs stripped.""" + output_file.parent.mkdir(parents=True, exist_ok=True) + header = _copy_filtered_output_header(source_las.header) + output_las = laspy.LasData(header) + if keep_mask is None: + output_las.points = source_las.points.copy() + else: + output_las.points = source_las.points[keep_mask].copy() + output_las.write(str(output_file), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) + + def compute_tile_bounds(points: np.ndarray) -> Tuple[float, float, float, float]: """Get the XY bounding box of a point cloud.""" return ( @@ -65,7 +126,7 @@ def get_instances_to_remove( ) -> Set[int]: """ Find instances whose centroid is in the buffer zone on inner edges. - + Args: points: Nx3 array of point coordinates instances: Array of instance IDs @@ -73,52 +134,52 @@ def get_instances_to_remove( tile_name: Name of the tile (e.g., "c00_r00") all_tile_names: List of all tile names to determine neighbors buffer: Buffer distance from inner edges - + Returns: Set of instance IDs to REMOVE (centroid in buffer zone) """ min_x, max_x, min_y, max_y = boundary - + # Determine which edges have neighbors neighbors = get_tile_neighbors(tile_name, all_tile_names) - + # Calculate tile dimensions and cap buffer tile_width = max_x - min_x tile_height = max_y - min_y min_dimension = min(tile_width, tile_height) actual_buffer = min(buffer, min_dimension * 0.4) actual_buffer = max(actual_buffer, 2.0) - + # Define buffer zone boundaries (only on inner edges) buf_min_x = min_x + (actual_buffer if neighbors['west'] else 0) buf_max_x = max_x - (actual_buffer if neighbors['east'] else 0) buf_min_y = min_y + (actual_buffer if neighbors['south'] else 0) buf_max_y = max_y - (actual_buffer if neighbors['north'] else 0) - + # Find instances to remove instances_to_remove = set() unique_ids = np.unique(instances) - + for inst_id in unique_ids: if inst_id <= 0: continue - + mask = instances == inst_id inst_points = points[mask] - + # Calculate centroid (XYZ) centroid = np.mean(inst_points, axis=0) cx, cy = centroid[0], centroid[1] - + # Check if centroid is in buffer zone in_west_buffer = neighbors['west'] and cx < buf_min_x in_east_buffer = neighbors['east'] and cx > buf_max_x in_south_buffer = neighbors['south'] and cy < buf_min_y in_north_buffer = neighbors['north'] and cy > buf_max_y - + if in_west_buffer or in_east_buffer or in_south_buffer or in_north_buffer: instances_to_remove.add(inst_id) - + return instances_to_remove @@ -131,191 +192,205 @@ def process_tile( ) -> Tuple[int, int, int]: """ Process a single tile: load, filter buffer instances, save. - + Returns: Tuple of (original_points, removed_points, removed_instances) """ print(f"Processing {input_file.name}...") - + try: las = laspy.read(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) except Exception as e: print(f" Error loading {input_file}: {e}") return 0, 0, 0 - + points = np.vstack(( np.array(las.x), np.array(las.y), np.array(las.z) )).T - + if hasattr(las, instance_dimension): instances = np.array(getattr(las, instance_dimension)) + validate_prediction_instance_labels(instances, instance_dimension, input_file) elif hasattr(las, 'treeID'): instances = np.array(las.treeID) + validate_prediction_instance_labels(instances, "treeID", input_file) else: print(f" Warning: No instance attribute ({instance_dimension}/treeID) found in {input_file}") - output_file.parent.mkdir(parents=True, exist_ok=True) - las.write(str(output_file), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) + _write_las_points(las, output_file) return len(points), 0, 0 - + original_point_count = len(points) - + # Extract tile name from filename - tile_name = input_file.stem - # Remove common suffixes - for suffix in ['_segmented_remapped', '_segmented', '_remapped', '_filtered']: - tile_name = tile_name.replace(suffix, '') - + tile_name = _tile_base_name(input_file) + # Compute tile boundary boundary = compute_tile_bounds(points) - + # Find instances to remove instances_to_remove = get_instances_to_remove( points, instances, boundary, tile_name, all_tile_names, buffer ) - + # Create boolean mask: True = keep point, False = remove point keep_mask = np.ones(len(points), dtype=bool) for inst_id in instances_to_remove: keep_mask[instances == inst_id] = False - - # Filter points - filtered_points = points[keep_mask] - removed_point_count = original_point_count - len(filtered_points) + + kept_point_count = int(np.count_nonzero(keep_mask)) + removed_point_count = original_point_count - kept_point_count removed_instance_count = len(instances_to_remove) - + if removed_instance_count == 0: print(f" {original_point_count:,} points, 0 instances removed") - # No filtering needed, just copy file - output_file.parent.mkdir(parents=True, exist_ok=True) - las.write(str(output_file), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) + _write_las_points(las, output_file) return original_point_count, 0, 0 - - # Create new LAS file with filtered points - output_file.parent.mkdir(parents=True, exist_ok=True) - - # Create new header (copy from original) - header = laspy.LasHeader(point_format=las.header.point_format.id, version=las.header.version) - header.offsets = las.header.offsets - header.scales = las.header.scales - - # Copy header metadata - for prop in ['system_identifier', 'generating_software', 'file_creation']: - if hasattr(las.header, prop): - setattr(header, prop, getattr(las.header, prop)) - - # Create new LAS data - output_las = laspy.LasData(header) - - # Copy all point dimensions (filtered) - for dim_name in las.point_format.dimension_names: - try: - dim_data = getattr(las, dim_name) - if hasattr(dim_data, '__len__') and len(dim_data) == len(points): - filtered_data = dim_data[keep_mask] - setattr(output_las, dim_name, filtered_data) - except Exception as e: - # Skip dimensions that can't be copied - pass - - # Write output - output_las.write(str(output_file), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) - - print(f" {original_point_count:,} → {len(filtered_points):,} points " + + _write_las_points(las, output_file, keep_mask=keep_mask) + + print(f" {original_point_count:,} -> {kept_point_count:,} points " f"({removed_point_count:,} removed, {removed_instance_count} instances)") - + return original_point_count, removed_point_count, removed_instance_count +def filter_buffer_instances_dir( + input_dir: Path, + output_dir: Path, + buffer: float = 10.0, + suffix: str = "_filtered", + instance_dimension: str = "PredInstance", + output_extension: Optional[str] = None, +) -> Dict[str, object]: + """Filter buffer instances for every LAZ/LAS file in a directory.""" + input_dir = Path(input_dir) + output_dir = Path(output_dir) + point_clouds = _input_point_clouds(input_dir) + if output_extension: + output_extension = output_extension if output_extension.startswith(".") else f".{output_extension}" + if output_extension.lower() == ".copc.laz": + raise ValueError("filter outputs are rewritten LAS/LAZ files; use .laz instead of .copc.laz") + + if len(point_clouds) == 0: + print(f"No LAZ/LAS files found in {input_dir}") + return { + "input_files": 0, + "output_files": [], + "total_original": 0, + "total_removed": 0, + "total_instances_removed": 0, + } + + print("=" * 60) + print("Buffer Filter Preprocessing") + print("=" * 60) + print(f"Input directory: {input_dir}") + print(f"Output directory: {output_dir}") + print(f"Buffer: {buffer}m") + print(f"Instance dimension: {instance_dimension}") + print(f"Found {len(point_clouds)} tiles to process") + print("=" * 60) + + all_tile_names = [_tile_base_name(path) for path in point_clouds] + total_original = 0 + total_removed = 0 + total_instances_removed = 0 + output_files = [] + + for input_file in point_clouds: + extension = output_extension or input_file.suffix + output_file = output_dir / f"{input_file.stem}{suffix}{extension}" + output_files.append(output_file) + + orig, removed, inst_removed = process_tile( + input_file, + output_file, + all_tile_names, + buffer, + instance_dimension=instance_dimension, + ) + + total_original += orig + total_removed += removed + total_instances_removed += inst_removed + + print("\n" + "=" * 60) + print("Summary") + print("=" * 60) + print(f"Total points: {total_original:,} -> {total_original - total_removed:,} " + f"({total_removed:,} removed, {100*total_removed/max(total_original,1):.1f}%)") + print(f"Total instances removed: {total_instances_removed}") + print("=" * 60) + + return { + "input_files": len(point_clouds), + "output_files": output_files, + "total_original": total_original, + "total_removed": total_removed, + "total_instances_removed": total_instances_removed, + } + + def main(): parser = argparse.ArgumentParser( description="Filter buffer zone instances - Remove instances whose centroids are in buffer zones", formatter_class=argparse.RawDescriptionHelpFormatter, ) - + parser.add_argument( "--input-dir", "-i", type=Path, required=True, help="Directory containing input LAZ tiles" ) - + parser.add_argument( "--output-dir", "-o", type=Path, required=True, help="Output directory for filtered LAZ files" ) - + parser.add_argument( "--buffer", type=float, default=10.0, help="Buffer zone distance in meters (default: 10.0)" ) - + parser.add_argument( "--suffix", type=str, default="_filtered", help="Suffix to add to output filenames (default: '_filtered')" ) - + + parser.add_argument( + "--instance-dimension", + type=str, + default="PredInstance", + help="Instance dimension to filter (default: PredInstance, fallback: treeID)" + ) + + parser.add_argument( + "--output-extension", + type=str, + default=None, + help="Optional output extension override, e.g. .laz for Galaxy collection discovery" + ) + args = parser.parse_args() - - # Find all input LAZ files - laz_files = sorted(args.input_dir.glob("*.laz")) - if not laz_files: - laz_files = sorted(args.input_dir.glob("*.las")) - - if len(laz_files) == 0: - print(f"No LAZ/LAS files found in {args.input_dir}") - return - - print("=" * 60) - print("Buffer Filter Preprocessing") - print("=" * 60) - print(f"Input directory: {args.input_dir}") - print(f"Output directory: {args.output_dir}") - print(f"Buffer: {args.buffer}m") - print(f"Found {len(laz_files)} tiles to process") - print("=" * 60) - - # Extract tile names for neighbor detection - all_tile_names = [] - for f in laz_files: - name = f.stem - for suffix in ['_segmented_remapped', '_segmented', '_remapped', '_filtered']: - name = name.replace(suffix, '') - all_tile_names.append(name) - - # Process each tile - total_original = 0 - total_removed = 0 - total_instances_removed = 0 - - for input_file in laz_files: - # Generate output filename - output_file = args.output_dir / f"{input_file.stem}{args.suffix}{input_file.suffix}" - - orig, removed, inst_removed = process_tile( - input_file, output_file, all_tile_names, args.buffer - ) - - total_original += orig - total_removed += removed - total_instances_removed += inst_removed - - print("\n" + "=" * 60) - print("Summary") - print("=" * 60) - print(f"Total points: {total_original:,} → {total_original - total_removed:,} " - f"({total_removed:,} removed, {100*total_removed/max(total_original,1):.1f}%)") - print(f"Total instances removed: {total_instances_removed}") - print("=" * 60) + filter_buffer_instances_dir( + input_dir=args.input_dir, + output_dir=args.output_dir, + buffer=args.buffer, + suffix=args.suffix, + instance_dimension=args.instance_dimension, + output_extension=args.output_extension, + ) if __name__ == "__main__": main() - diff --git a/src/get_bounds_from_tindex.py b/src/get_bounds_from_tindex.py index 13e742b..6cbed12 100755 --- a/src/get_bounds_from_tindex.py +++ b/src/get_bounds_from_tindex.py @@ -21,10 +21,10 @@ def load_extent_from_tindex(tindex_path: Path): """Load extent from tindex shapefile. - + Returns: Tuple of (minx, miny, maxx, maxy), crs_string - + The coordinates are returned in the native units of the data (assumed metric). """ with fiona.open(tindex_path) as src: @@ -36,16 +36,16 @@ def load_extent_from_tindex(tindex_path: Path): srs_info = crs.to_string() except Exception: srs_info = str(src.crs) - + print(f" Detected CRS: {srs_info} (Treating as Planar/Metric)", file=sys.stderr) - + if srs_info == "missing": print(f" ⚠ Warning: CRS missing; tiling in dataset-local planar coordinates.", file=sys.stderr) - + # Get bounds of all features minx = miny = math.inf maxx = maxy = -math.inf - + feature_count = 0 for feature in src: feature_count += 1 @@ -56,13 +56,13 @@ def load_extent_from_tindex(tindex_path: Path): coords = [c for poly in geom['coordinates'] for c in poly[0]] else: continue - + xs, ys = zip(*coords) minx = min(minx, min(xs)) miny = min(miny, min(ys)) maxx = max(maxx, max(xs)) maxy = max(maxy, max(ys)) - + if feature_count == 0: bounds = src.bounds if bounds and bounds != (0.0, 0.0, 0.0, 0.0): @@ -75,7 +75,7 @@ def load_extent_from_tindex(tindex_path: Path): def build_tiles(minx, miny, maxx, maxy, length, buffer, align_to_grid=False): """Build tile grid. - + Args: minx, miny, maxx, maxy: Data extent bounds length: Tile size in units @@ -88,7 +88,7 @@ def build_tiles(minx, miny, maxx, maxy, length, buffer, align_to_grid=False): f"Invalid bounds detected (infinity or NaN): " f"minx={minx}, miny={miny}, maxx={maxx}, maxy={maxy}." ) - + if align_to_grid: start_x = math.floor(minx / length) * length start_y = math.floor(miny / length) * length @@ -106,12 +106,12 @@ def build_tiles(minx, miny, maxx, maxy, length, buffer, align_to_grid=False): ) end_x = math.ceil(x_range / length) * length + start_x end_y = math.ceil(y_range / length) * length + start_y - + # Estimate number of tiles and warn if excessive num_tiles_x = int(math.ceil((end_x - start_x) / length)) num_tiles_y = int(math.ceil((end_y - start_y) / length)) total_tiles = num_tiles_x * num_tiles_y - + # Warn if creating too many tiles (more than 1 million) MAX_TILES = 1000000 if total_tiles > MAX_TILES: @@ -182,24 +182,24 @@ def main(): parser.add_argument( "--out", type=Path, - default=Path("/home/kg281/data/output/pdal_experiments/tile_bounds_tindex.json"), + default=Path("tile_bounds_tindex.json"), help="Where to write the tile bounds JSON summary", ) args = parser.parse_args() (minx, miny, maxx, maxy), srs = load_extent_from_tindex(args.tindex_path) - + # Always treat as planar/metric proj_minx, proj_miny, proj_maxx, proj_maxy = minx, miny, maxx, maxy proj_crs = srs if srs != "missing" else args.proj_crs - + # geo_extent matches proj_extent because we assume metric geo_minx, geo_miny, geo_maxx, geo_maxy = minx, miny, maxx, maxy - + # Use data-aligned tiling tiles, grid_bounds = build_tiles( - proj_minx, proj_miny, proj_maxx, proj_maxy, - args.tile_length, args.tile_buffer, + proj_minx, proj_miny, proj_maxx, proj_maxy, + args.tile_length, args.tile_buffer, align_to_grid=False, ) diff --git a/src/instance_labels.py b/src/instance_labels.py new file mode 100644 index 0000000..cfa04c5 --- /dev/null +++ b/src/instance_labels.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""SmartTile prediction instance ID output contract. + +SmartTile persists prediction instance dimensions as unsigned integer extra +bytes. The semantic contract is intentionally simple: + +- `0` means background/no tree. +- Positive values are tree instance IDs. +- Negative values are invalid at input/output boundaries. +- `uint32` is used only when an instance ID exceeds the `uint16` range. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Optional + +import laspy +import numpy as np + + +INSTANCE_UINT32_THRESHOLD = np.iinfo(np.uint16).max +INSTANCE_DEFAULT_OUTPUT_DTYPE = np.uint16 +INSTANCE_LARGE_OUTPUT_DTYPE = np.uint32 +MERGED_OUTPUT_SCALES = np.array([0.01, 0.01, 0.01], dtype=np.float64) + + +def validate_prediction_instance_labels( + instances: np.ndarray, + name: str = "PredInstance", + source: Optional[Path | str] = None, +) -> None: + """Fail fast when SmartTile receives invalid negative prediction labels.""" + arr = np.asarray(instances) + if arr.size == 0 or not np.any(arr < 0): + return + + negative_count = int(np.count_nonzero(arr < 0)) + min_value = int(np.min(arr)) + source_prefix = f"{source} " if source is not None else "" + raise ValueError( + f"{source_prefix}contains {negative_count} negative {name} values " + f"(minimum {min_value}). SmartTile expects {name}=0 for background/no tree " + "and positive values for tree instances." + ) + + +def instance_output_dtype(instances: Optional[np.ndarray] = None) -> np.dtype: + """Return uint32 only when any instance ID exceeds the uint16 range.""" + if instances is None: + return np.dtype(INSTANCE_DEFAULT_OUTPUT_DTYPE) + + arr = np.asarray(instances) + if arr.size == 0: + return np.dtype(INSTANCE_DEFAULT_OUTPUT_DTYPE) + validate_prediction_instance_labels(arr, "instance IDs") + + max_instance = int(np.max(arr)) + if max_instance > INSTANCE_UINT32_THRESHOLD: + return np.dtype(INSTANCE_LARGE_OUTPUT_DTYPE) + return np.dtype(INSTANCE_DEFAULT_OUTPUT_DTYPE) + + +def instance_extra_bytes_params(name: str, instances: Optional[np.ndarray] = None) -> laspy.ExtraBytesParams: + """Return the persisted SmartTile instance dimension schema.""" + return laspy.ExtraBytesParams(name=name, type=instance_output_dtype(instances)) + + +def cast_instances_for_output(instances: np.ndarray, name: str = "instance IDs") -> np.ndarray: + """Cast non-negative instance IDs to the persisted unsigned output dtype.""" + arr = np.asarray(instances) + validate_prediction_instance_labels(arr, name) + return arr.astype(instance_output_dtype(arr), copy=False) + + +def validate_merged_output_contract( + merged_laz: Path, + instance_dimension: str = "PredInstance", +) -> None: + """Verify persisted merged LAZ follows the SmartTile output contract.""" + with laspy.open(str(merged_laz), laz_backend=laspy.LazBackend.LazrsParallel) as f: + scales = np.asarray(f.header.scales, dtype=np.float64) + if not np.allclose(scales, MERGED_OUTPUT_SCALES): + raise ValueError( + f"{merged_laz} has XYZ scales {scales.tolist()}, expected " + f"{MERGED_OUTPUT_SCALES.tolist()} for 1cm merged output" + ) + + extra_dims = {dim.name: dim for dim in f.header.point_format.extra_dimensions} + if instance_dimension not in extra_dims: + raise ValueError(f"{merged_laz} is missing required {instance_dimension} extra dimension") + dtype = np.dtype(extra_dims[instance_dimension].dtype) + if dtype not in { + np.dtype(INSTANCE_DEFAULT_OUTPUT_DTYPE), + np.dtype(INSTANCE_LARGE_OUTPUT_DTYPE), + }: + raise ValueError( + f"{merged_laz} has {instance_dimension} dtype {dtype}, expected uint16 or uint32" + ) + + point_data = f.read() + instances = np.asarray(getattr(point_data, instance_dimension)) + expected_dtype = instance_output_dtype(instances) + if dtype != expected_dtype: + raise ValueError( + f"{merged_laz} has {instance_dimension} dtype {dtype}, expected " + f"{expected_dtype} for max instance ID {int(np.max(instances)) if instances.size else 0}" + ) diff --git a/src/main_create_merged_file.py b/src/main_create_merged_file.py new file mode 100644 index 0000000..6bfcdb0 --- /dev/null +++ b/src/main_create_merged_file.py @@ -0,0 +1,943 @@ +#!/usr/bin/env python3 +""" +Create prod-merged point-cloud products from Original-with-predictions files. + +Prod-merged products use original uploaded points as geometry and select real +points with PDAL's nearest-to-centroid voxel subsampling when a lower product +resolution is requested. +""" + +from __future__ import annotations + +import json +import math +import os +import re +import shutil +import subprocess +from pathlib import Path +from typing import Iterable, List, Optional, Tuple + +from point_cloud_metadata import ( + copc_files, + load_standardization_dims, + point_cloud_dimension_names, + point_cloud_source_key as source_key, + raw_point_cloud_files as point_cloud_files, +) +from copc_staging import ( + staged_copc_matches_source, + write_copc_stage_manifest_entry, +) + +SUPPORTED_OUTPUT_FORMATS = {"laz", "copc.laz", "ply"} + + +def get_pdal_path() -> str: + import shutil + + pdal_path = shutil.which("pdal") + if pdal_path is None: + raise RuntimeError("PDAL executable not found on PATH; create_merged_file requires pdal") + return pdal_path + + +def resolution_label(resolution: float) -> str: + centimeters = resolution * 100.0 + if math.isclose(centimeters, round(centimeters), rel_tol=0.0, abs_tol=1e-9): + return f"{int(round(centimeters))}cm" + return f"{centimeters:g}cm" + + +def parse_merged_resolutions( + value: str, + res1: float, + res2: float, +) -> List[Tuple[str, float]]: + """Parse create_merged_file resolution selector into unique label/resolution pairs.""" + aliases = { + "res1": ("res1", res1), + "resolution1": ("res1", res1), + "resolution_1": ("res1", res1), + "resolution-1": ("res1", res1), + "res2": ("res2", res2), + "resolution2": ("res2", res2), + "resolution_2": ("res2", res2), + "resolution-2": ("res2", res2), + } + parsed = [] + seen = set() + + for raw_token in (value or "res1,res2").split(","): + token = raw_token.strip().lower() + if not token: + continue + if token in aliases: + base_label, resolution = aliases[token] + label = resolution_label(resolution) + elif token.endswith("cm"): + number = token[:-2].strip() + resolution = float(number) / 100.0 + label = resolution_label(resolution) + else: + resolution = float(token) + label = resolution_label(resolution) + + if resolution <= 0: + raise ValueError("merged resolutions must be positive") + key = round(resolution, 9) + if key in seen: + continue + seen.add(key) + parsed.append((label, resolution)) + + if not parsed: + raise ValueError("No merged resolutions selected") + return parsed + + +def _selector_tokens(value) -> Iterable[str]: + """Yield selector tokens from CLI strings, Galaxy lists, or list-like strings.""" + if isinstance(value, (list, tuple, set)): + for item in value: + yield from _selector_tokens(item) + return + text = str(value or "") + if text.startswith("[") and text.endswith("]"): + text = text[1:-1] + for raw_token in text.split(","): + yield raw_token.strip().strip("'\"") + + +def parse_merged_output_formats(value: str) -> List[str]: + """Parse prod-merged output format selector into unique normalized formats.""" + aliases = { + "las": "laz", + "laz": "laz", + ".laz": "laz", + "copc": "copc.laz", + "copc_laz": "copc.laz", + "copc-laz": "copc.laz", + "copc.laz": "copc.laz", + ".copc.laz": "copc.laz", + "ply": "ply", + ".ply": "ply", + } + parsed = [] + seen = set() + for raw_token in _selector_tokens(value or "copc.laz"): + token = raw_token.strip().lower() + if not token: + continue + output_format = aliases.get(token) + if output_format is None: + raise ValueError( + f"Unsupported merged output format '{raw_token}'. " + "Use laz, copc.laz, or ply." + ) + if output_format in seen: + continue + seen.add(output_format) + parsed.append(output_format) + if not parsed: + raise ValueError("No merged output formats selected") + return parsed + + +def prod_merged_output_path(output_dir: Path, label: str, output_format: str = "copc.laz") -> Path: + safe_label = re.sub(r"[^A-Za-z0-9_.-]+", "_", label).strip("_") + normalized_format = parse_merged_output_formats(output_format)[0] + return output_dir / f"prod_merged_{safe_label}.{normalized_format}" + + +def expensive_prod_merged_warning(label: str, resolution: float, output_format: str) -> Optional[str]: + """Return a warning for product requests known to be scratch-disk heavy.""" + normalized_format = parse_merged_output_formats(output_format)[0] + if normalized_format == "copc.laz" and resolution <= 0.010000001: + return ( + f"{label} COPC prod-merged output can be very slow and scratch-disk heavy. " + "For large datasets, prefer 10cm COPC for routine downloads or request " + "1cm as LAZ unless a full-resolution COPC is explicitly needed." + ) + return None + + +def _keep_failed_chunk_work_dir() -> bool: + return os.environ.get("SMARTTILE_KEEP_FAILED_CHUNKS", "").strip().lower() in { + "1", + "true", + "yes", + } + + +def _cleanup_chunk_work_dir(work_dir: Path, success: bool) -> None: + """Remove scratch chunks by default so failed large jobs do not fill disk.""" + if not work_dir.exists(): + return + if success or not _keep_failed_chunk_work_dir(): + shutil.rmtree(work_dir, ignore_errors=True) + + +def _remove_existing_output(path: Path) -> None: + try: + path.unlink(missing_ok=True) + except OSError as exc: + raise RuntimeError(f"Could not remove existing output before rewrite: {path}: {exc}") from exc + + +def _is_reusable_copc(path: Path) -> bool: + """Return True when a staged COPC exists and has a readable LAS/COPC header.""" + if not path.exists() or path.stat().st_size == 0: + return False + try: + import laspy + + with laspy.open(str(path), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + return int(reader.header.point_count) > 0 + except Exception as exc: + print(f" Ignoring unreadable staged COPC {path.name}: {exc}") + return False + + +def _copc_by_source( + directory: Optional[Path], + source_by_key: Optional[dict[str, Path]] = None, +) -> dict[str, Path]: + if directory is None or not directory.exists(): + return {} + by_source = {} + for path in copc_files(directory): + key = source_key(path) + source_file = source_by_key.get(key) if source_by_key else None + if _is_reusable_copc(path) and staged_copc_matches_source(path, source_file): + by_source[key] = path + elif source_file is not None: + print( + f" Ignoring stale staged COPC {path.name}: " + f"no fresh source manifest for {source_file.name}" + ) + return by_source + + +def prepare_copc_inputs( + original_with_predictions_dir: Path, + output_dir: Path, + staged_copc_dir: Optional[Path] = None, +) -> List[Path]: + """Convert Original-with-predictions LAZ/LAS files to COPC using SmartTile's tiling converter.""" + if not original_with_predictions_dir.exists(): + raise FileNotFoundError(f"Original-with-predictions directory not found: {original_with_predictions_dir}") + + existing_copc = [path for path in copc_files(original_with_predictions_dir) if _is_reusable_copc(path)] + raw_files = point_cloud_files(original_with_predictions_dir) + if not raw_files and not existing_copc: + raise FileNotFoundError(f"No LAZ/LAS files found in {original_with_predictions_dir}") + + copc_dir = output_dir / "original_with_predictions_copc" + copc_dir.mkdir(parents=True, exist_ok=True) + all_input_files = [*raw_files, *existing_copc] + expected_keys = sorted({source_key(path) for path in all_input_files}) + raw_by_source = {source_key(path): path for path in raw_files} + selected_by_source = { + source_key(path): path + for path in existing_copc + if source_key(path) not in raw_by_source + or staged_copc_matches_source(path, raw_by_source[source_key(path)]) + } + selected_by_source.update({ + key: path + for key, path in _copc_by_source(staged_copc_dir, raw_by_source).items() + if key in expected_keys and key not in selected_by_source + }) + selected_by_source.update({ + key: path + for key, path in _copc_by_source(copc_dir, raw_by_source).items() + if key in expected_keys and key not in selected_by_source + }) + converter = None + + for input_file in raw_files: + key = source_key(input_file) + if key in selected_by_source: + print(f" Reusing existing COPC for {input_file.name}: {selected_by_source[key].name}") + continue + + output_copc = copc_dir / f"{key}.copc.laz" + print(f" Converting {input_file.name} -> {output_copc.name}") + if converter is None: + from main_tile import _convert_laz_to_copc + + converter = _convert_laz_to_copc + if not converter(input_file, output_copc, preserve_extra_dims=True): + raise RuntimeError(f"LAZ/LAS -> COPC conversion failed: {input_file}") + write_copc_stage_manifest_entry(input_file, output_copc) + selected_by_source[key] = output_copc + + return [selected_by_source[key] for key in expected_keys] + + +def prod_merged_writer_stage(output_file: Path, output_format: str) -> dict: + """Return the PDAL writer stage for a prod-merged output format.""" + normalized_format = parse_merged_output_formats(output_format)[0] + if normalized_format == "laz": + return { + "type": "writers.las", + "filename": str(output_file), + "compression": True, + "forward": "all", + "extra_dims": "all", + } + if normalized_format == "copc.laz": + return { + "type": "writers.copc", + "filename": str(output_file), + "forward": "all", + "extra_dims": "all", + } + if normalized_format == "ply": + return { + "type": "writers.ply", + "filename": str(output_file), + "storage_mode": "little endian", + } + raise ValueError(f"Unsupported merged output format '{output_format}'") + + +def _scale_offset_options(source_file: Path) -> dict: + import laspy + + with laspy.open(str(source_file)) as reader: + header = reader.header + return { + "scale_x": float(header.scales[0]), + "scale_y": float(header.scales[1]), + "scale_z": float(header.scales[2]), + "offset_x": float(header.offsets[0]), + "offset_y": float(header.offsets[1]), + "offset_z": float(header.offsets[2]), + } + + +def prod_merged_pipeline( + input_files: List[Path], + output_file: Path, + resolution: float, + output_format: str = "copc.laz", +) -> dict: + """Build the PDAL pipeline for one prod-merged output.""" + readers = [ + { + "type": "readers.copc" if path.name.lower().endswith(".copc.laz") else "readers.las", + "filename": str(path), + } + for path in input_files + ] + pipeline = [*readers] + if len(readers) > 1: + pipeline.append({"type": "filters.merge"}) + pipeline.extend([ + {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, + prod_merged_writer_stage(output_file, output_format), + ]) + return {"pipeline": pipeline} + + +def _run_pdal_pipeline(pipeline: dict, pipeline_file: Path) -> subprocess.CompletedProcess: + pipeline_file.parent.mkdir(parents=True, exist_ok=True) + with open(pipeline_file, "w") as f: + json.dump(pipeline, f, indent=2) + try: + return subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + finally: + if pipeline_file.exists(): + pipeline_file.unlink() + + +def _pdal_error(result: subprocess.CompletedProcess, limit: int = 2000) -> str: + details = "\n".join(part for part in (result.stderr, result.stdout) if part) + return details[:limit] + + +def _point_cloud_point_count(path: Path) -> int: + """Return the LAS/LAZ/COPC point count from the header.""" + import laspy + + from copc_metadata import laspy_laz_backend + + with laspy.open(str(path), laz_backend=laspy_laz_backend()) as reader: + return int(reader.header.point_count) + + +def _copc_union_bounds(input_files: List[Path]) -> Tuple[float, float, float, float]: + from main_subsample import get_file_bounds + + bounds = [] + for input_file in input_files: + file_bounds = get_file_bounds(input_file) + if file_bounds is None: + raise RuntimeError(f"Could not determine bounds for {input_file}") + bounds.append(file_bounds) + + return ( + min(bound[0] for bound in bounds), + max(bound[1] for bound in bounds), + min(bound[2] for bound in bounds), + max(bound[3] for bound in bounds), + ) + + +def _prod_merged_chunk_bounds( + input_files: List[Path], + resolution: float, + num_spatial_chunks: int, +) -> List[str]: + from main_subsample import _aligned_edges, get_file_xy_scales + + minx, maxx, miny, maxy = _copc_union_bounds(input_files) + if maxx - minx == 0 or maxy - miny == 0: + return [f"([{minx},{maxx}],[{miny},{maxy}])"] + + raw_x_step = (maxx - minx) / max(1, num_spatial_chunks) + x_step = max(resolution, math.ceil(raw_x_step / resolution) * resolution) + scale_x = min(get_file_xy_scales(input_file)[0] for input_file in input_files) + edges = _aligned_edges(minx, maxx, x_step, resolution) + + bounds = [] + for chunk_idx, (chunk_minx, chunk_maxx) in enumerate(edges): + query_maxx = chunk_maxx if chunk_idx == len(edges) - 1 else chunk_maxx - scale_x * 0.5 + if query_maxx < chunk_minx: + continue + bounds.append(f"([{chunk_minx},{query_maxx}],[{miny},{maxy}])") + return bounds + + +def _chunked_prod_merged_pipeline( + input_files: List[Path], + output_file: Path, + bounds_str: str, + resolution: float, + scale_offset_options: dict, +) -> dict: + readers = [ + { + "type": "readers.copc" if path.name.lower().endswith(".copc.laz") else "readers.las", + "filename": str(path), + "bounds": bounds_str, + } + for path in input_files + ] + stages = [*readers] + if len(readers) > 1: + stages.append({"type": "filters.merge"}) + stages.extend([ + {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, + { + "type": "writers.las", + "filename": str(output_file), + "compression": True, + "minor_version": 4, + "forward": "all", + "extra_dims": "all", + **scale_offset_options, + }, + ]) + return {"pipeline": stages} + + +def _merge_chunk_files_pipeline( + chunk_files: List[Path], + output_file: Path, + output_format: str, + scale_offset_options: Optional[dict] = None, +) -> dict: + stages = [{"type": "readers.las", "filename": str(path)} for path in chunk_files] + if len(stages) > 1: + stages.append({"type": "filters.merge"}) + normalized_format = parse_merged_output_formats(output_format)[0] + writer = prod_merged_writer_stage(output_file, normalized_format) + if scale_offset_options and normalized_format in {"laz", "copc.laz"}: + writer.update(scale_offset_options) + stages.append(writer) + return {"pipeline": stages} + + +def _validate_expected_dims( + files: List[Path], + expected_dims: Optional[set[str]], + context: str, +) -> None: + """Fail when LAS/COPC files do not expose expected standardized dimensions.""" + if not expected_dims: + return + + available = set() + unreadable = [] + for path in files: + try: + available.update(point_cloud_dimension_names(path)) + except Exception as exc: + unreadable.append(f"{path.name}: {exc}") + + if unreadable: + raise RuntimeError( + f"Could not validate standardized dimensions for {context}: " + + "; ".join(unreadable) + ) + + missing = sorted(expected_dims - available) + if missing: + raise RuntimeError( + f"{context} is missing {len(missing)} standardized dimension(s): {missing}" + ) + + print( + f" Standardization JSON: {context} contains " + f"{len(expected_dims)} expected dimension(s)", + flush=True, + ) + + +def _validate_preserved_product_dims( + source_files: List[Path], + output_file: Path, +) -> Tuple[bool, str]: + """Validate that LAS/COPC product writing did not drop point dimensions.""" + expected = set() + unreadable = [] + for path in source_files: + try: + expected.update(point_cloud_dimension_names(path)) + except Exception as exc: + unreadable.append(f"{path.name}: {exc}") + + if unreadable: + return ( + False, + "could not inspect source product dimensions: " + "; ".join(unreadable), + ) + + try: + output_dims = point_cloud_dimension_names(output_file) + except Exception as exc: + return (False, f"could not inspect output product dimensions: {exc}") + + missing = sorted(expected - output_dims) + if missing: + return ( + False, + f"prod-merged output dropped {len(missing)} point dimension(s): {missing}", + ) + return (True, "product dimensions preserved") + + +def _preserve_and_validate_las_metadata(source_metadata_file: Path, output_file: Path) -> Tuple[bool, str]: + """Ensure a LAS/LAZ/COPC product carries source CRS/GeoTIFF projection metadata.""" + from copc_metadata import ( + append_source_geotiff_projection_evlrs, + copc_preserves_source_crs, + ) + + preserved_geotiff, message = append_source_geotiff_projection_evlrs( + source_metadata_file, + output_file, + ) + if not preserved_geotiff: + return (False, f"GeoTIFF projection preservation failed: {message}") + + valid_crs, message = copc_preserves_source_crs(source_metadata_file, output_file) + if not valid_crs: + return (False, f"CRS validation failed: {message}") + return (True, "LAS metadata preserved") + + +def _preserve_and_validate_copc_metadata(source_metadata_file: Path, output_file: Path) -> Tuple[bool, str]: + """Backward-compatible alias for older tests/callers.""" + return _preserve_and_validate_las_metadata(source_metadata_file, output_file) + + +def _untwine_chunk_files_to_copc( + chunk_files: List[Path], + output_file: Path, + source_metadata_file: Path, + temp_dir: Optional[Path] = None, +) -> Tuple[bool, str]: + """Write final COPC directly from chunk files, avoiding a giant merged LAZ.""" + untwine_cmd = shutil.which("untwine") + if not untwine_cmd: + return (False, "untwine not available") + + try: + if output_file.exists(): + output_file.unlink() + + from copc_metadata import srs_assignment_from_file + + input_args = [] + for chunk_file in chunk_files: + input_args.extend(["-i", str(chunk_file)]) + srs_arg = srs_assignment_from_file(source_metadata_file) + srs_args = ["--a_srs", srs_arg] if srs_arg else [] + temp_args = [] + if temp_dir is not None: + temp_dir.mkdir(parents=True, exist_ok=True) + temp_args = ["--temp_dir", str(temp_dir)] + + result = subprocess.run( + [untwine_cmd] + input_args + ["-o", str(output_file)] + srs_args + temp_args, + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + return (False, f"untwine failed: {_pdal_error(result)}") + if not output_file.exists() or output_file.stat().st_size == 0: + return (False, "untwine produced no output") + + expected_points = sum(_point_cloud_point_count(chunk_file) for chunk_file in chunk_files) + actual_points = _point_cloud_point_count(output_file) + if actual_points != expected_points: + try: + output_file.unlink() + except OSError: + pass + return ( + False, + "untwine point-count mismatch: " + f"expected {expected_points}, got {actual_points}", + ) + + valid_metadata, message = _preserve_and_validate_las_metadata( + source_metadata_file, + output_file, + ) + if not valid_metadata: + return (False, message) + valid_dims, message = _validate_preserved_product_dims(chunk_files, output_file) + if not valid_dims: + try: + output_file.unlink() + except OSError: + pass + return (False, message) + return (True, "untwine") + except Exception as exc: + return (False, f"untwine error: {exc}") + + +def _merge_prod_chunks( + chunk_files: List[Path], + output_file: Path, + output_format: str, + work_dir: Path, + source_metadata_file: Path, + scale_offset_options: dict, +) -> None: + normalized_format = parse_merged_output_formats(output_format)[0] + if normalized_format == "copc.laz": + success, message = _untwine_chunk_files_to_copc( + chunk_files, + output_file, + source_metadata_file, + temp_dir=work_dir / "_untwine_tmp", + ) + if success: + print(" Direct untwine chunk->COPC complete") + return + print(f" Direct untwine chunk->COPC failed; falling back to PDAL merge: {message}") + + pipeline = _merge_chunk_files_pipeline( + chunk_files, + output_file, + output_format, + scale_offset_options, + ) + result = _run_pdal_pipeline(pipeline, work_dir / f"_{output_file.stem}_merge_chunks.json") + if result.returncode == 0 and output_file.exists() and output_file.stat().st_size > 0: + if normalized_format in {"laz", "copc.laz"}: + valid_dims, message = _validate_preserved_product_dims(chunk_files, output_file) + if not valid_dims: + raise RuntimeError(message) + valid_metadata, message = _preserve_and_validate_las_metadata( + source_metadata_file, + output_file, + ) + if not valid_metadata: + raise RuntimeError(message) + return + + if normalized_format != "copc.laz": + raise RuntimeError(f"PDAL prod-merged chunk merge failed: {_pdal_error(result)}") + + temp_laz = work_dir / f"_{output_file.stem}_for_copc.laz" + temp_pipeline = _merge_chunk_files_pipeline(chunk_files, temp_laz, "laz", scale_offset_options) + temp_result = _run_pdal_pipeline(temp_pipeline, work_dir / f"_{output_file.stem}_merge_chunks_laz.json") + if temp_result.returncode != 0 or not temp_laz.exists() or temp_laz.stat().st_size == 0: + raise RuntimeError( + "PDAL prod-merged chunk merge failed and LAZ fallback failed: " + f"{_pdal_error(result)}\n{_pdal_error(temp_result)}" + ) + + from subsample_outputs import convert_laz_output_to_copc + + if not convert_laz_output_to_copc( + temp_laz, + output_file, + source_metadata_file=source_metadata_file, + preserve_extra_dims=True, + ): + raise RuntimeError(f"Prod-merged COPC conversion failed: {output_file}") + try: + temp_laz.unlink() + except OSError: + pass + + +def create_chunked_prod_merged_file( + copc_input_files: List[Path], + output_file: Path, + resolution: float, + output_format: str, + num_spatial_chunks: int, +) -> Path: + """Create one prod-merged product using bounded COPC reads per spatial chunk.""" + chunk_bounds = _prod_merged_chunk_bounds(copc_input_files, resolution, num_spatial_chunks) + if not chunk_bounds: + raise RuntimeError("No spatial chunks available for prod-merged output") + + _remove_existing_output(output_file) + work_dir = output_file.parent / f"_{output_file.stem}_chunks" + if work_dir.exists(): + shutil.rmtree(work_dir) + work_dir.mkdir(parents=True, exist_ok=True) + scale_offset_options = _scale_offset_options(copc_input_files[0]) + + chunk_files: List[Path] = [] + success = False + try: + print(f" Spatial chunks: {len(chunk_bounds)}") + for chunk_idx, bounds_str in enumerate(chunk_bounds): + chunk_file = work_dir / f"{output_file.stem}_chunk{chunk_idx:04d}.laz" + pipeline = _chunked_prod_merged_pipeline( + copc_input_files, + chunk_file, + bounds_str, + resolution, + scale_offset_options, + ) + result = _run_pdal_pipeline(pipeline, work_dir / f"_chunk{chunk_idx:04d}.json") + if result.returncode != 0: + raise RuntimeError(f"PDAL prod-merged chunk {chunk_idx + 1} failed: {_pdal_error(result)}") + if not chunk_file.exists() or chunk_file.stat().st_size == 0: + print(f" Chunk {chunk_idx + 1}/{len(chunk_bounds)}: empty") + continue + chunk_files.append(chunk_file) + print(f" Chunk {chunk_idx + 1}/{len(chunk_bounds)} complete: {chunk_file.name}") + + if not chunk_files: + raise RuntimeError("No prod-merged chunks were created") + + _merge_prod_chunks( + chunk_files, + output_file, + output_format, + work_dir, + source_metadata_file=copc_input_files[0], + scale_offset_options=scale_offset_options, + ) + success = output_file.exists() and output_file.stat().st_size > 0 + finally: + _cleanup_chunk_work_dir(work_dir, success) + + if not output_file.exists() or output_file.stat().st_size == 0: + raise RuntimeError(f"Prod-merged output was not created: {output_file}") + return output_file + + +def create_chunked_prod_merged_files_for_resolution( + copc_input_files: List[Path], + outputs: List[Tuple[Path, str]], + resolution: float, + num_spatial_chunks: int, +) -> List[Path]: + """Create multiple output formats from one canonical set of chunk files.""" + if not copc_input_files: + raise ValueError("No COPC input files provided") + if not outputs: + return [] + + chunk_bounds = _prod_merged_chunk_bounds(copc_input_files, resolution, num_spatial_chunks) + if not chunk_bounds: + raise RuntimeError("No spatial chunks available for prod-merged output") + + for output_file, _output_format in outputs: + _remove_existing_output(output_file) + + first_output = outputs[0][0] + work_dir = first_output.parent / f"_{first_output.stem}_shared_chunks" + if work_dir.exists(): + shutil.rmtree(work_dir) + work_dir.mkdir(parents=True, exist_ok=True) + scale_offset_options = _scale_offset_options(copc_input_files[0]) + + chunk_files: List[Path] = [] + created: List[Path] = [] + success = False + try: + print(f" Spatial chunks: {len(chunk_bounds)}") + for chunk_idx, bounds_str in enumerate(chunk_bounds): + chunk_file = work_dir / f"prod_merged_chunk{chunk_idx:04d}.laz" + pipeline = _chunked_prod_merged_pipeline( + copc_input_files, + chunk_file, + bounds_str, + resolution, + scale_offset_options, + ) + result = _run_pdal_pipeline(pipeline, work_dir / f"_chunk{chunk_idx:04d}.json") + if result.returncode != 0: + raise RuntimeError(f"PDAL prod-merged chunk {chunk_idx + 1} failed: {_pdal_error(result)}") + if not chunk_file.exists() or chunk_file.stat().st_size == 0: + print(f" Chunk {chunk_idx + 1}/{len(chunk_bounds)}: empty") + continue + chunk_files.append(chunk_file) + print(f" Chunk {chunk_idx + 1}/{len(chunk_bounds)} complete: {chunk_file.name}") + + if not chunk_files: + raise RuntimeError("No prod-merged chunks were created") + + for output_file, output_format in outputs: + _merge_prod_chunks( + chunk_files, + output_file, + output_format, + work_dir, + source_metadata_file=copc_input_files[0], + scale_offset_options=scale_offset_options, + ) + if not output_file.exists() or output_file.stat().st_size == 0: + raise RuntimeError(f"Prod-merged output was not created: {output_file}") + created.append(output_file) + success = len(created) == len(outputs) and all( + path.exists() and path.stat().st_size > 0 for path in created + ) + finally: + _cleanup_chunk_work_dir(work_dir, success) + + return created + + +def create_prod_merged_file( + copc_input_files: List[Path], + output_file: Path, + resolution: float, + output_format: str = "copc.laz", + num_spatial_chunks: Optional[int] = None, +) -> Path: + """Create one prod-merged product at the requested resolution and format.""" + if not copc_input_files: + raise ValueError("No COPC input files provided") + + output_file.parent.mkdir(parents=True, exist_ok=True) + _remove_existing_output(output_file) + if num_spatial_chunks and num_spatial_chunks > 1: + return create_chunked_prod_merged_file( + copc_input_files, + output_file, + resolution, + output_format, + num_spatial_chunks, + ) + + pipeline = prod_merged_pipeline(copc_input_files, output_file, resolution, output_format) + pipeline_file = output_file.parent / f"_{output_file.stem}_pipeline.json" + result = _run_pdal_pipeline(pipeline, pipeline_file) + + if result.returncode != 0: + raise RuntimeError(f"PDAL prod-merged pipeline failed: {_pdal_error(result, 500)}") + if not output_file.exists() or output_file.stat().st_size == 0: + raise RuntimeError(f"Prod-merged output was not created: {output_file}") + if parse_merged_output_formats(output_format)[0] in {"laz", "copc.laz"}: + valid_dims, message = _validate_preserved_product_dims(copc_input_files, output_file) + if not valid_dims: + raise RuntimeError(message) + valid_metadata, message = _preserve_and_validate_las_metadata( + copc_input_files[0], + output_file, + ) + if not valid_metadata: + raise RuntimeError(message) + return output_file + + +def create_prod_merged_files( + original_with_predictions_dir: Path, + output_dir: Path, + resolution_selector: str, + output_format_selector: str, + res1: float, + res2: float, + num_spatial_chunks: Optional[int] = None, + staged_copc_dir: Optional[Path] = None, + standardization_json: Optional[Path] = None, +) -> List[Path]: + """Create all selected prod-merged products.""" + outputs = [] + expected_dims = None + if standardization_json: + expected_dims = load_standardization_dims(Path(standardization_json)) + print( + f" Standardization JSON: validating expected dimensions from {standardization_json}", + flush=True, + ) + + print(" Preparing COPC inputs with SmartTile untwine/PDAL fallback conversion") + copc_input_files = prepare_copc_inputs( + original_with_predictions_dir, + output_dir, + staged_copc_dir=staged_copc_dir, + ) + _validate_expected_dims(copc_input_files, expected_dims, "staged Original-with-predictions COPCs") + output_formats = parse_merged_output_formats(output_format_selector) + for label, resolution in parse_merged_resolutions(resolution_selector, res1, res2): + selected_outputs = [ + (prod_merged_output_path(output_dir, label, output_format), output_format) + for output_format in output_formats + ] + for output_file, output_format in selected_outputs: + warning = expensive_prod_merged_warning(label, resolution, output_format) + if warning: + print(f" Warning: {warning}") + print( + f" Creating {output_file.name} at {resolution:g}m " + f"with nearest-to-centroid" + ) + + if num_spatial_chunks and num_spatial_chunks > 1: + created_outputs = create_chunked_prod_merged_files_for_resolution( + copc_input_files, + selected_outputs, + resolution, + num_spatial_chunks, + ) + else: + created_outputs = [ + create_prod_merged_file( + copc_input_files, + output_file, + resolution, + output_format, + num_spatial_chunks=num_spatial_chunks, + ) + for output_file, output_format in selected_outputs + ] + + for created, output_format in zip(created_outputs, output_formats): + if parse_merged_output_formats(output_format)[0] in {"laz", "copc.laz"}: + _validate_expected_dims([created], expected_dims, created.name) + outputs.append(created) + return outputs diff --git a/src/main_merge.py b/src/main_merge.py new file mode 100644 index 0000000..b2f19d3 --- /dev/null +++ b/src/main_merge.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python3 +""" +Main merge script: Merge segmented tiles with instance matching. + +This script wraps the merge_tiles.py functionality to provide a clean interface +for the pipeline orchestrator. + +Pipeline: +1. Load and filter (centroid-based buffer zone filtering) +2. Assign global IDs +3. Cross-tile instance matching +4. Merge and deduplicate +5. Small volume merging +6. Retile to original files (required) + +Usage: + python main_merge.py --segmented_folder /path/to/segmented_remapped +""" + +from __future__ import annotations + +import argparse +import shutil +import sys +from pathlib import Path +from typing import List, Optional + +# Import parameters and core merge function +from instance_labels import MERGED_OUTPUT_SCALES, validate_merged_output_contract +from parameters import MERGE_PARAMS +from merge_tiles import merge_tiles as core_merge_tiles +from point_cloud_metadata import point_cloud_files as _point_cloud_files + + +def _original_with_predictions_name(path: Path) -> str: + name = path.name + if name.lower().endswith(".copc.laz"): + return name[:-9] + ".laz" + return name + + +def run_merge( + segmented_dir: Path, + output_tiles_dir: Path, + original_tiles_dir: Path, + tile_bounds_json: Path, + original_input_dir: Optional[Path] = None, + output_merged: Optional[Path] = None, + buffer: float = 10.0, + overlap_threshold: float = 0.3, + max_centroid_distance: float = 3.0, + correspondence_tolerance: float = 0.05, + max_volume_for_merge: float = 4.0, + border_zone_width: float = 10.0, + min_cluster_size: int = 300, + num_threads: int = 4, + enable_matching: bool = True, + require_overlap: bool = True, + enable_volume_merge: bool = True, + skip_merged_file: bool = False, + verbose: bool = True, + retile_buffer: float = 2.0, # Fixed to 2.0m + retile_max_radius: float = 0.1, + instance_dimension: str = "PredInstance", + transfer_original_dims_to_merged: bool = True, + threedtrees_dims: Optional[List[str]] = None, + threedtrees_suffix: str = "SAT", +) -> Path: + """ + Run the tile merge pipeline. + + Args: + segmented_dir: Directory containing segmented LAZ tiles + output_tiles_dir: Output directory for retiled files + original_tiles_dir: Directory with original tile files for retiling + original_input_dir: Directory with original input LAZ files for final remap (optional) + output_merged: Output path for merged LAZ file (auto-derived if None) + buffer: Buffer zone distance in meters + overlap_threshold: Overlap ratio threshold for instance matching + max_centroid_distance: Max distance between centroids to merge instances + correspondence_tolerance: Max distance for point correspondence (internal, not exposed via Parameters) + max_volume_for_merge: Max convex hull volume for small instance merging + num_threads: Number of workers for parallel processing + enable_matching: Enable cross-tile instance matching + require_overlap: Require overlap ratio check (vs centroid distance only) + enable_volume_merge: Enable small volume instance merging + skip_merged_file: Skip creating merged LAZ file (only retile) + verbose: Print detailed merge decisions + retile_buffer: Spatial buffer expansion in meters for filtering merged points during retiling + retile_max_radius: Maximum distance threshold in meters for cKDTree nearest neighbor matching during retiling + transfer_original_dims_to_merged: Legacy compatibility flag for callers that still pass the old merged-enrichment option. The orchestrator creates prod-merged outputs separately from Original-with-predictions files. + + Returns: + Path to merged output file + """ + print("=" * 60) + print("3DTrees Merge Pipeline") + print("=" * 60) + + # Validate input + if not segmented_dir.exists(): + raise ValueError(f"Segmented directory not found: {segmented_dir}") + + if not original_tiles_dir.exists(): + raise ValueError(f"Original tiles directory not found: {original_tiles_dir}") + + if not output_tiles_dir.exists(): + output_tiles_dir.mkdir(parents=True, exist_ok=True) + + # Validate tile bounds JSON (required) + if tile_bounds_json is None: + raise ValueError("tile_bounds_json is required but was not provided.") + if not tile_bounds_json.exists(): + raise FileNotFoundError( + f"tile_bounds_tindex.json not found: {tile_bounds_json}. " + "Merge requires this file and will not run without it." + ) + + # Auto-derive output path if not provided (inside segmented dir so it is writable e.g. in Docker /out) + if output_merged is None: + output_merged = segmented_dir / "merged.laz" + + # OPTIMIZATION: If only one file in each folder, skip merge and just remap + segmented_files = _point_cloud_files(segmented_dir) + original_tiles_files = _point_cloud_files(original_tiles_dir) + + # Check if we should use the single-file optimization + use_single_file_optimization = False + if len(segmented_files) == 1 and len(original_tiles_files) == 1: + # If original_input_dir is provided, also check it has exactly one file + if original_input_dir: + original_input_files = _point_cloud_files(original_input_dir) + if len(original_input_files) == 1: + use_single_file_optimization = True + else: + # No original_input_dir requirement + use_single_file_optimization = True + + if use_single_file_optimization: + print("\n" + "=" * 60) + print("SINGLE FILE DETECTED - Using optimized path") + print("=" * 60) + print(f"Segmented files: {len(segmented_files)}") + print(f"Original tiles: {len(original_tiles_files)}") + if original_input_dir: + print(f"Original inputs: {len(original_input_files)}") + print("\nSkipping merge steps (no cross-tile matching needed)") + print("Directly remapping segmented → 1cm → original") + print() + + # Import remap function + from main_remap import remap_single_tile + + # Step 1: Remap segmented to 1cm (original_tiles) + segmented_file = segmented_files[0] + target_file = original_tiles_files[0] + + print(f"Step 1: Remapping {segmented_file.name} → {target_file.name}") + remapped_1cm_file = output_tiles_dir / f"{target_file.stem}_segmented.laz" + + _, success, message, point_count = remap_single_tile( + segmented_file, + target_file, + remapped_1cm_file, + instance_dimension=instance_dimension, + output_scales=tuple(MERGED_OUTPUT_SCALES), + ) + + if not success: + raise RuntimeError(f"Failed to remap to 1cm: {message}") + + print(f" ✓ Remapped {point_count:,} points to 1cm resolution") + + # Step 2: If original_input_dir provided, remap to original + if original_input_dir: + original_file = original_input_files[0] + print(f"\nStep 2: Remapping {remapped_1cm_file.name} → {original_file.name}") + + original_output_dir = output_tiles_dir.parent / "original_with_predictions" + final_output_file = original_output_dir / _original_with_predictions_name(original_file) + + _, success, message, point_count = remap_single_tile( + remapped_1cm_file, + original_file, + final_output_file, + threedtrees_dims=set(threedtrees_dims) if threedtrees_dims else None, + threedtrees_suffix=threedtrees_suffix, + instance_dimension=instance_dimension, + output_scales=None, + ) + + if not success: + raise RuntimeError(f"Failed to remap to original: {message}") + + print(f" ✓ Remapped {point_count:,} points to original resolution") + + # Step 3: Write merged file (remapped file renamed to output_merged) + # Always use the target-resolution remap as merged source. If original + # inputs are present, Step 2 writes a separate original-resolution output. + merged_source = remapped_1cm_file + if not skip_merged_file and output_merged is not None: + output_merged = Path(output_merged) + output_merged.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(str(merged_source), str(output_merged)) + print(f"\nStep 3: Merged file written: {output_merged.name}") + + validate_merged_output_contract(output_merged, instance_dimension) + + print("\n" + "=" * 60) + print("Single-file optimization complete") + print("=" * 60) + print(f"Output: {output_tiles_dir}") + if not skip_merged_file and output_merged is not None: + print(f"Merged file: {output_merged}") + + return output_merged + + print(f"Input: {segmented_dir}") + print(f"Output merged: {output_merged}" + (" (SKIPPED)" if skip_merged_file else "")) + print(f"Buffer: {buffer}m") + print(f"Instance matching: {'ENABLED' if enable_matching else 'DISABLED'}") + if enable_matching: + print(f" Overlap threshold: {overlap_threshold}") + print(f" Max centroid distance: {max_centroid_distance}m") + print(f"Small cluster reassignment: ENABLED") + print(f" Min cluster size: {min_cluster_size} points") + print(f"Volume merge: {'ENABLED' if enable_volume_merge else 'DISABLED'}") + if enable_volume_merge: + print(f" Max volume: {max_volume_for_merge} m³") + print(f"Workers: {num_threads}") + print(f"Tile bounds JSON: {tile_bounds_json}") + if original_input_dir: + print(f"Original input dir: {original_input_dir} (Stage 7 enabled)") + print() + + # Run the core merge function + core_merge_tiles( + input_dir=segmented_dir, + original_tiles_dir=original_tiles_dir, + output_merged=output_merged, + output_tiles_dir=output_tiles_dir, + tile_bounds_json=tile_bounds_json, + original_input_dir=original_input_dir, + buffer=buffer, + overlap_threshold=overlap_threshold, + correspondence_tolerance=correspondence_tolerance, + max_volume_for_merge=max_volume_for_merge, + border_zone_width=border_zone_width, + min_cluster_size=min_cluster_size, + num_threads=num_threads, + enable_matching=enable_matching, + enable_volume_merge=enable_volume_merge, + skip_merged_file=skip_merged_file, + verbose=verbose, + retile_buffer=retile_buffer, + instance_dimension=instance_dimension, + transfer_original_dims_to_merged=transfer_original_dims_to_merged, + threedtrees_dims=threedtrees_dims, + threedtrees_suffix=threedtrees_suffix, + ) + + return output_merged + + +def main() -> None: + """CLI entry point.""" + parser = argparse.ArgumentParser( + description="3DTrees Merge Pipeline - Merge segmented tiles with instance matching", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + + parser.add_argument( + "--segmented_dir", "--segmented_folder", "-i", + type=Path, + required=True, + dest="segmented_dir", + help="Directory containing segmented LAZ tiles" + ) + + parser.add_argument( + "--output_merged", "-o", + type=Path, + default=None, + help="Output path for merged LAZ file (auto-derived if not specified)" + ) + + parser.add_argument( + "--output_tiles_dir", + type=Path, + required=True, + help="Output directory for retiled files (required)" + ) + + parser.add_argument( + "--original_tiles_dir", + type=Path, + required=True, + help="Directory with original tile files for retiling (required)" + ) + + parser.add_argument( + "--tile_bounds_json", + type=Path, + required=True, + help="Path to tile_bounds_tindex.json (required; used for neighbor graph)" + ) + + parser.add_argument( + "--original_input_dir", + type=Path, + default=None, + help="Directory with original input LAZ files for final remap (optional, enables Stage 7)" + ) + + parser.add_argument( + "--buffer", + type=float, + default=MERGE_PARAMS.get('buffer', 10.0), + help=f"Buffer zone distance in meters (default: {MERGE_PARAMS.get('buffer', 10.0)})" + ) + + parser.add_argument( + "--overlap_threshold", + type=float, + default=MERGE_PARAMS.get('overlap_threshold', 0.3), + help=f"Overlap ratio threshold (default: {MERGE_PARAMS.get('overlap_threshold', 0.3)})" + ) + + parser.add_argument( + "--max_centroid_distance", + type=float, + default=MERGE_PARAMS.get('max_centroid_distance', 3.0), + help=f"Max centroid distance (default: {MERGE_PARAMS.get('max_centroid_distance', 3.0)})" + ) + + parser.add_argument( + "--correspondence_tolerance", + type=float, + default=0.05, + help="Max distance for point correspondence during merge (default: 0.05m)" + ) + + parser.add_argument( + "--max_volume_for_merge", + type=float, + default=MERGE_PARAMS.get('max_volume_for_merge', 4.0), + help=f"Max volume for small instance merge (default: {MERGE_PARAMS.get('max_volume_for_merge', 4.0)})" + ) + + parser.add_argument( + "--min_cluster_size", + type=int, + default=MERGE_PARAMS.get('min_cluster_size', 300), + help=f"Minimum cluster size in points for reassignment (default: {MERGE_PARAMS.get('min_cluster_size', 300)})" + ) + + parser.add_argument( + "--num_threads", "--workers", + type=int, + default=MERGE_PARAMS.get('workers', 4), + dest="num_threads", + help=f"Number of workers (default: {MERGE_PARAMS.get('workers', 4)})" + ) + + parser.add_argument( + "--border_zone_width", + type=float, + default=MERGE_PARAMS.get('border_zone_width', 10.0), + help=f"Width of border zone beyond buffer for instance matching (default: {MERGE_PARAMS.get('border_zone_width', 10.0)})" + ) + + parser.add_argument( + "--retile_buffer", + type=float, + default=2.0, + help="Spatial buffer expansion in meters for retiling (fixed: 2.0m)" + ) + + parser.add_argument( + "--retile_max_radius", + type=float, + default=MERGE_PARAMS.get('retile_max_radius', 0.1), + help=f"Max distance for nearest neighbor matching during retiling (default: {MERGE_PARAMS.get('retile_max_radius', 0.1)})" + ) + + parser.add_argument( + "--disable_matching", + action="store_true", + help="Disable cross-tile instance matching" + ) + + parser.add_argument( + "--disable_overlap_check", + action="store_true", + help="Disable overlap ratio check (centroid distance only)" + ) + + parser.add_argument( + "--disable_volume_merge", + action="store_true", + help="Disable small volume instance merging" + ) + + parser.add_argument( + "--skip_merged_file", + action="store_true", + help="Skip creating merged LAZ file (only create retiled outputs)" + ) + + parser.add_argument( + "--verbose", "-v", + action="store_true", + help="Print detailed merge decisions" + ) + + args = parser.parse_args() + + # Run pipeline + try: + output_file = run_merge( + segmented_dir=args.segmented_dir, + output_tiles_dir=args.output_tiles_dir, + original_tiles_dir=args.original_tiles_dir, + tile_bounds_json=args.tile_bounds_json, + original_input_dir=args.original_input_dir, + output_merged=args.output_merged, + buffer=args.buffer, + overlap_threshold=args.overlap_threshold, + max_centroid_distance=args.max_centroid_distance, + correspondence_tolerance=args.correspondence_tolerance, + max_volume_for_merge=args.max_volume_for_merge, + min_cluster_size=args.min_cluster_size, + num_threads=args.num_threads, + enable_matching=not args.disable_matching, + require_overlap=not args.disable_overlap_check, + enable_volume_merge=not args.disable_volume_merge, + skip_merged_file=args.skip_merged_file, + verbose=args.verbose, + border_zone_width=args.border_zone_width, + retile_buffer=args.retile_buffer, + retile_max_radius=args.retile_max_radius, + ) + if not args.skip_merged_file: + print(f"\nMerged output: {output_file}") + except Exception as e: + print(f"Error: {e}") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/src/main_remap.py b/src/main_remap.py new file mode 100644 index 0000000..5198488 --- /dev/null +++ b/src/main_remap.py @@ -0,0 +1,853 @@ +#!/usr/bin/env python3 +""" +Main remap script: Remap predictions by matching spatial bounds. + +This script handles remapping of segmented predictions from source files to target files +by matching files based on their spatial boundaries, then using KDTree nearest neighbor +lookup to transfer attributes. + +Usage: + python main_remap.py --source_folder /path/to/segmented --target_folder /path/to/res1 --output_folder /path/to/output +""" + +from __future__ import annotations + +import argparse +import os +import re +import sys +from pathlib import Path +from typing import Dict, List, Optional, Set, Tuple + +import numpy as np +import laspy +from concurrent.futures import ProcessPoolExecutor +from scipy.spatial import cKDTree +from laspy.vlrs.vlrlist import VLRList + +from instance_labels import ( + cast_instances_for_output, + instance_extra_bytes_params, + validate_prediction_instance_labels, +) +from point_cloud_metadata import extra_bytes_params_from_dimension_info, point_cloud_files +from tile_bounds_graph import ( + build_neighbor_graph_from_bounds_json, + match_tiles_to_json_bounds, +) +from worker_budget import kdtree_query_workers + + +def _strip_copc_records_for_laspy_write(header: laspy.LasHeader) -> None: + """Remove COPC hierarchy records before writing a regular LAZ with laspy.""" + def is_copc_record(vlr) -> bool: + return getattr(vlr, "user_id", "") == "copc" + + header.vlrs = VLRList([ + vlr for vlr in (getattr(header, "vlrs", []) or []) + if not is_copc_record(vlr) + ]) + + evlrs = getattr(header, "evlrs", None) + if evlrs is not None: + header.evlrs = VLRList([ + vlr for vlr in evlrs + if not is_copc_record(vlr) + ]) + + +def get_file_bounds(filepath: Path) -> Optional[Tuple[float, float, float, float]]: + """ + Get spatial bounds of a point cloud file using laspy header only (no point loading). + + Args: + filepath: Path to LAZ file + + Returns: + Tuple of (minx, maxx, miny, maxy) or None on error + """ + try: + # Use laspy.open() to read only header, not all points + with laspy.open(str(filepath), laz_backend=laspy.LazBackend.LazrsParallel) as las: + return (las.header.x_min, las.header.x_max, las.header.y_min, las.header.y_max) + except Exception: + return None + + +def _remap_point_cloud_files(directory: Path) -> List[Path]: + """Return LAS/LAZ inputs for remap matching, preferring COPC twins.""" + return point_cloud_files(directory) + + +def calculate_bounds_overlap( + bounds1: Tuple[float, float, float, float], + bounds2: Tuple[float, float, float, float] +) -> float: + """ + Calculate IoU (Intersection over Union) between two bounding boxes. + + IoU is more robust than "overlap % of smaller file" because: + - It's symmetric + - Penalizes size mismatches + - Prevents nested files from all getting 100% + + Args: + bounds1: Tuple of (minx, maxx, miny, maxy) for first file + bounds2: Tuple of (minx, maxx, miny, maxy) for second file + + Returns: + IoU as percentage (0-100), where 100% = perfect overlap + """ + if bounds1 is None or bounds2 is None: + return 0.0 + + minx1, maxx1, miny1, maxy1 = bounds1 + minx2, maxx2, miny2, maxy2 = bounds2 + + # Calculate intersection region + overlap_minx = max(minx1, minx2) + overlap_maxx = min(maxx1, maxx2) + overlap_miny = max(miny1, miny2) + overlap_maxy = min(maxy1, maxy2) + + # Check if there's actual overlap + if overlap_minx >= overlap_maxx or overlap_miny >= overlap_maxy: + return 0.0 + + # Calculate intersection area + intersection = (overlap_maxx - overlap_minx) * (overlap_maxy - overlap_miny) + + # Calculate union area + area1 = (maxx1 - minx1) * (maxy1 - miny1) + area2 = (maxx2 - minx2) * (maxy2 - miny2) + union = area1 + area2 - intersection + + # Calculate IoU as percentage + iou = (intersection / union) * 100 if union > 0 else 0.0 + + return iou + + +def bounds_match_exact( + bounds1: Tuple[float, float, float, float], + bounds2: Tuple[float, float, float, float], + precision: float = 0.01 +) -> bool: + """ + Check if two bounds are identical (within floating point precision). + + Args: + bounds1: Tuple of (minx, maxx, miny, maxy) for first file + bounds2: Tuple of (minx, maxx, miny, maxy) for second file + precision: Maximum difference for exact match (default: 0.01m = 1cm) + + Returns: + True if bounds are essentially identical + """ + if bounds1 is None or bounds2 is None: + return False + + minx1, maxx1, miny1, maxy1 = bounds1 + minx2, maxx2, miny2, maxy2 = bounds2 + + return (abs(minx1 - minx2) <= precision and + abs(maxx1 - maxx2) <= precision and + abs(miny1 - miny2) <= precision and + abs(maxy1 - maxy2) <= precision) + + +def bounds_match_tolerance( + bounds1: Tuple[float, float, float, float], + bounds2: Tuple[float, float, float, float], + tolerance: float = 1.0 +) -> bool: + """ + Check if two bounds match within a strict tolerance. + + Args: + bounds1: Tuple of (minx, maxx, miny, maxy) for first file + bounds2: Tuple of (minx, maxx, miny, maxy) for second file + tolerance: Maximum difference in meters for each bound component (default: 1.0m) + + Returns: + True if all bound components match within tolerance + """ + if bounds1 is None or bounds2 is None: + return False + + minx1, maxx1, miny1, maxy1 = bounds1 + minx2, maxx2, miny2, maxy2 = bounds2 + + return (abs(minx1 - minx2) <= tolerance and + abs(maxx1 - maxx2) <= tolerance and + abs(miny1 - miny2) <= tolerance and + abs(maxy1 - maxy2) <= tolerance) + + +def bounds_match( + bounds1: Tuple[float, float, float, float], + bounds2: Tuple[float, float, float, float] +) -> bool: + """ + Check if two bounds match using two-stage approach: + 1. First try strict tolerance matching (1m) for exact matches + 2. If that fails, fallback to IoU matching (30%) for robust matching + + IoU threshold of 30% works for: + - Dataset 257 (near-identical bounds): ~98% IoU ✓ + - GFZ correct matches (partial overlap): ~33% IoU ✓ + - GFZ wrong matches (minimal overlap): ~5% IoU ✗ + + Args: + bounds1: Tuple of (minx, maxx, miny, maxy) for first file + bounds2: Tuple of (minx, maxx, miny, maxy) for second file + + Returns: + True if bounds match by either method + """ + if bounds1 is None or bounds2 is None: + return False + + # Stage 1: Try strict tolerance matching (1m) + if bounds_match_tolerance(bounds1, bounds2, tolerance=1.0): + return True + + # Stage 2: Fallback to IoU matching (30%) + iou = calculate_bounds_overlap(bounds1, bounds2) + return iou >= 30.0 + + +def remap_single_tile( + segmented_file: Path, + target_file: Path, + output_file: Path, + threedtrees_dims: Optional[Set[str]] = None, + threedtrees_suffix: str = "SAT", + instance_dimension: str = "PredInstance", + output_scales: Optional[Tuple[float, float, float]] = None, + kdtree_workers: int = 1, +) -> Tuple[str, bool, str, int]: + """ + Remap predictions from segmented file to target resolution file. + + Uses KDTree nearest neighbor search to transfer attributes from + the segmented (coarse) file to the target (fine) file. + + Args: + segmented_file: Path to segmented LAZ file (e.g., 10cm with predictions) + target_file: Path to target resolution LAZ file (e.g., resolution-1 subsample) + output_file: Path for output LAZ file + threedtrees_dims: If set, only transfer these dims, renamed to {name}_{suffix} + threedtrees_suffix: Suffix for branding (default: "SAT") + + Returns: + Tuple of (tile_id, success, message, point_count) + """ + tile_id = segmented_file.stem.replace('_segmented', '').replace('_results', '') + + try: + # Load segmented point cloud (source of predictions) + print(f" Loading segmented file...") + segmented_las = laspy.read( + str(segmented_file), + laz_backend=laspy.LazBackend.LazrsParallel + ) + segmented_points = np.vstack(( + segmented_las.x, + segmented_las.y, + segmented_las.z + )).T + print(f" Segmented file: {len(segmented_points):,} points") + if hasattr(segmented_las, instance_dimension): + validate_prediction_instance_labels( + getattr(segmented_las, instance_dimension), + instance_dimension, + segmented_file, + ) + + # Create KDTree from segmented points with progress indication + print(f" Building KDTree from {len(segmented_points):,} points...", end="", flush=True) + tree = cKDTree(segmented_points) + print(" ✓") + + source_extra_dims = list(segmented_las.point_format.extra_dimensions) + with laspy.open(str(target_file), laz_backend=laspy.LazBackend.LazrsParallel) as target_reader: + output_header = target_reader.header.copy() + target_point_count = target_reader.header.point_count + + target_extra_dim_names = set(output_header.point_format.dimension_names) + + if len(source_extra_dims) == 0: + print(f" Warning: No extra dimensions found in segmented file") + + # Resolve names and collect params for batch add + dims_to_add = [] # (extra_params, source_dim_name, cast_as_instance) + for dim_info in source_extra_dims: + dim_name = dim_info.name + cast_as_instance = dim_name == instance_dimension + # If branding is active, only transfer 3DTrees dims with branded names + if threedtrees_dims is not None: + if dim_name not in threedtrees_dims: + continue + out_name = f"{dim_name}_{threedtrees_suffix}" if threedtrees_suffix else dim_name + cast_as_instance = False + else: + # No branding — use collision-safe naming + out_name = dim_name + if out_name in target_extra_dim_names: + suffix = 1 + while f"{dim_name}_{suffix}" in target_extra_dim_names: + suffix += 1 + out_name = f"{dim_name}_{suffix}" + extra_params = ( + instance_extra_bytes_params(out_name, getattr(segmented_las, dim_name)) + if cast_as_instance + else extra_bytes_params_from_dimension_info(dim_info, name=out_name) + ) + dims_to_add.append((extra_params, dim_name, cast_as_instance)) + target_extra_dim_names.add(out_name) + + if output_scales is not None: + output_header.scales = np.asarray(output_scales, dtype=np.float64) + _strip_copc_records_for_laspy_write(output_header) + if dims_to_add: + output_header.add_extra_dims([params for params, _, _ in dims_to_add]) + + source_dim_values = { + src_name: getattr(segmented_las, src_name) + for _, src_name, _ in dims_to_add + } + + def copy_target_chunk( + target_chunk: laspy.ScaleAwarePointRecord, + ) -> laspy.ScaleAwarePointRecord: + out_chunk = laspy.ScaleAwarePointRecord.zeros(len(target_chunk), header=output_header) + out_chunk.x = np.asarray(target_chunk.x) + out_chunk.y = np.asarray(target_chunk.y) + out_chunk.z = np.asarray(target_chunk.z) + for name in target_chunk.point_format.dimension_names: + if name in {"X", "Y", "Z"}: + continue + if name in out_chunk.point_format.dimension_names: + out_chunk[name] = target_chunk[name] + return out_chunk + + # Create output directory if needed + output_file.parent.mkdir(parents=True, exist_ok=True) + + chunk_size = int(os.environ.get("SMARTTILE_REMAP_CHUNK_SIZE", "5000000")) + processed = 0 + print( + f" Streaming nearest-neighbor remap for {target_point_count:,} target points " + f"in chunks of {chunk_size:,}...", + flush=True, + ) + + with laspy.open(str(target_file), laz_backend=laspy.LazBackend.LazrsParallel) as target_reader: + with laspy.open( + str(output_file), + mode="w", + header=output_header, + laz_backend=laspy.LazBackend.LazrsParallel, + ) as writer: + for target_chunk in target_reader.chunk_iterator(chunk_size): + target_points = np.vstack(( + target_chunk.x, + target_chunk.y, + target_chunk.z, + )).T + _, indices = tree.query(target_points, workers=kdtree_workers) + out_chunk = copy_target_chunk(target_chunk) + for params, src_name, cast_as_instance in dims_to_add: + values = source_dim_values[src_name][indices] + if cast_as_instance: + values = cast_instances_for_output(values, src_name) + out_chunk[params.name] = values + writer.write_points(out_chunk) + processed += len(target_chunk) + if processed % 25_000_000 < len(target_chunk) or processed == target_point_count: + print( + f" Remapped {processed:,}/{target_point_count:,} points", + flush=True, + ) + + return (tile_id, True, "Success", target_point_count) + + except Exception as e: + return (tile_id, False, str(e), 0) + + +def _match_files_via_json( + tile_bounds_json: Path, + source_folder: Path, + target_folder: Path, + verbose: bool = False, +) -> List[Tuple[Path, Path, str]]: + """ + Match source and target files using tile_bounds_tindex.json. + Both source and target files are matched to JSON entries (stepwise bounds/centroid); + pairs are formed by shared JSON index. Uses the same stable matching as merge. + """ + source_files = _remap_point_cloud_files(source_folder) + target_files = _remap_point_cloud_files(target_folder) + if not source_files or not target_files: + return [] + + # Single-file shortcut: 1 source and 1 target -> pair directly + if len(source_files) == 1 and len(target_files) == 1: + src = source_files[0] + tgt = target_files[0] + tile_id = re.sub(r"_segmented$|_results$|_subsampled[\d.]+m$", "", src.stem) + if not tile_id: + tile_id = src.stem + print(f" Single file pair: {src.name} <-> {tgt.name}") + return [(src, tgt, tile_id)] + + json_bounds, centers, _ = build_neighbor_graph_from_bounds_json(tile_bounds_json) + + source_boundaries: Dict[str, Tuple[float, float, float, float]] = {} + stem_to_source_path: Dict[str, Path] = {} + for f in source_files: + b = get_file_bounds(f) + if b is not None: + stem = f.stem + source_boundaries[stem] = b + stem_to_source_path[stem] = f + + target_boundaries: Dict[str, Tuple[float, float, float, float]] = {} + stem_to_target_path: Dict[str, Path] = {} + for f in target_files: + b = get_file_bounds(f) + if b is not None: + stem = f.stem + target_boundaries[stem] = b + stem_to_target_path[stem] = f + + if not source_boundaries: + raise ValueError("Could not read bounds from any source file") + if not target_boundaries: + raise ValueError("Could not read bounds from any target file") + + source_to_json, json_to_source = match_tiles_to_json_bounds( + source_boundaries, json_bounds, centers + ) + target_to_json, json_to_target = match_tiles_to_json_bounds( + target_boundaries, json_bounds, centers + ) + + matches: List[Tuple[Path, Path, str]] = [] + for j in range(len(json_bounds)): + src_stem = json_to_source.get(j) + tgt_stem = json_to_target.get(j) + if src_stem is None or tgt_stem is None: + if src_stem is not None: + raise ValueError( + f"Source file {stem_to_source_path[src_stem].name} matched JSON tile index {j} " + "but no target file matched that tile. Cannot remap." + ) + continue + src_path = stem_to_source_path[src_stem] + tgt_path = stem_to_target_path[tgt_stem] + tile_id = re.sub(r"_segmented$|_results$|_subsampled[\d.]+m$", "", src_stem) + if not tile_id: + tile_id = src_stem + matches.append((src_path, tgt_path, tile_id)) + if verbose: + print(f" ✓ Matched (JSON): {src_path.name} <-> {tgt_path.name} ({tile_id})") + + return matches + + +def find_matching_files( + source_folder: Path, + target_folder: Path, + overlap_threshold: float = 99.0, + verbose: bool = False, + tile_bounds_json: Optional[Path] = None, +) -> List[Tuple[Path, Path, str]]: + """ + Find matching files between source and target folders. + If tile_bounds_json is provided and exists, uses JSON-based matching (stepwise + bounds/centroid, same as merge). Otherwise uses two-stage matching: + 1. Strict tolerance matching (1m) + 2. IoU matching (30%) fallback + + Args: + source_folder: Directory containing source LAZ files (e.g., segmented files) + target_folder: Directory containing target LAZ files (e.g., resolution-1 subsampled files) + overlap_threshold: DEPRECATED - not used (kept for compatibility) + verbose: If True, print detailed matching diagnostics + tile_bounds_json: Optional path to tile_bounds_tindex.json for grid-based matching + + Returns: + List of (source_file, target_file, tile_id) tuples + """ + if tile_bounds_json is not None and tile_bounds_json.exists(): + print(f" Using tile_bounds_tindex.json for matching: {tile_bounds_json}") + return _match_files_via_json(tile_bounds_json, source_folder, target_folder, verbose) + + matches = [] + + # Get all LAZ/LAS files from both folders (flat structure) + source_files = _remap_point_cloud_files(source_folder) + target_files = _remap_point_cloud_files(target_folder) + + if not source_files: + print(f" Warning: No LAZ/LAS files found in source folder: {source_folder}") + return matches + + if not target_files: + print(f" Warning: No LAZ/LAS files found in target folder: {target_folder}") + return matches + + print(f" Found {len(source_files)} source files and {len(target_files)} target files") + print(f" Two-stage matching: 1m tolerance → 30% IoU fallback") + + # Extract bounds for all target files once + target_bounds_map = {} + for target_file in target_files: + bounds = get_file_bounds(target_file) + if bounds: + target_bounds_map[target_file] = bounds + + # Match each source file to target files using two-stage approach + unmatched_count = 0 + tolerance_matches = 0 + iou_matches = 0 + + for source_file in source_files: + source_bounds = get_file_bounds(source_file) + if source_bounds is None: + print(f" Warning: Could not extract bounds from {source_file.name}") + continue + + # Find matching target file(s) and track overlap for each + matched_targets = [] + + for target_file, target_bounds in target_bounds_map.items(): + # Check both matching methods + tolerance_match = bounds_match_tolerance(source_bounds, target_bounds, tolerance=1.0) + iou = calculate_bounds_overlap(source_bounds, target_bounds) + iou_match = iou >= 30.0 + + # Determine which method succeeded (priority: tolerance > iou) + if tolerance_match: + match_method = 'tolerance' + elif iou_match: + match_method = 'iou' + else: + continue # No match + + matched_targets.append((target_file, match_method, iou)) + + if len(matched_targets) == 0: + unmatched_count += 1 + print(f" ⚠ Warning: No matching target file found for {source_file.name}") + + # Provide helpful diagnostics for best candidate + if verbose: + best_iou = 0.0 + best_target = None + for target_file, target_bounds in target_bounds_map.items(): + iou = calculate_bounds_overlap(source_bounds, target_bounds) + if iou > best_iou: + best_iou = iou + best_target = target_file + if best_target: + print(f" Best candidate: {best_target.name} (IoU: {best_iou:.2f}%)") + + continue + + # Sort by match quality (method priority, then IoU) + def match_score(match_tuple): + target_file, match_method, iou = match_tuple + method_priority = {'tolerance': 2, 'iou': 1} + return (method_priority[match_method], iou) + + matched_targets.sort(key=match_score, reverse=True) + + # Check for ambiguous matches (same method and IoU) + if len(matched_targets) > 1: + best_method = matched_targets[0][1] + best_iou = matched_targets[0][2] + + # Count how many have the same score as the best + ambiguous_matches = [ + m for m in matched_targets + if m[1] == best_method and abs(m[2] - best_iou) < 0.01 + ] + + if len(ambiguous_matches) > 1: + # Ambiguous match - cannot determine correct target + ambiguous_names = [m[0].name for m in ambiguous_matches] + raise ValueError( + f"Ambiguous match for {source_file.name}: " + f"Multiple targets with identical bounds and IoU ({best_iou:.2f}%): " + f"{', '.join(ambiguous_names)}. " + f"Cannot determine correct target file." + ) + + target_file, match_method, best_iou = matched_targets[0] + + # Update counters + if match_method == 'tolerance': + tolerance_matches += 1 + else: + iou_matches += 1 + + # Extract tile_id from filename if possible, otherwise use stem + tile_id_match = re.search(r'(c\d+_r\d+)', source_file.stem) + if tile_id_match: + tile_id = tile_id_match.group(1) + else: + # Fallback: use filename stem without extension + tile_id = source_file.stem.replace('_segmented', '').replace('_results', '') + + matches.append((source_file, target_file, tile_id)) + + if verbose: + if match_method == 'tolerance': + method_str = "1m tolerance" + else: + method_str = f"IoU {best_iou:.1f}%" + print(f" ✓ Matched: {source_file.name} <-> {target_file.name} ({method_str})") + else: + print(f" Matched: {source_file.name} <-> {target_file.name}") + + # Summary + print() + if tolerance_matches > 0: + print(f" ✓ {tolerance_matches} file(s) matched by tolerance (1m)") + if iou_matches > 0: + print(f" ✓ {iou_matches} file(s) matched by IoU fallback (30%)") + if unmatched_count > 0: + print(f" ⚠ {unmatched_count} file(s) could not be matched") + + return matches + + +def _remap_worker_item(item): + """Unpack work item and call remap_single_tile; must be at module level for ProcessPoolExecutor pickle.""" + src, tgt, out, _tid, instance_dimension, output_scales, kdtree_workers = item + return remap_single_tile( + src, + tgt, + out, + instance_dimension=instance_dimension, + output_scales=output_scales, + kdtree_workers=kdtree_workers, + ) + + +def remap_all_tiles( + source_folder: Path, + target_folder: Path, + output_folder: Path, + overlap_threshold: float = 99.0, + verbose: bool = False, + tile_bounds_json: Optional[Path] = None, + num_workers: int = 4, + instance_dimension: str = "PredInstance", + output_scales: Optional[Tuple[float, float, float]] = None, +) -> Path: + """ + Remap predictions from source files to target files for all tiles. + + Matches files between source and target folders by spatial overlap. + If tile_bounds_json is provided, uses JSON-based matching (same grid as merge). + + Args: + source_folder: Path to folder containing source LAZ files (e.g., segmented files) + target_folder: Path to folder containing target LAZ files (e.g., resolution-1 subsampled files) + output_folder: Output folder for remapped files + overlap_threshold: Minimum spatial overlap percentage required (default: 99.0%) + verbose: If True, print detailed matching diagnostics + tile_bounds_json: Optional path to tile_bounds_tindex.json for grid-based matching + num_workers: Number of parallel processes (default: 4); use parameters.workers in run.py + + Returns: + Path to output folder + """ + print("=" * 60) + print("3DTrees Remap Pipeline") + print("=" * 60) + print(f"Source folder: {source_folder}") + print(f"Target folder: {target_folder}") + print(f"Output folder: {output_folder}") + if tile_bounds_json and tile_bounds_json.exists(): + print(f"Matching: tile_bounds_tindex.json ({tile_bounds_json})") + else: + print("Matching: Two-stage (1m tolerance → 30% IoU)") + print() + + # Validate directories exist + if not source_folder.exists(): + raise ValueError(f"Source directory not found: {source_folder}") + + if not target_folder.exists(): + raise ValueError(f"Target directory not found: {target_folder}") + + # Create output directory + output_folder.mkdir(parents=True, exist_ok=True) + + # Find matching files by spatial overlap (or JSON when provided) + print("Matching files by spatial bounds...") + matches = find_matching_files( + source_folder, target_folder, overlap_threshold, verbose, tile_bounds_json + ) + + if not matches: + n_src = len(list(source_folder.glob("*.laz")) + list(source_folder.glob("*.las"))) + n_tgt = len(list(target_folder.glob("*.laz")) + list(target_folder.glob("*.las"))) + msg = ( + "No matching source/target file pairs found. " + f"Source folder has {n_src} LAZ/LAS file(s), target folder has {n_tgt}. " + "With tile_bounds_json, both folders must contain one file per tile (same grid); " + "file bounds are matched to the JSON tile bounds." + ) + raise ValueError(msg) + + print(f"Found {len(matches)} matching file pairs") + print() + + # Build work items, skipping already-processed tiles + successful = 0 + failed = 0 + total_points = 0 + work_items = [] + + for source_file, target_file, tile_id in matches: + output_file = output_folder / f"{tile_id}_segmented_remapped.laz" + if output_file.exists() and output_file.stat().st_size > 0: + successful += 1 + continue + work_items.append((source_file, target_file, output_file, tile_id, instance_dimension, output_scales)) + + if successful > 0: + print(f" Skipping {successful} already processed tiles") + + if len(work_items) == 0: + print(f" All tiles already processed!") + else: + n_procs = min(max(1, num_workers), len(work_items)) + query_workers = kdtree_query_workers(num_workers, n_procs) + work_items = [(*item, query_workers) for item in work_items] + print( + f" Processing {len(work_items)} tiles with {n_procs} workers; " + f"{query_workers} KDTree query worker(s) each..." + ) + + with ProcessPoolExecutor(max_workers=n_procs) as executor: + for i, result in enumerate(executor.map(_remap_worker_item, work_items)): + tile_id_result, success, message, point_count = result + tile_id = work_items[i][3] + if success: + successful += 1 + total_points += point_count + print(f" [{i+1}/{len(work_items)}] ✓ {tile_id}: {point_count:,} points") + else: + failed += 1 + print(f" [{i+1}/{len(work_items)}] ✗ {tile_id}: {message}") + + # Summary + print() + print("=" * 60) + print("Remap Pipeline Complete") + print("=" * 60) + print(f" Successful: {successful}") + print(f" Failed: {failed}") + print(f" Total points: {total_points:,}") + print(f" Output: {output_folder}") + + return output_folder + + +def main(): + """CLI entry point.""" + parser = argparse.ArgumentParser( + description="3DTrees Remap Pipeline - Remap predictions by matching spatial bounds", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Remap files matching by bounds + python main_remap.py --source_folder /path/to/segmented --target_folder /path/to/res1 --output_folder /path/to/output + + # Remap with custom tolerance + python main_remap.py --source_folder /path/to/segmented --target_folder /path/to/res1 --output_folder /path/to/output --tolerance 10.0 + """ + ) + + parser.add_argument( + "--source_folder", + type=Path, + required=True, + help="Path to folder containing source LAZ files (e.g., segmented files with predictions)" + ) + + parser.add_argument( + "--target_folder", + type=Path, + required=True, + help="Path to folder containing target LAZ files (e.g., resolution-1 subsampled files)" + ) + + parser.add_argument( + "--output_folder", + type=Path, + required=True, + help="Output folder for remapped files" + ) + + parser.add_argument( + "--tolerance", + type=float, + default=5.0, + help="Maximum difference in meters for bounds matching when not using --tile_bounds_json (default: 5.0)" + ) + parser.add_argument( + "--tile_bounds_json", + type=Path, + default=None, + help="Path to tile_bounds_tindex.json for grid-based matching (same as merge); optional" + ) + parser.add_argument( + "--verbose", + action="store_true", + help="Print detailed matching diagnostics" + ) + parser.add_argument( + "--workers", + type=int, + default=4, + help="Number of parallel processes (default: 4)" + ) + parser.add_argument( + "--instance_dimension", + "--instance-dimension", + type=str, + default="PredInstance", + help="Name of the instance ID dimension to persist as uint16, or uint32 above 65,535 (default: PredInstance)" + ) + + args = parser.parse_args() + + # Run pipeline + try: + output_folder = remap_all_tiles( + source_folder=args.source_folder, + target_folder=args.target_folder, + output_folder=args.output_folder, + overlap_threshold=99.0, + verbose=args.verbose, + tile_bounds_json=args.tile_bounds_json, + num_workers=args.workers, + instance_dimension=args.instance_dimension, + ) + print(f"\nRemapped files ready: {output_folder}") + except Exception as e: + print(f"Error: {e}") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/src/main_subsample.py b/src/main_subsample.py index 9524680..9a1d780 100644 --- a/src/main_subsample.py +++ b/src/main_subsample.py @@ -1,39 +1,72 @@ #!/usr/bin/env python3 """ -Main subsampling script: Parallel subsampling to resolution 1 (2cm) and resolution 2 (10cm). +Main subsampling script: Parallel subsampling to resolution 1 (1cm) and resolution 2 (10cm). This script handles subsampling of tiled point clouds: -1. Subsample tiles to resolution 1 (default: 2cm) +1. Subsample tiles to resolution 1 (default: 1cm) 2. Subsample resolution 1 files to resolution 2 (default: 10cm) Files are split across available CPU cores for parallel processing. COPC Optimizations: - Uses COPC native bounds filtering in readers.copc (more efficient than filters.crop) -- Writes resolution-1 outputs as COPC by default (better performance for subsequent steps) +- Writes output as COPC format when input is COPC (better performance for subsequent steps) - Leverages COPC's spatial indexing for efficient chunk-based processing - Multi-threaded COPC writing for improved performance Usage: - python main_subsample.py --tiles_dir /path/to/tiles --res1 0.02 --res2 0.1 + python main_subsample.py --tiles_dir /path/to/tiles --res1 0.01 --res2 0.1 """ from __future__ import annotations import argparse import json +import math import os +import shutil import subprocess import sys from concurrent.futures import ProcessPoolExecutor, as_completed from pathlib import Path -from typing import Dict, List, Optional, Tuple +from typing import List, Optional, Tuple + +import laspy # Import parameters from parameters import TILE_PARAMS - - -COPC_CHUNK_TIMEOUT_SECONDS = 120 +from point_cloud_metadata import point_cloud_files +from subsample_chunk_worker import subsample_tile_chunk +from subsample_com import ( + COPC_COM_DENSE_BIN_LIMIT, + COPC_COM_MAX_WINDOW_SIZE, + COPC_COM_TARGET_WINDOW_CELLS, + aggregate_center_of_mass_xyz as _aggregate_center_of_mass_xyz, + aligned_edges as _aligned_edges, + center_of_mass_subsample_copc, + center_of_mass_subsample_las, + iter_copc_center_of_mass_windows as _iter_copc_center_of_mass_windows, + process_pool_kwargs as _process_pool_kwargs, + write_center_of_mass_points as _write_center_of_mass_points, +) +from subsample_methods import ( + SUBSAMPLING_METHOD_CENTER_OF_MASS, + SUBSAMPLING_METHOD_NEAREST_TO_CENTROID, + SUBSAMPLING_METHODS, + is_copc_file as _is_copc_file, + normalize_subsampling_method, + voxel_subsampling_filter as _voxel_subsampling_filter, +) +from subsample_outputs import ( + convert_laz_output_to_copc, + copc_output_path, + laz_output_path, + subsample_output_files, + temporary_laz_path, +) + + +SUBSAMPLE_MANIFEST_FILENAME = ".smarttile_subsample_manifest.json" def get_pdal_path() -> str: @@ -44,15 +77,6 @@ def get_pdal_path() -> str: return pdal_path if pdal_path else "pdal" -def get_untwine_path(require: bool = True) -> str: - """Get the path to untwine executable.""" - import shutil - untwine_path = shutil.which("untwine") - if not untwine_path and require: - raise RuntimeError("untwine is required for COPC output but was not found in PATH") - return untwine_path - - def get_cpu_count() -> int: """Get available CPU count.""" try: @@ -61,506 +85,226 @@ def get_cpu_count() -> int: return 4 -def get_file_bounds(filepath: Path) -> Optional[Tuple[float, float, float, float]]: - """ - Get spatial bounds of a point cloud file using pdal info. - - Returns: - Tuple of (minx, maxx, miny, maxy) or None on error - """ - try: - pdal_cmd = get_pdal_path() - result = subprocess.run( - [pdal_cmd, "info", "--metadata", str(filepath)], - capture_output=True, - text=True, - check=True - ) - - import re - minx = float(re.search(r'"minx":\s*([\d.-]+)', result.stdout).group(1)) - maxx = float(re.search(r'"maxx":\s*([\d.-]+)', result.stdout).group(1)) - miny = float(re.search(r'"miny":\s*([\d.-]+)', result.stdout).group(1)) - maxy = float(re.search(r'"maxy":\s*([\d.-]+)', result.stdout).group(1)) - - return (minx, maxx, miny, maxy) - except Exception: - return None - - -def get_laspy_laz_backend(): - """Return the preferred laspy LAZ backend when available.""" - try: - import laspy - - if hasattr(laspy.LazBackend, "LazrsParallel"): - return laspy.LazBackend.LazrsParallel - if hasattr(laspy.LazBackend, "Lazrs"): - return laspy.LazBackend.Lazrs - except Exception: - pass - return None - - -def build_las_writer_options( - output_file: Path, - dimension_reduction: bool, - compression: Optional[bool] = None, -) -> Dict[str, object]: - """Build writers.las options while respecting dimension reduction.""" - writer_opts: Dict[str, object] = { - "type": "writers.las", - "filename": str(output_file), - } - if compression is not None: - writer_opts["compression"] = compression - if dimension_reduction: - writer_opts["minor_version"] = 2 - writer_opts["dataformat_id"] = 0 - else: - writer_opts["minor_version"] = 4 - writer_opts["extra_dims"] = "all" - return writer_opts - - -def make_laspy_output_header(source_header): - """Create a writable laspy header from a source header without COPC VLRs.""" - import laspy - - header = laspy.LasHeader( - point_format=source_header.point_format, - version=source_header.version, - ) - header.offsets = source_header.offsets - header.scales = source_header.scales - - existing_vlrs = {(getattr(vlr, "user_id", ""), vlr.record_id) for vlr in header.vlrs} - for vlr in source_header.vlrs: - if vlr.record_id in (1, 2) and getattr(vlr, "user_id", "") == "copc": - continue - vlr_key = (getattr(vlr, "user_id", ""), vlr.record_id) - if vlr_key not in existing_vlrs: - header.vlrs.append(vlr) - existing_vlrs.add(vlr_key) - - return header - - -def parse_bounds(bounds_str: str) -> Optional[Tuple[float, float, float, float]]: - """Parse the PDAL 2D bounds string created by this module.""" - import re - - match = re.fullmatch( - r"\(\s*\[\s*([-+0-9.eE]+)\s*,\s*([-+0-9.eE]+)\s*\]\s*,\s*" - r"\[\s*([-+0-9.eE]+)\s*,\s*([-+0-9.eE]+)\s*\]\s*\)", - bounds_str.strip(), - ) - if not match: - return None - minx, maxx, miny, maxy = (float(value) for value in match.groups()) - return (minx, maxx, miny, maxy) - - -def safe_unlink(path: Path) -> None: - """Best-effort unlink for temporary or partial outputs.""" - try: - if path.exists(): - path.unlink() - except Exception: - pass - - -def run_pdal_pipeline( - pipeline: Dict[str, object], - pipeline_file: Path, - timeout_seconds: Optional[int] = None, -) -> Tuple[int, str, str, bool]: - """Run a PDAL pipeline and report whether it timed out.""" - with open(pipeline_file, "w") as f: - json.dump(pipeline, f, indent=2) - - pdal_cmd = get_pdal_path() - try: - result = subprocess.run( - [pdal_cmd, "pipeline", str(pipeline_file)], - capture_output=True, - text=True, - check=False, - timeout=timeout_seconds if timeout_seconds and timeout_seconds > 0 else None, - ) - return result.returncode, result.stdout or "", result.stderr or "", False - except subprocess.TimeoutExpired as e: - stdout = e.stdout or "" - stderr = e.stderr or "" - if isinstance(stdout, bytes): - stdout = stdout.decode(errors="replace") - if isinstance(stderr, bytes): - stderr = stderr.decode(errors="replace") - if not stderr: - stderr = f"PDAL pipeline timed out after {timeout_seconds}s" - return 124, stdout, stderr, True - finally: - if pipeline_file.exists(): - pipeline_file.unlink() +def _subsample_input_files(input_dir: Path) -> List[Path]: + """Return point-cloud files for subsampling, preferring COPC twins when present.""" + return point_cloud_files(input_dir) -def summarize_pipeline_error(returncode: int, stdout: str, stderr: str) -> str: - """Return a compact subprocess error message.""" - message = (stderr or stdout or "").strip()[:200] - return message or f"no stderr/stdout (rc={returncode})" +def _subsample_manifest_path(output_dir: Path) -> Path: + return output_dir / SUBSAMPLE_MANIFEST_FILENAME -def count_points(filepath: Path) -> int: - """Count points in a LAS/LAZ file via pdal info.""" +def _read_subsample_manifest(output_dir: Path) -> dict: + manifest_file = _subsample_manifest_path(output_dir) + if not manifest_file.exists(): + return {"version": 1, "outputs": {}} try: - pdal_cmd = get_pdal_path() - info_result = subprocess.run( - [pdal_cmd, "info", "--metadata", str(filepath)], - capture_output=True, - text=True, - check=True, - ) - import re - - match = re.search(r'"count":\s*(\d+)', info_result.stdout) - if match: - return int(match.group(1)) + with manifest_file.open() as handle: + manifest = json.load(handle) except Exception: - pass - return 0 - - -def get_laspy_fallback_sources(input_file: Path, fallback_laz_dir: Optional[Path]) -> List[Path]: - """Prefer original LAS/LAZ inputs for fallback, excluding generated COPC files.""" - if fallback_laz_dir: - fallback_laz_dir = Path(fallback_laz_dir) - if fallback_laz_dir.is_file(): - return [fallback_laz_dir] - if fallback_laz_dir.is_dir(): - sources = sorted( - [ - *fallback_laz_dir.glob("*.las"), - *[ - path - for path in fallback_laz_dir.glob("*.laz") - if not path.name.endswith(".copc.laz") - ], - ] - ) - if sources: - return sources - return [input_file] + return {"version": 1, "outputs": {}} + if not isinstance(manifest, dict): + return {"version": 1, "outputs": {}} + manifest.setdefault("version", 1) + manifest.setdefault("outputs", {}) + return manifest -def crop_bounds_to_las_with_laspy_chunks( - input_files: List[Path], - bounds: Tuple[float, float, float, float], - output_file: Path, - chunk_size: int, -) -> Tuple[bool, int, str]: - """Stream source LAS/LAZ files and write points inside one failed chunk bound.""" - import laspy - import numpy as np - - minx, maxx, miny, maxy = bounds - laz_backend = get_laspy_laz_backend() - writer = None - written = 0 +def _write_subsample_manifest(output_dir: Path, manifest: dict) -> None: + output_dir.mkdir(parents=True, exist_ok=True) + with _subsample_manifest_path(output_dir).open("w") as handle: + json.dump(manifest, handle, indent=2, sort_keys=True) - try: - for input_file in input_files: - open_kwargs = {} - if input_file.suffix.lower() == ".laz" and laz_backend is not None: - open_kwargs["laz_backend"] = laz_backend - - with laspy.open(str(input_file), **open_kwargs) as reader: - for chunk in reader.chunk_iterator(max(1, int(chunk_size))): - if len(chunk) == 0: - continue - mask = ( - (np.asarray(chunk.x) >= minx) - & (np.asarray(chunk.x) <= maxx) - & (np.asarray(chunk.y) >= miny) - & (np.asarray(chunk.y) <= maxy) - ) - selected = int(np.count_nonzero(mask)) - if selected == 0: - continue - if writer is None: - output_header = make_laspy_output_header(reader.header) - writer = laspy.open( - str(output_file), - mode="w", - header=output_header, - do_compress=False, - ) - writer.write_points(chunk[mask]) - written += selected - except Exception as e: - if writer is not None: - try: - writer.close() - except Exception: - pass - safe_unlink(output_file) - return False, written, str(e) - finally: - if writer is not None: - try: - writer.close() - except Exception: - pass - if written == 0: - safe_unlink(output_file) - return True, written, "OK" +def _subsample_source_fingerprint(input_file: Path) -> dict: + stat = input_file.stat() + return { + "name": input_file.name, + "size": int(stat.st_size), + "mtime_ns": int(stat.st_mtime_ns), + } -def voxelize_las_part( +def _subsample_request_signature( input_file: Path, - output_file: Path, resolution: float, - pipeline_dir: Path, dimension_reduction: bool, -) -> Tuple[bool, str]: - """Apply the normal PDAL 1cm voxel reduction to one extracted fallback part.""" - safe_unlink(output_file) - writer_opts = build_las_writer_options(output_file, dimension_reduction) - pipeline = { - "pipeline": [ - {"type": "readers.las", "filename": str(input_file)}, - {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, - writer_opts, - ] + subsampling_method: str, + output_copc: bool, +) -> dict: + return { + "source": _subsample_source_fingerprint(input_file), + "resolution": float(resolution), + "dimension_reduction": bool(dimension_reduction), + "subsampling_method": normalize_subsampling_method(subsampling_method), + "output_copc": bool(output_copc), } - rc, stdout, stderr, _ = run_pdal_pipeline( - pipeline, - pipeline_dir / f"voxel_{output_file.stem}.json", - ) - if rc != 0: - return False, summarize_pipeline_error(rc, stdout, stderr) - return True, "OK" -def extract_chunk_with_laspy_fallback( - input_file: Path, - fallback_laz_dir: Optional[Path], - bounds_str: str, - resolution: float, +def _existing_subsample_is_reusable( + output_file: Path, + manifest: dict, + expected_signature: dict, +) -> bool: + """Return True when an existing subsample matches the current request.""" + if not output_file.exists() or output_file.stat().st_size == 0: + return False + entry = manifest.get("outputs", {}).get(output_file.name) + if not isinstance(entry, dict): + return False + return entry.get("signature") == expected_signature + + +def _record_subsample_output( output_dir: Path, - chunk_file: Path, - chunk_idx: int, - total_chunks: int, - dimension_reduction: bool, - chunk_size: int, -) -> Tuple[bool, int, str]: - """Fallback after COPC stalls: stream original LAZ bounds with laspy.""" - import shutil + output_file: Path, + signature: dict, + point_count: int, +) -> None: + manifest = _read_subsample_manifest(output_dir) + outputs = manifest.setdefault("outputs", {}) + outputs[output_file.name] = { + "signature": signature, + "point_count": int(point_count), + } + _write_subsample_manifest(output_dir, manifest) - bounds = parse_bounds(bounds_str) - if bounds is None: - return False, 0, f"Could not parse bounds: {bounds_str}" - fallback_sources = get_laspy_fallback_sources(input_file, fallback_laz_dir) - fallback_dir = output_dir / f"_chunk{chunk_idx}_laspy_fallback" - fallback_dir.mkdir(parents=True, exist_ok=True) - cropped_chunk = fallback_dir / "bounded_chunk.las" +def _parse_bounds_metadata(metadata) -> Optional[Tuple[float, float, float, float]]: + """Extract finite XY bounds from PDAL metadata.""" + if not isinstance(metadata, dict): + return None - try: - ok, selected_count, msg = crop_bounds_to_las_with_laspy_chunks( - fallback_sources, - bounds, - cropped_chunk, - chunk_size, - ) - if not ok: - return False, 0, msg - if selected_count == 0: - return False, 0, "laspy fallback produced no points inside failed chunk bounds" - - ok, msg = voxelize_las_part( - cropped_chunk, - chunk_file, - resolution, - fallback_dir, - dimension_reduction, - ) - safe_unlink(cropped_chunk) - if not ok: - return False, 0, msg - - if not chunk_file.exists() or chunk_file.stat().st_size == 0: - return False, 0, "laspy fallback produced empty chunk output" - - point_count = count_points(chunk_file) - print( - f" ✓ Chunk {chunk_idx}/{total_chunks}: laspy fallback produced " - f"{point_count:,} points", - flush=True, - ) - return True, point_count, "" - finally: + keys = ("minx", "maxx", "miny", "maxy") + if all(key in metadata for key in keys): try: - shutil.rmtree(fallback_dir) - except Exception: - pass + minx, maxx, miny, maxy = (float(metadata[key]) for key in keys) + except (TypeError, ValueError): + return None + if ( + all(math.isfinite(value) for value in (minx, maxx, miny, maxy)) + and maxx >= minx + and maxy >= miny + ): + return (minx, maxx, miny, maxy) + return None + + for value in metadata.values(): + if isinstance(value, dict): + bounds = _parse_bounds_metadata(value) + if bounds is not None: + return bounds + elif isinstance(value, list): + for item in value: + bounds = _parse_bounds_metadata(item) + if bounds is not None: + return bounds + return None -def subsample_tile_chunk( - args: Tuple[Path, str, float, Path, int, int, bool, Optional[Path], int] -) -> Tuple[int, Optional[Path], int, str]: +def get_file_bounds(filepath: Path) -> Optional[Tuple[float, float, float, float]]: """ - Subsample a spatial chunk of a tile using PDAL with COPC-optimized bounds filter. - - COPC optimizations: - - Uses bounds parameter directly in readers.copc (more efficient than filters.crop) - - Intermediate chunk output is plain LAS for simpler downstream reads - - Args: - args: Tuple of (input_file, bounds_str, resolution, output_dir, chunk_idx, total_chunks, dimension_reduction, fallback_laz_dir, chunk_size) - dimension_reduction: If True, write only standard dimensions (no extra_dims); minimal output. - + Get spatial bounds of a point cloud file using pdal info. + Returns: - Tuple of (chunk_idx, output_file_or_none, point_count, error_message) + Tuple of (minx, maxx, miny, maxy) or None on error """ - input_file, bounds_str, resolution, output_dir, chunk_idx, total_chunks, dimension_reduction, fallback_laz_dir, chunk_size = args - try: - # Determine reader type - can read COPC or LAS - is_copc = input_file.name.endswith('.copc.laz') - reader_type = "readers.copc" if is_copc else "readers.las" - - # Write intermediate chunks as plain LAS so merge/retry paths do not - # pay decompression overhead again. - chunk_file = output_dir / f"{input_file.stem}_chunk{chunk_idx}.las" - - # Build pipeline - use COPC bounds filtering if available - # When keeping all dims, do not set dataformat_id=0 (format 0 has no extra bytes); use LAS 1.4 for format 6/7. - writer_opts = build_las_writer_options(chunk_file, dimension_reduction) - - if is_copc: - # COPC: Use bounds parameter directly in reader (most efficient) - pipeline = { - "pipeline": [ - { - "type": reader_type, - "filename": str(input_file), - "bounds": bounds_str # COPC native bounds filtering - very efficient - }, - {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, - writer_opts, - ] - } - else: - # LAS: Use filters.crop as fallback (LAS readers don't support bounds parameter) - pipeline = { - "pipeline": [ - {"type": reader_type, "filename": str(input_file)}, - {"type": "filters.crop", "bounds": bounds_str}, - {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, - writer_opts, - ] - } - - # Write and execute pipeline - pipeline_file = output_dir / f"_pipeline_chunk{chunk_idx}.json" - safe_unlink(chunk_file) - returncode, stdout, stderr, timed_out = run_pdal_pipeline( - pipeline, - pipeline_file, - timeout_seconds=COPC_CHUNK_TIMEOUT_SECONDS if is_copc else None, + pdal_cmd = get_pdal_path() + result = subprocess.run( + [pdal_cmd, "info", "--metadata", str(filepath)], + capture_output=True, + text=True, + check=True ) - if returncode != 0: - if is_copc and timed_out: - print( - f" ⚠ Chunk {chunk_idx}/{total_chunks}: COPC read timed out after " - f"{COPC_CHUNK_TIMEOUT_SECONDS}s; falling back to laspy chunked LAZ reads", - flush=True, - ) - ok, point_count, msg = extract_chunk_with_laspy_fallback( - input_file, - fallback_laz_dir, - bounds_str, - resolution, - output_dir, - chunk_file, - chunk_idx, - total_chunks, - dimension_reduction, - chunk_size, - ) - if ok: - return (chunk_idx, chunk_file, point_count, "") - print(f" ⚠ Chunk {chunk_idx}/{total_chunks} laspy fallback error: {msg}") - return (chunk_idx, None, 0, msg) - - # Fallback: if COPC reader failed, bypass the COPC hierarchy and - # stream original LAZ/LAS sources with laspy. - if is_copc and ("copc" in stderr.lower() or "vlr" in stderr.lower()): - print(f" ⚠ Chunk {chunk_idx}/{total_chunks}: COPC reader failed, falling back to laspy chunked LAZ reads") - ok, point_count, msg = extract_chunk_with_laspy_fallback( - input_file, - fallback_laz_dir, - bounds_str, - resolution, - output_dir, - chunk_file, - chunk_idx, - total_chunks, - dimension_reduction, - chunk_size, - ) - if ok: - return (chunk_idx, chunk_file, point_count, "") - print(f" ⚠ Chunk {chunk_idx}/{total_chunks} laspy fallback error: {msg}") - return (chunk_idx, None, 0, msg) - else: - msg = summarize_pipeline_error(returncode, stdout, stderr) - print(f" ⚠ Chunk {chunk_idx}/{total_chunks} error (rc={returncode}): {msg}") - return (chunk_idx, None, 0, msg) + payload = json.loads(result.stdout) + return _parse_bounds_metadata(payload.get("metadata", payload)) + except Exception: + return None - if not chunk_file.exists() or chunk_file.stat().st_size == 0: - return (chunk_idx, None, 0, "empty output") - - # Get point count - point_count = count_points(chunk_file) - - print(f" ✓ Chunk {chunk_idx}/{total_chunks}: {point_count:,} points") - return (chunk_idx, chunk_file, point_count, "") - - except Exception as e: - print(f" ✗ Chunk {chunk_idx}/{total_chunks} failed: {e}") - return (chunk_idx, None, 0, str(e)) +def get_file_xy_scales(filepath: Path) -> Tuple[float, float]: + """Return LAS/COPC XY scales for half-open chunk bounds.""" + try: + with laspy.open(str(filepath), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + scales = reader.header.scales + return (float(scales[0]), float(scales[1])) + except Exception: + return (0.001, 0.001) -def subsample_single_file( - args: Tuple[Path, Path, float, Path, int, bool, bool, Optional[Path], int] -) -> Tuple[str, bool, str, int]: + +def subsample_single_file(args: Tuple[Path, Path, float, Path, int, bool, str, bool]) -> Tuple[str, bool, str, int]: """ Subsample a single file by splitting it into subtiles along X-axis and processing in parallel. - + Process: 1. Split tile into num_threads subtiles along X-axis only 2. Subsample each subtile in parallel using ProcessPoolExecutor (true CPU parallelism) 3. Merge all subsampled subtiles back together - + Args: - args: Tuple of (input_file, output_file, resolution, pipeline_dir, num_threads, dimension_reduction, output_copc, fallback_laz_dir, chunk_size) - + args: Tuple of (input_file, output_file, resolution, pipeline_dir, num_threads, dimension_reduction, subsampling_method, output_copc) + Returns: Tuple of (filename, success, message, point_count) """ - input_file, output_file, resolution, pipeline_dir, num_threads, dimension_reduction, output_copc, fallback_laz_dir, chunk_size = args + if len(args) == 7: + input_file, output_file, resolution, pipeline_dir, num_threads, dimension_reduction, subsampling_method = args + output_copc = False + else: + input_file, output_file, resolution, pipeline_dir, num_threads, dimension_reduction, subsampling_method, output_copc = args + subsampling_method = normalize_subsampling_method(subsampling_method) + target_output_file = copc_output_path(output_file) if output_copc else laz_output_path(output_file) try: print(f" → Processing {input_file.name}...") - + + if ( + subsampling_method == SUBSAMPLING_METHOD_CENTER_OF_MASS + and dimension_reduction + and _is_copc_file(input_file) + ): + print(" → Using COPC voxel-aligned center-of-mass windows") + work_output_file = ( + temporary_laz_path(target_output_file, pipeline_dir) + if output_copc + else target_output_file + ) + point_count = center_of_mass_subsample_copc( + input_file, + work_output_file, + resolution, + num_workers=num_threads, + ) + if output_copc: + print(f" → Converting {work_output_file.name} to COPC") + if not convert_laz_output_to_copc( + work_output_file, + target_output_file, + source_metadata_file=input_file, + ): + return (input_file.name, False, "COPC conversion failed", 0) + try: + work_output_file.unlink() + except OSError: + pass + print(f" ✓ {input_file.name}: {point_count:,} points") + return (input_file.name, True, "Success", point_count) + # Get file bounds bounds = get_file_bounds(input_file) if not bounds: # Fall back to simple single-pass subsampling when bounds cannot be determined - return subsample_simple(input_file, output_file, resolution, pipeline_dir, dimension_reduction, output_copc) - + return subsample_simple( + input_file, + output_file, + resolution, + pipeline_dir, + dimension_reduction, + subsampling_method, + output_copc, + ) + minx, maxx, miny, maxy = bounds # If the file has zero extent in X (or Y), splitting into X-chunks @@ -568,131 +312,114 @@ def subsample_single_file( # from the COPC reader. In that case, fall back to the simple # single-pass subsampling without spatial chunking. if maxx - minx == 0 or maxy - miny == 0: - return subsample_simple(input_file, output_file, resolution, pipeline_dir, dimension_reduction, output_copc) + return subsample_simple( + input_file, + output_file, + resolution, + pipeline_dir, + dimension_reduction, + subsampling_method, + output_copc, + ) - # Split into num_threads subtiles along X-axis only + # Split into num_threads subtiles along X-axis only. + # Align boundaries to the voxel grid and make non-final chunks half-open + # so one output voxel is never computed in two chunks. grid_x = num_threads grid_y = 1 - + # Calculate step size for X-axis - x_step = (maxx - minx) / grid_x - + raw_x_step = (maxx - minx) / grid_x + x_step = max(resolution, math.ceil(raw_x_step / resolution) * resolution) + scale_x, _ = get_file_xy_scales(input_file) + print(f" Splitting into {num_threads} subtiles along X-axis only ({grid_x}x{grid_y} grid)") - + # Create chunk tasks - exactly num_threads chunks chunk_dir = pipeline_dir / f"{input_file.stem}_chunks" chunk_dir.mkdir(parents=True, exist_ok=True) - + chunk_tasks = [] - for chunk_idx in range(num_threads): - chunk_minx = minx + chunk_idx * x_step - chunk_maxx = minx + (chunk_idx + 1) * x_step + chunk_edges = _aligned_edges(minx, maxx, x_step, resolution) + for chunk_idx, (chunk_minx, chunk_maxx) in enumerate(chunk_edges): # Keep full Y range for each chunk chunk_miny = miny chunk_maxy = maxy - - bounds_str = f"([{chunk_minx},{chunk_maxx}],[{chunk_miny},{chunk_maxy}])" + query_maxx = chunk_maxx if chunk_idx == len(chunk_edges) - 1 else chunk_maxx - scale_x * 0.5 + if query_maxx < chunk_minx: + continue + + bounds_str = f"([{chunk_minx},{query_maxx}],[{chunk_miny},{chunk_maxy}])" chunk_tasks.append(( input_file, bounds_str, resolution, chunk_dir, chunk_idx, - num_threads, + len(chunk_edges), dimension_reduction, - fallback_laz_dir, - chunk_size, + subsampling_method, )) - + # Process chunks in parallel using ProcessPoolExecutor for true CPU parallelism chunk_files = [] - failures: List[Tuple[int, str]] = [] total_points = 0 - + print(f" → Subsampling {len(chunk_tasks)} subtiles in parallel...") - with ProcessPoolExecutor(max_workers=max(1, num_threads // 2)) as executor: + with ProcessPoolExecutor(max_workers=num_threads, **_process_pool_kwargs()) as executor: futures = [executor.submit(subsample_tile_chunk, task) for task in chunk_tasks] for future in as_completed(futures): - chunk_idx, chunk_file, point_count, err = future.result() + chunk_file, point_count = future.result() if chunk_file and chunk_file.exists(): chunk_files.append(chunk_file) total_points += point_count - else: - failures.append((chunk_idx, err)) - + if not chunk_files: - # Keep chunk_dir for inspection. return (input_file.name, False, "No chunks produced", 0) - # Never silently produce partial outputs: missing chunks indicate a failure. - if len(chunk_files) != num_threads: - failures_sorted = ", ".join( - f"{idx}({err})" if err else str(idx) for idx, err in sorted(failures, key=lambda x: x[0]) - ) - return ( - input_file.name, - False, - f"{len(failures)}/{num_threads} chunk(s) failed: {failures_sorted}", - 0, - ) - print(f" → Merging {len(chunk_files)} subsampled subtiles...") - merge_pipeline_file = None - if output_copc: - if not output_file.name.endswith(".copc.laz"): - output_file = output_file.parent / f"{output_file.stem}.copc.laz" - try: - untwine_cmd = get_untwine_path(require=True) - except RuntimeError as e: - return (input_file.name, False, str(e), 0) - - input_args = [] - for chunk_file in chunk_files: - input_args.extend(["-i", str(chunk_file)]) + # Merge chunks using PDAL to regular LAZ, then optionally convert the merge to COPC. + # Chunk files are already LAZ from subsample_tile_chunk + reader_type = "readers.las" # Chunks are LAZ files + merge_output_file = ( + temporary_laz_path(target_output_file, pipeline_dir) + if output_copc + else target_output_file + ) - result = subprocess.run( - [untwine_cmd] + input_args + ["-o", str(output_file)], - capture_output=True, - text=True, - check=False, - ) + merge_writer = { + "type": "writers.las", + "filename": str(merge_output_file), + "compression": True, + "forward": "all", + } + if dimension_reduction: + merge_writer["minor_version"] = 2 + merge_writer["dataformat_id"] = 0 else: - # Merge chunks using PDAL into LAZ - reader_type = "readers.las" - if not output_file.name.endswith(".laz"): - output_file = output_file.parent / (output_file.stem + ".laz") - merge_writer = { - "type": "writers.las", - "filename": str(output_file), - "compression": True, - } - if dimension_reduction: - merge_writer["minor_version"] = 2 - merge_writer["dataformat_id"] = 0 - else: - merge_writer["minor_version"] = 4 - merge_writer["extra_dims"] = "all" - merge_pipeline = { - "pipeline": [ - *[{"type": reader_type, "filename": str(f)} for f in chunk_files], - {"type": "filters.merge"}, - merge_writer, - ] - } - - merge_pipeline_file = chunk_dir / "merge.json" - with open(merge_pipeline_file, 'w') as f: - json.dump(merge_pipeline, f, indent=2) - - pdal_cmd = get_pdal_path() - result = subprocess.run( - [pdal_cmd, "pipeline", str(merge_pipeline_file)], - capture_output=True, - text=True, - check=False - ) - + merge_writer["minor_version"] = 4 + merge_writer["extra_dims"] = "all" + merge_pipeline = { + "pipeline": [ + *[{"type": reader_type, "filename": str(f)} for f in chunk_files], + {"type": "filters.merge"}, + merge_writer, + ] + } + + merge_pipeline_file = chunk_dir / "merge.json" + with open(merge_pipeline_file, 'w') as f: + json.dump(merge_pipeline, f, indent=2) + + pdal_cmd = get_pdal_path() + result = subprocess.run( + [pdal_cmd, "pipeline", str(merge_pipeline_file)], + capture_output=True, + text=True, + check=False + ) + # Clean up chunks and temporary files for chunk_file in chunk_files: if chunk_file.exists(): @@ -700,32 +427,46 @@ def subsample_single_file( chunk_file.unlink() except Exception: pass - + # Clean up merge pipeline - if merge_pipeline_file is not None and merge_pipeline_file.exists(): + if merge_pipeline_file.exists(): try: merge_pipeline_file.unlink() except Exception: pass - - # Remove chunk directory (safe now that all chunks succeeded). + + # Remove chunk directory if chunk_dir.exists(): try: import shutil shutil.rmtree(chunk_dir) except Exception: pass - + if result.returncode != 0: - prefix = "Untwine failed" if output_copc else "Merge failed" - return (input_file.name, False, f"{prefix}: {result.stderr[:100]}", 0) - - if not output_file.exists() or output_file.stat().st_size == 0: + return (input_file.name, False, f"Merge failed: {result.stderr[:100]}", 0) + + if not merge_output_file.exists() or merge_output_file.stat().st_size == 0: return (input_file.name, False, "Output file empty", 0) - + + if output_copc: + print(f" → Converting {merge_output_file.name} to COPC") + if not convert_laz_output_to_copc( + merge_output_file, + target_output_file, + source_metadata_file=input_file, + ): + return (input_file.name, False, "COPC conversion failed", 0) + try: + merge_output_file.unlink() + except OSError: + pass + if not target_output_file.exists() or target_output_file.stat().st_size == 0: + return (input_file.name, False, "COPC output file empty", 0) + print(f" ✓ {input_file.name}: {total_points:,} points") return (input_file.name, True, "Success", total_points) - + except Exception as e: print(f" ✗ {input_file.name}: {e}") return (input_file.name, False, str(e), 0) @@ -737,64 +478,88 @@ def subsample_simple( resolution: float, pipeline_dir: Path, dimension_reduction: bool = True, + subsampling_method: str = SUBSAMPLING_METHOD_CENTER_OF_MASS, output_copc: bool = False, ) -> Tuple[str, bool, str, int]: """ Simple single-pass subsampling (fallback method) with COPC reader optimization. - - Uses COPC reader for efficient reading. + + Uses COPC reader for efficient reading, but always outputs LAZ format. When dimension_reduction is True, only standard dimensions are written (minimal output). - + Args: input_file: Input file path output_file: Output file path resolution: Voxel resolution pipeline_dir: Directory for pipeline files dimension_reduction: If True, write only standard dimensions (no extra_dims). - output_copc: If True, write COPC output (".copc.laz"). - + subsampling_method: SmartTile subsampling method. + output_copc: If True, write final output as COPC LAZ. + Returns: Tuple of (filename, success, message, point_count) """ + subsampling_method = normalize_subsampling_method(subsampling_method) try: - is_copc = input_file.name.endswith('.copc.laz') + is_copc = _is_copc_file(input_file) reader_type = "readers.copc" if is_copc else "readers.las" - - if output_copc: - if not output_file.name.endswith('.copc.laz'): - output_file = output_file.parent / f"{output_file.stem}.copc.laz" - writer_opts = { - "type": "writers.copc", - "filename": str(output_file), - } - if not dimension_reduction: - writer_opts["extra_dims"] = "all" + + target_output_file = copc_output_path(output_file) if output_copc else laz_output_path(output_file) + work_output_file = ( + temporary_laz_path(target_output_file, pipeline_dir) + if output_copc + else target_output_file + ) + + writer_opts = { + "type": "writers.las", + "filename": str(work_output_file), + "compression": True, + "forward": "all", + } + if dimension_reduction: + writer_opts["minor_version"] = 2 + writer_opts["dataformat_id"] = 0 else: - if not output_file.name.endswith('.laz'): - output_file = output_file.parent / (output_file.stem + '.laz') - writer_opts = { - "type": "writers.las", - "filename": str(output_file), - "compression": True, - } - if dimension_reduction: - writer_opts["minor_version"] = 2 - writer_opts["dataformat_id"] = 0 + writer_opts["minor_version"] = 4 + writer_opts["extra_dims"] = "all" + + if subsampling_method == SUBSAMPLING_METHOD_CENTER_OF_MASS: + if is_copc and dimension_reduction: + point_count = center_of_mass_subsample_copc(input_file, work_output_file, resolution) else: - writer_opts["minor_version"] = 4 - writer_opts["extra_dims"] = "all" + point_count = center_of_mass_subsample_las( + input_file, + work_output_file, + resolution, + dimension_reduction=dimension_reduction, + ) + if output_copc: + print(f" → Converting {work_output_file.name} to COPC") + if not convert_laz_output_to_copc( + work_output_file, + target_output_file, + source_metadata_file=input_file, + ): + return (input_file.name, False, "COPC conversion failed", 0) + try: + work_output_file.unlink() + except OSError: + pass + return (input_file.name, True, "Success", point_count) + pipeline = { "pipeline": [ {"type": reader_type, "filename": str(input_file)}, - {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, + _voxel_subsampling_filter(resolution, subsampling_method), writer_opts, ] } - + pipeline_file = pipeline_dir / f"{input_file.stem}_simple.json" with open(pipeline_file, 'w') as f: json.dump(pipeline, f, indent=2) - + pdal_cmd = get_pdal_path() result = subprocess.run( [pdal_cmd, "pipeline", str(pipeline_file)], @@ -802,10 +567,10 @@ def subsample_simple( text=True, check=False ) - + if pipeline_file.exists(): pipeline_file.unlink() - + if result.returncode != 0: # Fallback: if COPC reader failed, retry with readers.las if is_copc and ("copc" in result.stderr.lower() or "vlr" in result.stderr.lower()): @@ -813,7 +578,7 @@ def subsample_simple( pipeline = { "pipeline": [ {"type": "readers.las", "filename": str(input_file)}, - {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution}, + _voxel_subsampling_filter(resolution, subsampling_method), writer_opts, ] } @@ -833,14 +598,14 @@ def subsample_simple( else: return (input_file.name, False, result.stderr[:200], 0) - if not output_file.exists() or output_file.stat().st_size == 0: + if not work_output_file.exists() or work_output_file.stat().st_size == 0: return (input_file.name, False, "Output file empty", 0) # Get point count point_count = 0 try: info_result = subprocess.run( - [pdal_cmd, "info", "--metadata", str(output_file)], + [pdal_cmd, "info", "--metadata", str(work_output_file)], capture_output=True, text=True, check=True @@ -851,9 +616,24 @@ def subsample_simple( point_count = int(match.group(1)) except Exception: pass - + + if output_copc: + print(f" → Converting {work_output_file.name} to COPC") + if not convert_laz_output_to_copc( + work_output_file, + target_output_file, + source_metadata_file=input_file, + ): + return (input_file.name, False, "COPC conversion failed", 0) + try: + work_output_file.unlink() + except OSError: + pass + if not target_output_file.exists() or target_output_file.stat().st_size == 0: + return (input_file.name, False, "COPC output file empty", 0) + return (input_file.name, True, "Success", point_count) - + except Exception as e: return (input_file.name, False, str(e), 0) @@ -864,79 +644,77 @@ def subsample_parallel( resolution: float, num_cores: int, num_threads: int, - chunk_size: int, output_prefix: Optional[str] = None, dimension_reduction: bool = True, + subsampling_method: str = SUBSAMPLING_METHOD_CENTER_OF_MASS, output_copc: bool = False, - fallback_laz_dir: Optional[Path] = None, ) -> List[Path]: """ Subsample all files in directory using parallel chunk processing. - + Files are processed sequentially (one at a time), but each file is split spatially into chunks along X-axis and processed in parallel. - Uses PDAL voxelcentroidnearestneighbor filter. + Uses the selected SmartTile subsampling method. When dimension_reduction is True, only standard LAS dimensions are written (minimal output). - + Args: input_dir: Directory containing input files output_dir: Directory for output files resolution: Voxel resolution in meters num_cores: Not used (kept for compatibility) num_threads: Number of spatial chunks per file (from TILE_PARAMS['threads']) - chunk_size: Points per laspy streaming chunk for timeout fallback. output_prefix: Optional prefix for output filenames dimension_reduction: If True, write only standard dimensions (no extra_dims); default True = minimal. - output_copc: If True, write COPC outputs (".copc.laz") instead of LAZ. - fallback_laz_dir: Optional original LAS/LAZ directory for laspy timeout fallback. - + subsampling_method: "center-of-mass" or "nearest-to-centroid". + output_copc: If True, write final outputs as COPC LAZ. + Returns: List of created output file paths """ # Create output directory + subsampling_method = normalize_subsampling_method(subsampling_method) output_dir.mkdir(parents=True, exist_ok=True) - + # Create pipeline directory pipeline_dir = output_dir / "pipelines" pipeline_dir.mkdir(exist_ok=True) - - # Find input files - # Get all LAZ files (both .laz and .copc.laz) - # Note: *.laz will match both .laz and .copc.laz, so we use set to deduplicate - input_files = sorted(set(list(input_dir.glob("*.laz")) + list(input_dir.glob("*.copc.laz")))) - + + input_files = _subsample_input_files(input_dir) + if not input_files: print(f" No input files found in {input_dir}") return [] - + # Convert resolution to cm for filename res_cm = int(resolution * 100) - + output_ext = ".copc.laz" if output_copc else ".laz" + # Prepare tasks tasks = [] + manifest = _read_subsample_manifest(output_dir) for input_file in sorted(input_files): # Generate output filename stem = input_file.stem # Remove .copc suffix if present if stem.endswith('.copc'): stem = stem[:-5] - + # Extract original base filename by removing prefixes and resolution suffixes import re base_name = stem - - # Remove resolution suffix patterns from earlier subsampling stages. + + # Remove resolution suffix patterns from previous subsampling stages. base_name = re.sub(r'_subsampled[\d.]+m$', '', base_name) - base_name = re.sub(r'_subsampled_\d+cm$', '', base_name) + base_name = re.sub(r'_subsampled_\d+(?:\.\d+)?cm$', '', base_name) base_name = re.sub(r'_\d+cm$', '', base_name) - + # Remove output_prefix if present at the start (e.g., "output_dir_100m_") if output_prefix and base_name.startswith(output_prefix + '_'): base_name = base_name[len(output_prefix) + 1:] - + # Remove any remaining prefix patterns that look like "something_100m_" or "output_dir_100m_" base_name = re.sub(r'^[^_]+_\d+m_', '', base_name) - + # For tiled files, try to extract tile ID (c##_r##) pattern tile_match = re.search(r'(c\d+_r\d+)', base_name) if tile_match: @@ -950,25 +728,37 @@ def subsample_parallel( if base_before_tile: base_before_tile = re.sub(r'^[^_]+_\d+m_', '', base_before_tile) if base_before_tile: - output_name = f"{base_before_tile}_{tile_id}_subsampled_{res_cm}cm" + output_name = f"{base_before_tile}_{tile_id}_subsampled_{res_cm}cm{output_ext}" else: - output_name = f"{tile_id}_subsampled_{res_cm}cm" + output_name = f"{tile_id}_subsampled_{res_cm}cm{output_ext}" else: - output_name = f"{tile_id}_subsampled_{res_cm}cm" + output_name = f"{tile_id}_subsampled_{res_cm}cm{output_ext}" else: # Single file or no tile ID - use clean base name # Remove any remaining prefix patterns base_name = re.sub(r'^[^_]+_\d+m_', '', base_name) - output_name = f"{base_name}_subsampled_{res_cm}cm" - - output_ext = ".copc.laz" if output_copc else ".laz" - output_file = output_dir / f"{output_name}{output_ext}" - - # Skip if already exists - if output_file.exists() and output_file.stat().st_size > 0: - print(f" ⊙ Skipping {input_file.name} (already exists)") + output_name = f"{base_name}_subsampled_{res_cm}cm{output_ext}" + + output_file = output_dir / output_name + target_output_file = copc_output_path(output_file) if output_copc else laz_output_path(output_file) + signature = _subsample_request_signature( + input_file, + resolution, + dimension_reduction, + subsampling_method, + output_copc, + ) + + if _existing_subsample_is_reusable(target_output_file, manifest, signature): + print(f" ⊙ Skipping {input_file.name} (existing output matches request)") continue - + if target_output_file.exists() and target_output_file.stat().st_size > 0: + print(f" ↻ Rebuilding stale subsample for {input_file.name}") + try: + target_output_file.unlink() + except OSError as exc: + raise RuntimeError(f"Could not replace stale subsample {target_output_file}: {exc}") from exc + tasks.append(( input_file, output_file, @@ -976,44 +766,50 @@ def subsample_parallel( pipeline_dir, num_threads, dimension_reduction, + subsampling_method, output_copc, - fallback_laz_dir, - chunk_size, + signature, )) - + if not tasks: print(f" ✓ All files already subsampled") - return list(output_dir.glob("*.copc.laz" if output_copc else "*.laz")) - + return subsample_output_files(output_dir, output_copc) + print(f" Files to process: {len(tasks)}") print(f" Processing mode: Sequential (one file at a time)") print(f" Chunk parallelism: {num_threads} chunks per file (parallel)") + print(f" Subsampling method: {subsampling_method}") + print(f" Output format: {'COPC LAZ' if output_copc else 'LAZ'}") print() - + # Process files sequentially, but chunks within each file in parallel successful = 0 failed = 0 total_points = 0 - + for task in tasks: - filename, success, message, point_count = subsample_single_file(task) + worker_task = task[:-1] + signature = task[-1] + filename, success, message, point_count = subsample_single_file(worker_task) if success: successful += 1 total_points += point_count + target_output_file = copc_output_path(task[1]) if output_copc else laz_output_path(task[1]) + _record_subsample_output(output_dir, target_output_file, signature, point_count) else: failed += 1 print(f" ✗ {filename}: {message}") - + # Clean up pipeline directory if pipeline_dir.exists() and not any(pipeline_dir.iterdir()): pipeline_dir.rmdir() - + print() print(f" ═══ Summary ═══") print(f" Complete: {successful} successful, {failed} failed") print(f" Total points: {total_points:,}") - - return list(output_dir.glob("*.copc.laz" if output_copc else "*.laz")) + + return subsample_output_files(output_dir, output_copc) def run_subsample_pipeline( @@ -1025,22 +821,21 @@ def run_subsample_pipeline( output_prefix: Optional[str] = None, output_base_dir: Optional[Path] = None, dimension_reduction: bool = True, + subsampling_method: str = SUBSAMPLING_METHOD_CENTER_OF_MASS, output_copc_res1: bool = True, output_copc_res2: bool = False, - fallback_laz_dir: Optional[Path] = None, - chunk_size: int = 20_000_000, ) -> Tuple[Path, Path]: """ Run the complete subsampling pipeline. - + Steps: - 1. Subsample tiles to resolution 1 (default: 2cm) + 1. Subsample tiles to resolution 1 (default: 1cm) 2. Subsample resolution 1 files to resolution 2 (default: 10cm) - + Files are processed sequentially (one at a time), but each file is split spatially into num_threads chunks along X-axis and processed in parallel. When dimension_reduction is True (default), only standard LAS dimensions are written (minimal output). - + Args: tiles_dir: Directory containing tile COPC files (input) res1: First resolution in meters (default: 0.01 = 1cm) @@ -1050,34 +845,34 @@ def run_subsample_pipeline( output_prefix: Optional prefix for output filenames output_base_dir: Base directory for output (default: parent of tiles_dir) dimension_reduction: If True, write only standard dimensions (minimal); if False, keep extra_dims (e.g. PredInstance). - output_copc_res1: If True, res1 outputs are written as COPC (".copc.laz"). - output_copc_res2: If True, res2 outputs are written as COPC (".copc.laz"). - fallback_laz_dir: Optional original LAS/LAZ directory for laspy timeout fallback. - chunk_size: Points per laspy streaming chunk for timeout fallback. - + subsampling_method: "center-of-mass" or "nearest-to-centroid". + output_copc_res1: Write first-resolution outputs as COPC LAZ. + output_copc_res2: Write second-resolution outputs as COPC LAZ. + Returns: Tuple of (subsampled_res1_dir, subsampled_res2_dir) """ + subsampling_method = normalize_subsampling_method(subsampling_method) # Auto-detect CPU count if num_cores is None: num_cores = get_cpu_count() - + # Get num_threads from TILE_PARAMS if num_threads is None: num_threads = TILE_PARAMS.get('threads', 10) - + # Convert to cm for display/filenames (but use simple directory names) res1_cm = int(res1 * 100) res2_cm = int(res2 * 100) - + # Define output directories - use output_base_dir if provided, otherwise use tiles_dir's parent if output_base_dir is None: output_base_dir = tiles_dir.parent - + # Create output directories directly under output_base_dir subsampled_res1_dir = output_base_dir / "subsampled_res1" subsampled_res2_dir = output_base_dir / "subsampled_res2" - + print("=" * 60) print("3DTrees Subsampling Pipeline") print("=" * 60) @@ -1086,61 +881,78 @@ def run_subsample_pipeline( print(f"Resolution 2: {res2}m ({res2_cm}cm)") print(f"CPU cores: {num_cores}") print(f"Threads (chunks per file): {num_threads}") - print(f"Chunk size: {chunk_size:,} points") print(f"Dimension reduction: {dimension_reduction} ({'minimal (standard dims only)' if dimension_reduction else 'keep all (extra_dims preserved)'})") - print(f"Resolution 1 output: {'COPC (.copc.laz)' if output_copc_res1 else 'LAZ (.laz)'}") - print(f"Resolution 2 output: {'COPC (.copc.laz)' if output_copc_res2 else 'LAZ (.laz)'}") + print(f"Subsampling method: {subsampling_method}") + print(f"Resolution 1 output: {'COPC LAZ' if output_copc_res1 else 'LAZ'}") + print(f"Resolution 2 output: {'COPC LAZ' if output_copc_res2 else 'LAZ'}") print() - + # Step 1: Subsample to resolution 1 print("=" * 60) print(f"Step 1: Subsampling to {res1_cm}cm ({res1}m)") print("=" * 60) - + res1_files = subsample_parallel( input_dir=tiles_dir, output_dir=subsampled_res1_dir, resolution=res1, num_cores=num_cores, num_threads=num_threads, - chunk_size=chunk_size, output_prefix=output_prefix, dimension_reduction=dimension_reduction, + subsampling_method=subsampling_method, output_copc=output_copc_res1, - fallback_laz_dir=fallback_laz_dir, ) - + if not res1_files: raise ValueError(f"No files created in {subsampled_res1_dir}") - + print(f"\n ✓ {res1_cm}cm subsampling complete: {len(res1_files)} files") print(f" Output: {subsampled_res1_dir}") - + + if abs(float(res1) - float(res2)) < 1e-12 and output_copc_res1 == output_copc_res2: + print() + print("=" * 60) + print(f"Step 2: Reusing {res1_cm}cm output for matching resolution 2") + print("=" * 60) + if subsampled_res2_dir.exists(): + shutil.rmtree(subsampled_res2_dir) + shutil.copytree(subsampled_res1_dir, subsampled_res2_dir) + res2_files = subsample_output_files(subsampled_res2_dir, output_copc_res2) + print(f" ✓ Resolution 2 equals resolution 1; copied {len(res2_files)} file(s)") + print(f" Output: {subsampled_res2_dir}") + print() + print("=" * 60) + print("Subsampling Pipeline Complete") + print("=" * 60) + print(f" Resolution 1 ({res1_cm}cm): {len(res1_files)} files in {subsampled_res1_dir}") + print(f" Resolution 2 ({res2_cm}cm): {len(res2_files)} files in {subsampled_res2_dir}") + return subsampled_res1_dir, subsampled_res2_dir + # Step 2: Subsample resolution 1 to resolution 2 print() print("=" * 60) print(f"Step 2: Subsampling to {res2_cm}cm ({res2}m)") print("=" * 60) - + res2_files = subsample_parallel( input_dir=subsampled_res1_dir, output_dir=subsampled_res2_dir, resolution=res2, num_cores=num_cores, num_threads=num_threads, - chunk_size=chunk_size, output_prefix=output_prefix, dimension_reduction=dimension_reduction, + subsampling_method=subsampling_method, output_copc=output_copc_res2, - fallback_laz_dir=None, ) - + if not res2_files: raise ValueError(f"No files created in {subsampled_res2_dir}") - + print(f"\n ✓ {res2_cm}cm subsampling complete: {len(res2_files)} files") print(f" Output: {subsampled_res2_dir}") - + # Summary print() print("=" * 60) @@ -1148,7 +960,7 @@ def run_subsample_pipeline( print("=" * 60) print(f" Resolution 1 ({res1_cm}cm): {len(res1_files)} files in {subsampled_res1_dir}") print(f" Resolution 2 ({res2_cm}cm): {len(res2_files)} files in {subsampled_res2_dir}") - + return subsampled_res1_dir, subsampled_res2_dir @@ -1158,76 +970,90 @@ def main(): description="3DTrees Subsampling Pipeline - Parallel subsampling to multiple resolutions", formatter_class=argparse.RawDescriptionHelpFormatter, ) - + parser.add_argument( "--tiles_dir", "-i", type=Path, required=True, help="Directory containing tile COPC files" ) - + parser.add_argument( "--res1", type=float, - default=TILE_PARAMS.get('resolution_1', 0.02), - help=f"First resolution in meters (default: {TILE_PARAMS.get('resolution_1', 0.02)})" + default=TILE_PARAMS.get('resolution_1', 0.01), + help=f"First resolution in meters (default: {TILE_PARAMS.get('resolution_1', 0.01)})" ) - + parser.add_argument( "--res2", type=float, default=TILE_PARAMS.get('resolution_2', 0.1), help=f"Second resolution in meters (default: {TILE_PARAMS.get('resolution_2', 0.1)})" ) - + + parser.add_argument( + "--output-copc-res1", + "--output_copc_res1", + action=argparse.BooleanOptionalAction, + default=TILE_PARAMS.get("output_copc_res1", True), + help=( + "Write resolution 1 outputs as COPC LAZ " + f"(default: {TILE_PARAMS.get('output_copc_res1', True)})" + ), + ) + + parser.add_argument( + "--output-copc-res2", + "--output_copc_res2", + action=argparse.BooleanOptionalAction, + default=TILE_PARAMS.get("output_copc_res2", False), + help=( + "Write resolution 2 outputs as COPC LAZ " + f"(default: {TILE_PARAMS.get('output_copc_res2', False)})" + ), + ) + parser.add_argument( "--num_cores", type=int, default=None, help="Number of CPU cores (default: auto-detect, not used for chunking)" ) - + parser.add_argument( "--num_threads", type=int, default=None, help=f"Number of spatial chunks per file for parallel processing (default: {TILE_PARAMS.get('threads', 5)})" ) + parser.add_argument( - "--chunk-size", - "--chunk_size", - dest="chunk_size", - type=int, - default=TILE_PARAMS.get("chunk_size", 20_000_000), - help="Points per laspy streaming chunk for timeout fallback.", + "--subsampling-method", + "--subsampling_method", + choices=sorted(SUBSAMPLING_METHODS), + default=TILE_PARAMS.get("subsampling_method", SUBSAMPLING_METHOD_CENTER_OF_MASS), + help=( + "Subsampling method: center-of-mass averages XYZ per voxel; " + "nearest-to-centroid preserves the previous PDAL voxel centroid nearest-neighbor behavior " + f"(default: {TILE_PARAMS.get('subsampling_method', SUBSAMPLING_METHOD_CENTER_OF_MASS)})" + ), ) - + parser.add_argument( "--output_prefix", type=str, default=None, help="Optional prefix for output filenames" ) - parser.add_argument( - "--output-copc-res1", - action=argparse.BooleanOptionalAction, - default=True, - help="Write resolution-1 outputs as COPC (.copc.laz).", - ) - parser.add_argument( - "--output-copc-res2", - action=argparse.BooleanOptionalAction, - default=False, - help="Write resolution-2 outputs as COPC (.copc.laz).", - ) - + args = parser.parse_args() - + # Validate input if not args.tiles_dir.exists(): print(f"Error: Tiles directory does not exist: {args.tiles_dir}") sys.exit(1) - + # Run pipeline try: res1_dir, res2_dir = run_subsample_pipeline( @@ -1236,8 +1062,8 @@ def main(): res2=args.res2, num_cores=args.num_cores, num_threads=args.num_threads, - chunk_size=args.chunk_size, output_prefix=args.output_prefix, + subsampling_method=args.subsampling_method, output_copc_res1=args.output_copc_res1, output_copc_res2=args.output_copc_res2, ) diff --git a/src/main_tile.py b/src/main_tile.py index 29e6bb9..e00676b 100644 --- a/src/main_tile.py +++ b/src/main_tile.py @@ -1,15 +1,13 @@ #!/usr/bin/env python3 """ -Main tiling script: COPC-first source preparation, indexing, and tiling. +Main tiling script: index building and tiling from LAZ/LAS input. This script handles the first phase of the 3DTrees pipeline: -1. Normalize source LAZ/LAS files to COPC while preserving all dimensions -2. Build a spatial index (tindex) from the COPC sources -3. Calculate tile bounds -4. Create overlapping COPC tiles +1. Build spatial index (tindex) from input LAZ/LAS files +2. Calculate tile bounds +3. Create overlapping tiles (laspy crop, COPC conversion via PDAL or untwine) -Uses COPC-aware reads where helpful during tile creation and keeps all source -dimensions intact until the later subsampling stage decides what to retain. +Uses laspy for Phase 1 (distribute/crop) and PDAL or untwine for Phase 2 (COPC). Usage: python main_tile.py --input_dir /path/to/input --output_dir /path/to/output @@ -18,525 +16,52 @@ from __future__ import annotations import argparse -import json import os import shutil -import subprocess import sys -import tempfile -from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed +from concurrent.futures import ProcessPoolExecutor, as_completed from pathlib import Path from typing import Dict, List, Optional, Tuple import plot_tiles_and_copc +from copc_metadata import ( + append_source_geotiff_projection_evlrs as _append_source_geotiff_projection_evlrs, + copc_preserves_source_crs as _copc_preserves_source_crs, + first_crs_source as _first_crs_source, + laspy_laz_backend as _laspy_laz_backend, +) from parameters import TILE_PARAMS - - - -def get_pdal_path() -> str: - """Get the path to pdal executable.""" - import shutil - # Use shutil.which to find pdal in PATH - pdal_path = shutil.which("pdal") - return pdal_path if pdal_path else "pdal" - - -def get_pdal_wrench_path() -> str: - """Get the path to pdal_wrench executable.""" - import shutil - # Use shutil.which to find pdal_wrench in PATH - wrench_path = shutil.which("pdal_wrench") - return wrench_path if wrench_path else "pdal_wrench" - - -def get_untwine_path(require: bool = False) -> Optional[str]: - """Get the path to untwine, optionally failing if it is unavailable.""" - untwine_path = shutil.which("untwine") - if untwine_path: - return untwine_path - if require: - raise RuntimeError( - "untwine is required for the COPC-first tiling pipeline but was not found in PATH" - ) - return None - - -def _laspy_laz_backend(): - """Return the LAZ backend to use for laspy (Lazrs or LazrsParallel when available).""" - try: - import laspy - if hasattr(laspy.LazBackend, "LazrsParallel"): - return laspy.LazBackend.LazrsParallel - if hasattr(laspy.LazBackend, "Lazrs"): - return laspy.LazBackend.Lazrs - except Exception: - pass - return None - - -def list_point_cloud_files(input_dir: Path, include_copc: bool = True) -> List[Path]: - """List point cloud files in a directory, optionally including COPC.""" - files = sorted(list(input_dir.glob("*.las")) + list(input_dir.glob("*.laz"))) - if include_copc: - return files - return [f for f in files if not f.name.endswith(".copc.laz")] - - -def _copc_name_for_source(source_file: Path) -> str: - """Return the canonical COPC filename for a source point cloud.""" - if source_file.name.endswith(".copc.laz"): - return source_file.name - return f"{source_file.stem}.copc.laz" - - -def _format_pdal_bounds(bounds: Tuple[float, float, float, float]) -> str: - """Format bounds for PDAL readers.copc.""" - xmin, ymin, xmax, ymax = bounds - return f"([{xmin},{xmax}],[{ymin},{ymax}])" - - -def _union_bounds(bounds_list: List[Tuple[float, float, float, float]]) -> Tuple[float, float, float, float]: - """Return the union bbox for a list of bounds.""" - return ( - min(b[0] for b in bounds_list), - min(b[1] for b in bounds_list), - max(b[2] for b in bounds_list), - max(b[3] for b in bounds_list), - ) - - -def build_tindex(input_dir: Path, output_gpkg: Path) -> Path: - """ - Build spatial index (tindex) from point cloud files. - - Uses pdal tindex to create a GeoPackage containing the spatial - extents of all point cloud files for efficient spatial queries. - - Args: - input_dir: Directory containing input point cloud files - output_gpkg: Output path for tindex GeoPackage - - Returns: - Path to created tindex file - """ - print() - print("=" * 60) - print("Building spatial index (tindex)") - print("=" * 60) - - # Check if tindex already exists - if output_gpkg.exists(): - print(f" Using existing tindex: {output_gpkg}") - return output_gpkg - - # Create output directory - output_gpkg.parent.mkdir(parents=True, exist_ok=True) - - # Find LAZ/LAS/COPC source files. - source_files = list_point_cloud_files(input_dir, include_copc=True) - if not source_files: - raise ValueError(f"No point cloud files found in {input_dir}") - - # Try to get SRS from the first file to avoid default EPSG:4326 in tindex - tindex_srs = None - try: - pdal_cmd = get_pdal_path() - info_cmd = [pdal_cmd, "info", "--metadata", str(source_files[0])] - info_result = subprocess.run(info_cmd, capture_output=True, text=True, check=False) - if info_result.returncode == 0: - meta = json.loads(info_result.stdout) - tindex_srs = meta.get("metadata", {}).get("srs", {}).get("compoundwkt") or \ - meta.get("metadata", {}).get("spatialreference") - except Exception as e: - print(f" Warning: Could not extract SRS for tindex: {e}") - - print(f" Found {len(source_files)} source files") - print(f" Output: {output_gpkg}") - - # Create file list for pdal tindex (absolute paths, one per line). - # Use the path as-is (do not resolve symlinks) so the path keeps .laz/.las extension; - # Galaxy stages files as .dat and we symlink to input_dir/*.laz - resolving would give .dat and PDAL would fail. - with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as f: - for pf in source_files: - f.write(f"{pf.absolute()}\n") - file_list_path = Path(f.name) - - try: - pdal_cmd = get_pdal_path() - cmd = [ - pdal_cmd, "tindex", "create", - str(output_gpkg), - "--filelist", str(file_list_path), - "--tindex_name=Location", - "--ogrdriver=GPKG", - "--fast_boundary", - "--write_absolute_path", - ] - if tindex_srs: - cmd.append(f"--t_srs={tindex_srs}") - - result = subprocess.run( - cmd, - capture_output=True, - text=True, - check=False, - ) - - if result.returncode != 0: - raise RuntimeError( - f"pdal tindex failed: {result.stderr or result.stdout or 'unknown error'}" - ) - - print(f" ✓ Tindex created: {output_gpkg}") - - finally: - if file_list_path.exists(): - file_list_path.unlink() - - return output_gpkg - - -def calculate_tile_bounds( - tindex_file: Path, - tile_length: float, - tile_buffer: float, - output_dir: Path, -) -> Tuple[Path, Path, dict]: - """ - Calculate tile bounds from tindex. - - Uses prepare_tile_jobs.py to compute tile grid based on the - spatial extent of input files. - - Args: - tindex_file: Path to tindex GeoPackage - tile_length: Tile size in meters - tile_buffer: Buffer overlap in meters - output_dir: Directory for output files - - Returns: - Tuple of (tile_jobs_file, tile_bounds_json, env_dict) - """ - print() - print("=" * 60) - print("Calculating tile bounds") - print("=" * 60) - - script_dir = Path(__file__).parent - prepare_jobs_script = script_dir / "prepare_tile_jobs.py" - - jobs_file = output_dir / f"tile_jobs_{int(tile_length)}m.txt" - bounds_json = output_dir / "tile_bounds_tindex.json" - - cmd = [ - sys.executable, - str(prepare_jobs_script), - str(tindex_file), - f"--tile-length={tile_length}", - f"--tile-buffer={tile_buffer}", - f"--jobs-out={jobs_file}", - f"--bounds-out={bounds_json}", - ] - - print(f" Tile length: {tile_length}m") - print(f" Tile buffer: {tile_buffer}m") - - result = subprocess.run(cmd, capture_output=True, text=True, check=False) - - if result.returncode != 0: - raise RuntimeError(f"prepare_tile_jobs.py failed: {result.stderr}") - - # Parse environment variables from output - env = {} - for line in result.stdout.splitlines(): - if "=" in line: - key, value = line.split("=", 1) - env[key.strip()] = value.strip().strip('"') - - tile_count = env.get('tile_count', 'unknown') - print(f" ✓ Calculated {tile_count} tiles") - print(f" Jobs file: {jobs_file}") - print(f" Bounds file: {bounds_json}") - - return jobs_file, bounds_json, env - - -def update_tile_bounds_json_from_files( - tile_bounds_json: Path, - files_dir: Path, - file_glob: str = "*.laz", -) -> int: - """ - Update tile_bounds_tindex.json with the actual file header bounds from the - created tiles (e.g. subsampled LAZ), while preserving the planned tiling - geometry used for ownership and neighbor logic. - - Matches tiles by label c{col:02d}_r{row:02d} (e.g. c00_r00) to filenames - that start with that label (e.g. c00_r00_subsampled_1cm.laz). - - Returns: - Number of tiles whose bounds were updated. - """ - from merge_tiles import get_tile_bounds_from_header - - if not tile_bounds_json.exists(): - return 0 - with tile_bounds_json.open() as f: - data = json.load(f) - tiles = data.get("tiles", []) - if not tiles: - return 0 - - # Build label -> file path from files_dir - label_to_path: Dict[str, Path] = {} - for f in files_dir.glob(file_glob): - stem = f.stem - # Match c00_r00 (prefix before _subsampled or similar) - for sep in ("_subsampled", "_chunk", "."): - if sep in stem: - stem = stem.split(sep)[0] - break - if stem and stem not in label_to_path: - label_to_path[stem] = f - - updated = 0 - for tile in tiles: - col, row = tile["col"], tile["row"] - label = f"c{col:02d}_r{row:02d}" - path = label_to_path.get(label) - if path is None: - continue - bounds = get_tile_bounds_from_header(path) - if bounds is None: - continue - minx, maxx, miny, maxy = bounds - if "planned_bounds" not in tile and "bounds" in tile: - tile["planned_bounds"] = tile["bounds"] - tile["actual_bounds"] = [[minx, maxx], [miny, maxy]] - updated += 1 - - if updated > 0: - with tile_bounds_json.open("w") as f: - json.dump(data, f, indent=2) - return updated - - -def _read_pointcloud_header_bounds(pointcloud_file: Path) -> Tuple[float, float, float, float]: - """Read XY header bounds from a LAZ/LAS/COPC file.""" - import laspy - - if not pointcloud_file.exists(): - raise FileNotFoundError(f"Point cloud file not found: {pointcloud_file}") - - open_kwargs = {} - laz_backend = _laspy_laz_backend() - if laz_backend is not None: - open_kwargs["laz_backend"] = laz_backend - - try: - with laspy.open(str(pointcloud_file), **open_kwargs) as las: - return ( - float(las.header.x_min), - float(las.header.x_max), - float(las.header.y_min), - float(las.header.y_max), - ) - except Exception as exc: - raise RuntimeError( - f"Could not read header bounds from {pointcloud_file}: {exc}" - ) from exc - - -def rewrite_tile_bounds_json_for_single_file_skip( - tile_bounds_json: Path, - pointcloud_file: Path, -) -> Dict[str, float]: - """ - Rewrite tile_bounds_tindex.json to one tile based on actual file bounds. - - This is used when tiling is intentionally skipped for a single small file. - """ - if not tile_bounds_json.exists(): - raise FileNotFoundError(f"tile_bounds_tindex.json not found: {tile_bounds_json}") - - with tile_bounds_json.open() as f: - data = json.load(f) - - minx, maxx, miny, maxy = _read_pointcloud_header_bounds(pointcloud_file) - actual_bounds = [[minx, maxx], [miny, maxy]] - extent = {"minx": minx, "miny": miny, "maxx": maxx, "maxy": maxy} - - template_tile = {} - tiles = data.get("tiles", []) - if isinstance(tiles, list) and tiles: - template_tile = dict(tiles[0]) - - single_tile = dict(template_tile) - single_tile["col"] = 0 - single_tile["row"] = 0 - single_tile["core"] = actual_bounds - single_tile["planned_bounds"] = actual_bounds - single_tile["bounds"] = actual_bounds - single_tile["actual_bounds"] = actual_bounds - data["tiles"] = [single_tile] - - data["geo_extent"] = extent - data["proj_extent"] = extent - data["grid_bounds"] = { - "xmin": minx, - "xmax": maxx, - "ymin": miny, - "ymax": maxy, - } - - with tile_bounds_json.open("w") as f: - json.dump(data, f, indent=2) - - return extent - - -def get_source_files_from_tindex(tindex_file: Path) -> List[str]: - """Get list of source point cloud files (LAZ/LAS paths) from tindex database.""" - import sqlite3 - - conn = sqlite3.connect(str(tindex_file)) - cursor = conn.cursor() - - # Get table name from gpkg_contents - cursor.execute('SELECT table_name FROM gpkg_contents WHERE data_type = "features" LIMIT 1') - result = cursor.fetchone() - - if not result: - conn.close() - return [] - - table_name = result[0] - cursor.execute(f'SELECT DISTINCT Location FROM "{table_name}"') - files = [row[0] for row in cursor.fetchall()] - conn.close() - - return files - - -def get_source_bounds_from_tindex(tindex_file: Path) -> Dict[str, Tuple[float, float, float, float]]: - """Get spatial bounds for each source file from tindex GeoPackage geometry. - - Returns dict mapping file path -> (minx, miny, maxx, maxy). - """ - import sqlite3 - import struct - - conn = sqlite3.connect(str(tindex_file)) - cursor = conn.cursor() - - cursor.execute('SELECT table_name, column_name FROM gpkg_geometry_columns LIMIT 1') - row = cursor.fetchone() - if not row: - conn.close() - return {} - table_name, geom_col = row - - cursor.execute(f'SELECT Location, "{geom_col}" FROM "{table_name}"') - bounds_map = {} - for filepath, geom_blob in cursor.fetchall(): - if not geom_blob or not filepath: - continue - try: - # GeoPackage geometry binary: header (magic GP, version, flags, srs_id, envelope) - # flags byte at offset 3 tells envelope type - flags = geom_blob[3] - envelope_type = (flags >> 1) & 0x07 - header_size = 8 # magic(2) + version(1) + flags(1) + srs_id(4) - if envelope_type == 1: # [minx, maxx, miny, maxy] - minx, maxx, miny, maxy = struct.unpack_from(' Optional[Tuple[float, float, float, float]]: - """Parse '([xmin,xmax],[ymin,ymax])' into (xmin, ymin, xmax, ymax).""" - try: - s = proj_bounds.strip().strip("()") - parts = s.split("],[") - xpart = parts[0].strip("([])").split(",") - ypart = parts[1].strip("([])").split(",") - xmin, xmax = float(xpart[0]), float(xpart[1]) - ymin, ymax = float(ypart[0]), float(ypart[1]) - return (xmin, ymin, xmax, ymax) - except (ValueError, IndexError): - return None - - -def _load_core_bounds_from_tile_bounds_json( - tile_bounds_json: Optional[Path], -) -> Dict[str, Tuple[float, float, float, float]]: - """Load tile core bounds as (xmin, ymin, xmax, ymax) by cXX_rYY label.""" - if tile_bounds_json is None or not tile_bounds_json.exists(): - return {} - - with tile_bounds_json.open() as f: - data = json.load(f) - - core_bounds: Dict[str, Tuple[float, float, float, float]] = {} - for tile in data.get("tiles", []): - try: - col = int(tile["col"]) - row = int(tile["row"]) - core = tile["core"] - xmin, xmax = float(core[0][0]), float(core[0][1]) - ymin, ymax = float(core[1][0]), float(core[1][1]) - except (KeyError, TypeError, ValueError, IndexError): - continue - core_bounds[f"c{col:02d}_r{row:02d}"] = (xmin, ymin, xmax, ymax) - return core_bounds - - -def _bounds_overlap(a: Tuple[float, float, float, float], - b: Tuple[float, float, float, float]) -> bool: - """Check if two (minx, miny, maxx, maxy) boxes overlap.""" - return a[0] < b[2] and a[2] > b[0] and a[1] < b[3] and a[3] > b[1] - - -def _get_bounds( - filepath: str, - source_bounds: Dict[str, Tuple[float, float, float, float]], - bounds_by_basename: Optional[Dict[str, Tuple[float, float, float, float]]] = None, -) -> Optional[Tuple[float, float, float, float]]: - """Look up bounds by path, with basename fallback.""" - fb = source_bounds.get(filepath) - if fb is not None: - return fb - if bounds_by_basename is not None: - return bounds_by_basename.get(Path(filepath).name) - return None - - -def filter_source_files_for_tile( - source_files: List[str], - source_bounds: Dict[str, Tuple[float, float, float, float]], - tile_bounds: Tuple[float, float, float, float], - bounds_by_basename: Optional[Dict[str, Tuple[float, float, float, float]]] = None, -) -> List[str]: - """Return only source files whose bounds overlap the tile bounds.""" - result = [] - for f in source_files: - fb = _get_bounds(f, source_bounds, bounds_by_basename) - if fb is None: - result.append(f) # no bounds info, keep as candidate - elif _bounds_overlap(fb, tile_bounds): - result.append(f) - return result +from point_cloud_metadata import point_cloud_files +from tile_copc import ( + convert_laz_to_copc as _convert_laz_to_copc, + convert_laz_to_copc_pdal as _convert_laz_to_copc_pdal, + finalize_tile_to_copc as _finalize_tile_to_copc, + finalize_tile_to_copc_pdal as _finalize_tile_to_copc_pdal, + finalize_tile_to_copc_untwine as _finalize_tile_to_copc_untwine, +) +from tile_tindex import ( + bounds_overlap as _bounds_overlap, + build_tindex, + calculate_tile_bounds, + filter_source_files_for_tile, + get_bounds as _get_bounds, + get_pdal_path, + get_pdal_wrench_path, + get_source_bounds_from_tindex, + get_source_files_from_tindex, + parse_proj_bounds as _parse_proj_bounds, + update_tile_bounds_json_from_files, +) + + + +def _tiling_input_files(input_dir: Path) -> List[Path]: + """Return tiling inputs, preferring COPC twins over matching raw files.""" + return point_cloud_files(input_dir) def _make_tile_header(header_snapshot, offsets=None, scales=None): - """Create a LasHeader from a source header, copying VLRs and extra dimensions. + """Create a LasHeader from a source header, preserving source metadata. Args: header_snapshot: Source laspy header to copy from. @@ -547,25 +72,44 @@ def _make_tile_header(header_snapshot, offsets=None, scales=None): A new laspy.LasHeader ready for writing. """ import laspy + from laspy.vlrs.vlrlist import VLRList + # Rebuild instead of using header_snapshot.copy(): laspy marks COPC headers + # as non-writable even after stale COPC VLRs are removed. hdr = laspy.LasHeader( point_format=header_snapshot.point_format, version=header_snapshot.version, ) + hdr.point_count = 0 hdr.offsets = offsets if offsets is not None else header_snapshot.offsets hdr.scales = scales if scales is not None else header_snapshot.scales - # Copy non-COPC VLRs, avoiding duplicates - existing_vlr_keys = { - (getattr(v, "user_id", ""), v.record_id) for v in hdr.vlrs - } - for vlr in header_snapshot.vlrs: - if vlr.record_id in (1, 2) and getattr(vlr, "user_id", "") == "copc": - continue - vlr_key = (getattr(vlr, "user_id", ""), vlr.record_id) - if vlr_key not in existing_vlr_keys: - hdr.vlrs.append(vlr) - existing_vlr_keys.add(vlr_key) + for attr in ( + "file_source_id", + "global_encoding", + "uuid", + "system_identifier", + "generating_software", + "creation_date", + ): + if hasattr(header_snapshot, attr): + try: + setattr(hdr, attr, getattr(header_snapshot, attr)) + except Exception: + pass + + # COPC hierarchy/index records describe a specific COPC container layout. + # Tile-part LAZ files and regenerated COPC outputs must not inherit stale ones. + def is_stale_copc_vlr(vlr) -> bool: + return getattr(vlr, "user_id", "") == "copc" + + hdr.vlrs = VLRList([ + vlr for vlr in getattr(header_snapshot, "vlrs", []) + if not is_stale_copc_vlr(vlr) + ]) + source_evlrs = getattr(header_snapshot, "evlrs", None) + if source_evlrs is not None: + hdr.evlrs = VLRList([vlr for vlr in source_evlrs if not is_stale_copc_vlr(vlr)]) # Copy extra dimensions try: @@ -646,184 +190,66 @@ def _crop_with_laspy( return (False, 0, str(e)) -def _materialize_copc_subset( - src_file: str, - bounds: Tuple[float, float, float, float], - label: str, -) -> Tuple[Optional[Path], str]: - """Materialize a spatial COPC subset to a temporary LAS for local chunked processing.""" - pdal_cmd = get_pdal_path() - tmp_fd, tmp_name = tempfile.mkstemp(prefix=f"{label}_subset_", suffix=".las") - os.close(tmp_fd) - subset_path = Path(tmp_name) - - pipeline = { - "pipeline": [ - { - "type": "readers.copc", - "filename": str(src_file), - "bounds": _format_pdal_bounds(bounds), - }, - { - "type": "writers.las", - "filename": str(subset_path), - "forward": "all", - "extra_dims": "all", - }, - ] - } - - with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: - json.dump(pipeline, f) - pipeline_file = Path(f.name) - - try: - result = subprocess.run( - [pdal_cmd, "pipeline", str(pipeline_file)], - capture_output=True, - text=True, - check=False, - ) - if result.returncode != 0: - if subset_path.exists(): - subset_path.unlink() - stderr = (result.stderr or result.stdout or "unknown error").strip() - return (None, stderr[:200]) - if not subset_path.exists() or subset_path.stat().st_size == 0: - return (None, "subset pipeline produced no output") - return (subset_path, "OK") - finally: - if pipeline_file.exists(): - pipeline_file.unlink() - - -def _distribute_source_file(args: Tuple) -> Tuple[List[Tuple[str, int, int]], str]: +def _distribute_source_file(args: Tuple) -> List[Tuple[str, int]]: """ Phase 1: Read one source file (chunked) and write cropped parts for all overlapping tiles. Each source file is read exactly once. Points are streamed in chunks of CHUNK_SIZE to limit peak memory, and for each chunk the bounding-box - mask is applied for every overlapping tile. Matching points are - appended to per-tile LAZ part files (one per source file). + mask is applied for every overlapping tile. Matching points are written + immediately to per-tile LAS part files (one per source/chunk). When the LazrsParallel backend is used, RAYON_NUM_THREADS is set from the threads argument so chunk decompression uses multiple threads. Args: args: (source_idx, src_file, overlapping_tiles, tiles_dir, decompress_threads, chunk_size) - overlapping_tiles: list of (label, buffered_bounds, core_bounds) + overlapping_tiles: list of (label, (xmin, ymin, xmax, ymax)) decompress_threads: threads for LAZ decompression (LazrsParallel / Rayon) chunk_size: points per chunk (smaller = less peak RAM, more overhead) Returns: - Tuple of: - - list of (tile_label, point_count, core_point_count) for tiles that received points - - description of how the source was read + list of (tile_label, point_count) for tiles that received points """ import laspy import numpy as np source_idx, src_file, overlapping_tiles, tiles_dir, decompress_threads, chunk_size = args - def _count_existing_part( - part_file: Path, - core_bounds: Optional[Tuple[float, float, float, float]], - ) -> Tuple[int, int]: - with laspy.open(part_file) as reader: - point_count = int(reader.header.point_count) - if core_bounds is None or point_count == 0: - return point_count, point_count - - cxmin, cymin, cxmax, cymax = core_bounds - core_count = 0 - for part_chunk in reader.chunk_iterator(chunk_size): - px = np.asarray(part_chunk.x) - py = np.asarray(part_chunk.y) - core_mask = ( - (px >= cxmin) - & (px <= cxmax) - & (py >= cymin) - & (py <= cymax) - ) - core_count += int(core_mask.sum()) - return point_count, core_count - - # Skip tiles that already have the part file for this source to avoid - # regenerating intermediate LAS chunks during reruns/resumes. - existing_results: List[Tuple[str, int, int]] = [] - pending_overlaps: List[ - Tuple[ - str, - Tuple[float, float, float, float], - Optional[Tuple[float, float, float, float]], - ] - ] = [] - for label, bounds, core_bounds in overlapping_tiles: - part_file = Path(tiles_dir) / label / f"part_{source_idx}.las" - if part_file.exists() and part_file.stat().st_size > 0: - try: - point_count, core_count = _count_existing_part(part_file, core_bounds) - if point_count > 0: - existing_results.append((label, point_count, core_count)) - continue - except Exception: - # Rebuild unreadable/incomplete existing parts below. - pass - pending_overlaps.append((label, bounds, core_bounds)) - if not pending_overlaps: - return (existing_results, "all parts already present") - overlapping_tiles = pending_overlaps - # So LazrsParallel (Rayon) uses N threads for chunk decompression if decompress_threads and decompress_threads > 0: os.environ["RAYON_NUM_THREADS"] = str(decompress_threads) if not os.path.isfile(src_file): - return ([], "missing source file") + return [] try: - stream_file = src_file - temp_subset_path: Optional[Path] = None - read_mode = "full scan" - - if src_file.lower().endswith(".copc.laz") and overlapping_tiles: - query_bounds = _union_bounds([bounds for _, bounds, _ in overlapping_tiles]) - temp_subset_path, subset_message = _materialize_copc_subset( - src_file, query_bounds, f"src{source_idx}" - ) - if temp_subset_path is not None: - stream_file = str(temp_subset_path) - read_mode = f"COPC subset {query_bounds}" - else: - read_mode = f"COPC fallback full scan ({subset_message})" - laz_backend = _laspy_laz_backend() open_kwargs = {} - if stream_file.lower().endswith(".laz") and laz_backend is not None: + if src_file.lower().endswith(".laz") and laz_backend is not None: open_kwargs["laz_backend"] = laz_backend # --- stream through the file in chunks -------------------------------- - # We accumulate per-tile arrays and flush once at the end so that - # each tile gets exactly one part file from this source. - tile_arrays: Dict[str, list] = {label: [] for label, _, _ in overlapping_tiles} - tile_counts: Dict[str, int] = {label: 0 for label, _, _ in overlapping_tiles} - tile_core_counts: Dict[str, int] = {label: 0 for label, _, _ in overlapping_tiles} + # Write chunk parts immediately. The COPC finalization already merges + # part_*.las files, so keeping one part per source/chunk avoids holding + # all duplicated buffered-tile points in memory for large multi-tile runs. + tile_counts: Dict[str, int] = {label: 0 for label, _ in overlapping_tiles} # Build a compact bounds array for vectorised overlap tests - tile_labels = [label for label, _, _ in overlapping_tiles] - tile_xmin = np.array([b[0] for _, b, _ in overlapping_tiles]) - tile_xmax = np.array([b[2] for _, b, _ in overlapping_tiles]) - tile_ymin = np.array([b[1] for _, b, _ in overlapping_tiles]) - tile_ymax = np.array([b[3] for _, b, _ in overlapping_tiles]) - tile_core_bounds = [core for _, _, core in overlapping_tiles] + tile_labels = [label for label, _ in overlapping_tiles] + tile_xmin = np.array([b[0] for _, b in overlapping_tiles]) + tile_xmax = np.array([b[2] for _, b in overlapping_tiles]) + tile_ymin = np.array([b[1] for _, b in overlapping_tiles]) + tile_ymax = np.array([b[3] for _, b in overlapping_tiles]) - header_snapshot = None # will be captured from the first chunk - - with laspy.open(stream_file, **open_kwargs) as reader: + with laspy.open(src_file, **open_kwargs) as reader: header_snapshot = reader.header + src_scales = header_snapshot.scales + src_offsets = header_snapshot.offsets + tile_bounds_map = {lbl: bnds for lbl, bnds in overlapping_tiles} - for chunk in reader.chunk_iterator(chunk_size): + for chunk_index, chunk in enumerate(reader.chunk_iterator(chunk_size)): cx = np.asarray(chunk.x) cy = np.asarray(chunk.y) @@ -837,482 +263,47 @@ def _count_existing_part( cnt = int(mask.sum()) if cnt == 0: continue - # Store the raw packed array slice (compact, avoids header copy) - tile_arrays[label].append(chunk.array[mask]) - tile_counts[label] += cnt - core_bounds = tile_core_bounds[i] - if core_bounds is None: - tile_core_counts[label] += cnt - else: - cxmin, cymin, cxmax, cymax = core_bounds - core_mask = ( - mask - & (cx >= cxmin) - & (cx <= cxmax) - & (cy >= cymin) - & (cy <= cymax) - ) - tile_core_counts[label] += int(core_mask.sum()) - - if header_snapshot is None: - return ([], read_mode) - - # --- write one part file per tile that received points ----------------- - # Keep intermediate parts uncompressed so Phase 1 avoids repeated LAZ - # compression work before the final COPC write. - # Build per-tile headers with tile-centered offsets to prevent int32 - # overflow when coordinates are far from the source file's offset. - src_scales = header_snapshot.scales - src_offsets = header_snapshot.offsets - tile_bounds_map = {lbl: bnds for lbl, bnds, _ in overlapping_tiles} - - results: List[Tuple[str, int, int]] = list(existing_results) - for label in tile_labels: - if not tile_arrays[label]: - continue - tile_dir = tiles_dir / label - tile_dir.mkdir(exist_ok=True) - part_file = tile_dir / f"part_{source_idx}.las" - - combined = np.concatenate(tile_arrays[label]) - - # Compute tile-centered offsets to keep scaled values within int32 - bxmin, bymin, bxmax, bymax = tile_bounds_map[label] - tile_offsets = np.array([ - (bxmin + bxmax) / 2.0, - (bymin + bymax) / 2.0, - src_offsets[2], - ]) - - # Re-encode X/Y with tile-specific offsets - real_x = combined['X'] * src_scales[0] + src_offsets[0] - real_y = combined['Y'] * src_scales[1] + src_offsets[1] - combined['X'] = np.round((real_x - tile_offsets[0]) / src_scales[0]).astype(np.int32) - combined['Y'] = np.round((real_y - tile_offsets[1]) / src_scales[1]).astype(np.int32) - - new_header = _make_tile_header(header_snapshot, offsets=tile_offsets) - point_record = laspy.ScaleAwarePointRecord( - combined, new_header.point_format, new_header.scales, new_header.offsets, - ) - new_las = laspy.LasData(new_header) - new_las.points = point_record - new_las.write(str(part_file)) - - results.append((label, tile_counts[label], tile_core_counts[label])) - - return (results, read_mode) - - except Exception as e: - print(f" ⚠ Error processing {Path(src_file).name}: {e}") - return ([], f"error: {e}") - finally: - if 'temp_subset_path' in locals() and temp_subset_path is not None and temp_subset_path.exists(): - temp_subset_path.unlink() - - -def _finalize_tile_to_copc(args: Tuple) -> Tuple[str, bool, str]: - """ - Phase 2: Merge a tile's LAZ part files into a single COPC tile. - - Uses untwine for COPC generation. - - Args: - args: (label, tiles_dir, log_dir, tile_bounds) - tile_bounds: (xmin, ymin, xmax, ymax) or None - - Returns: - (label, success, message) - """ - label, tiles_dir, log_dir, tile_bounds = args - - final_tile = tiles_dir / f"{label}.copc.laz" - - # Skip if already finalised - if final_tile.exists() and final_tile.stat().st_size > 0: - return (label, True, "Already exists") - - tile_dir = tiles_dir / label - if not tile_dir.exists(): - return (label, True, "No data in bounds") - - parts = sorted(tile_dir.glob("part_*.las")) - if not parts: - if not any(tile_dir.iterdir()): - tile_dir.rmdir() - return (label, True, "No data in bounds") - - try: - success, message = _finalize_tile_to_copc_untwine(parts, final_tile, log_dir, label) - - if not success: - return (label, False, message) - - # Clean up parts - for part in parts: - if part.exists(): - part.unlink() - if tile_dir.exists() and not any(tile_dir.iterdir()): - tile_dir.rmdir() - - return (label, True, f"{len(parts)} parts merged ({message})") - - except Exception as e: - return (label, False, str(e)) - - -def _read_copc_point_count(path: Path) -> Optional[int]: - """Return the point count from a COPC header, or None when unavailable.""" - try: - if not path.exists(): - return None - import laspy - with laspy.open(str(path)) as reader: - return int(reader.header.point_count) - except Exception: - return None - - -def _finalize_tile_to_copc_pdal( - parts: List[Path], - final_tile: Path, - log_dir: Path, - label: str, - tile_bounds: Optional[Tuple[float, float, float, float]] = None, -) -> Tuple[bool, str]: - """Finalize a tile with the existing PDAL merge pipeline.""" - pdal_cmd = get_pdal_path() - - # Build COPC writer config with explicit offsets from tile bounds - # to prevent int32 overflow on scaled coordinate values. - writer_opts = { - "type": "writers.copc", - "filename": str(final_tile), - "forward": "all", - "extra_dims": "all", - } - if tile_bounds is not None: - bxmin, bymin, bxmax, bymax = tile_bounds - writer_opts["offset_x"] = (bxmin + bxmax) / 2.0 - writer_opts["offset_y"] = (bymin + bymax) / 2.0 - - if len(parts) == 1: - pipeline = { - "pipeline": [ - {"type": "readers.las", "filename": str(parts[0])}, - writer_opts, - ] - } - else: - readers = [{"type": "readers.las", "filename": str(p)} for p in parts] - pipeline = { - "pipeline": readers + [ - {"type": "filters.merge"}, - writer_opts, - ] - } - - pipeline_file = log_dir / f"{label}_pipeline.json" - with open(pipeline_file, "w") as f: - json.dump(pipeline, f) - - try: - result = subprocess.run( - [pdal_cmd, "pipeline", str(pipeline_file)], - capture_output=True, text=True, check=False, - ) - finally: - if pipeline_file.exists(): - pipeline_file.unlink() - - if result.returncode != 0: - return (False, f"COPC conversion failed: {result.stderr[:200]}") - return (True, "OK") - - -def _finalize_tile_to_copc_untwine( - parts: List[Path], - final_tile: Path, - log_dir: Path, - label: str, -) -> Tuple[bool, str]: - """Finalize a tile using untwine for fast COPC conversion. - - Untwine is purpose-built for COPC generation and significantly faster - than PDAL's writers.copc, especially for large point clouds. - - For multiple parts, passes all files directly to untwine (it supports - multiple input files natively). - """ - try: - untwine_cmd = get_untwine_path(require=True) - except RuntimeError as e: - return (False, str(e)) - - try: - input_args = [] - for p in parts: - input_args.extend(["-i", str(p)]) - - result = subprocess.run( - [untwine_cmd] + input_args + ["-o", str(final_tile)], - capture_output=True, text=True, check=False, - ) - - if result.returncode != 0: - return (False, f"untwine failed: {result.stderr[:200]}") - - if not final_tile.exists() or final_tile.stat().st_size == 0: - return (False, "untwine produced no output") - - return (True, "untwine") - - except Exception as e: - return (False, f"untwine error: {e}") - - -def _stream_source_into_las_parts( - input_laz: Path, - parts_dir: Path, - chunk_size: int, -) -> Tuple[bool, List[Path], str]: - """Split one source file into temporary LAS parts using chunked laspy reads.""" - import laspy - - laz_backend = _laspy_laz_backend() - open_kwargs = {} - if input_laz.suffix.lower() == ".laz" and laz_backend is not None: - open_kwargs["laz_backend"] = laz_backend - - parts_dir.mkdir(parents=True, exist_ok=True) - part_paths: List[Path] = [] - - try: - with laspy.open(str(input_laz), **open_kwargs) as reader: - for chunk_idx, chunk in enumerate(reader.chunk_iterator(chunk_size)): - if len(chunk) == 0: - continue - part_path = parts_dir / f"part_{chunk_idx:05d}.las" - part_header = _make_tile_header(reader.header) - with laspy.open(str(part_path), mode="w", header=part_header) as writer: - writer.write_points(chunk) - part_paths.append(part_path) - except Exception as e: - return (False, [], str(e)) - - if not part_paths: - return (False, [], "no points were written to temporary LAS parts") - - return (True, part_paths, f"{len(part_paths)} part(s)") + tile_dir = tiles_dir / label + tile_dir.mkdir(exist_ok=True) + part_file = tile_dir / f"part_{source_idx}_{chunk_index:06d}.las" + + selected = chunk.array[mask].copy() + + # Compute tile-centered offsets to keep scaled values within int32. + bxmin, bymin, bxmax, bymax = tile_bounds_map[label] + tile_offsets = np.array([ + (bxmin + bxmax) / 2.0, + (bymin + bymax) / 2.0, + src_offsets[2], + ]) + + # Re-encode X/Y with tile-specific offsets. + real_x = selected['X'] * src_scales[0] + src_offsets[0] + real_y = selected['Y'] * src_scales[1] + src_offsets[1] + selected['X'] = np.round((real_x - tile_offsets[0]) / src_scales[0]).astype(np.int32) + selected['Y'] = np.round((real_y - tile_offsets[1]) / src_scales[1]).astype(np.int32) + + new_header = _make_tile_header(header_snapshot, offsets=tile_offsets) + point_record = laspy.ScaleAwarePointRecord( + selected, new_header.point_format, new_header.scales, new_header.offsets, + ) + new_las = laspy.LasData(new_header) + new_las.points = point_record + new_las.write(str(part_file)) -def _convert_laz_to_copc_direct(input_laz: Path, output_copc: Path) -> Tuple[bool, str]: - """Convert a single LAZ/LAS file to COPC via a direct untwine call. + tile_counts[label] += cnt - Uses untwine so the tiling pipeline has one consistent COPC writer. - """ - untwine_cmd = get_untwine_path(require=True) - try: - r = subprocess.run( - [untwine_cmd, "-i", str(input_laz), "-o", str(output_copc)], - capture_output=True, text=True, check=False, - ) - success = r.returncode == 0 and output_copc.exists() and output_copc.stat().st_size > 0 - if success: - return (True, "untwine direct") - stderr = (r.stderr or r.stdout or "").strip() - if r.returncode < 0: - signal_msg = f"terminated by signal {-r.returncode}" - elif r.returncode > 128: - signal_msg = f"terminated by signal {r.returncode - 128}" - else: - signal_msg = f"rc={r.returncode}" - detail = stderr[:200] if stderr else "unknown error" - return (False, f"{signal_msg}: {detail}") - except Exception as e: - return (False, str(e)) - - -def _convert_laz_to_copc_chunked( - input_laz: Path, - output_copc: Path, - chunk_size: int, -) -> Tuple[bool, str]: - """Convert a single LAZ/LAS file to COPC via temporary chunked LAS parts.""" - temp_dir = Path( - tempfile.mkdtemp( - prefix=f"{output_copc.stem}_chunked_", - dir=str(output_copc.parent), - ) - ) - parts_dir = temp_dir / "parts" + results: List[Tuple[str, int]] = [] + for label in tile_labels: + if tile_counts[label] > 0: + results.append((label, tile_counts[label])) - try: - chunk_success, parts, chunk_message = _stream_source_into_las_parts( - input_laz=input_laz, - parts_dir=parts_dir, - chunk_size=chunk_size, - ) - if not chunk_success: - return (False, f"chunk split failed: {chunk_message}") - - copc_success, copc_message = _finalize_tile_to_copc_untwine( - parts=parts, - final_tile=output_copc, - log_dir=temp_dir, - label=input_laz.stem, - ) - if not copc_success: - return (False, copc_message) + return results - return (True, f"chunked laspy -> untwine ({chunk_message})") except Exception as e: - return (False, str(e)) - finally: - if not (output_copc.exists() and output_copc.stat().st_size > 0): - output_copc.unlink(missing_ok=True) - shutil.rmtree(temp_dir, ignore_errors=True) - - -def _convert_laz_to_copc( - input_laz: Path, - output_copc: Path, - chunk_size: int, - chunkwise_source_creation: bool = False, -) -> Tuple[bool, str]: - """Convert a single LAZ/LAS file to COPC. - - When chunkwise_source_creation is enabled, the source is first streamed into - temporary LAS parts before untwine builds the final COPC. This trades disk - I/O for lower peak RAM during source normalization. - """ - if chunkwise_source_creation: - return _convert_laz_to_copc_chunked( - input_laz=input_laz, - output_copc=output_copc, - chunk_size=chunk_size, - ) - direct_success, direct_message = _convert_laz_to_copc_direct(input_laz, output_copc) - if direct_success: - return (True, direct_message) - - output_copc.unlink(missing_ok=True) - chunked_success, chunked_message = _convert_laz_to_copc_chunked( - input_laz=input_laz, - output_copc=output_copc, - chunk_size=chunk_size, - ) - if chunked_success: - return ( - True, - f"chunked fallback after direct untwine failure ({direct_message})", - ) - return ( - False, - "direct untwine failed " - f"({direct_message}); chunked fallback failed ({chunked_message})", - ) - - -def ensure_copc_sources( - input_dir: Path, - copc_dir: Path, - max_workers: int = 4, - chunk_size: int = 20_000_000, - chunkwise_source_creation: bool = False, -) -> Tuple[Path, List[Path], List[Path]]: - """ - Normalize source inputs to COPC, preserving all dimensions. - - Returns: - Tuple of: - - path to the original_copc directory - - original non-COPC source files discovered in input_dir - - COPC files available for downstream tiling/subsampling - """ - original_sources = list_point_cloud_files(input_dir, include_copc=False) - existing_copc_inputs = sorted(input_dir.glob("*.copc.laz")) - - if not original_sources and not existing_copc_inputs: - raise ValueError(f"No point cloud files found in {input_dir}") - - copc_dir.mkdir(parents=True, exist_ok=True) - - if not original_sources: - print() - print("=" * 60) - print("Step 1: Using existing COPC sources") - print("=" * 60) - for src in existing_copc_inputs: - dest = copc_dir / src.name - if dest.exists() and dest.stat().st_size > 0: - continue - shutil.copy2(src, dest) - copc_files = sorted(copc_dir.glob("*.copc.laz")) - print(f" Reused {len(copc_files)} COPC file(s)") - return copc_dir, [], copc_files - - print() - print("=" * 60) - print("Step 1: Converting source files to COPC") - print("=" * 60) - print(" All source dimensions are preserved during COPC conversion.") - if chunkwise_source_creation: - print( - " COPC writer: chunked laspy staging -> untwine " - f"({chunk_size:,} pts/chunk)" - ) - else: - print(" COPC writer: untwine") - print(f" Input files: {len(original_sources)}") - print(f" COPC directory: {copc_dir}") - - tasks: List[Tuple[Path, Path]] = [] - expected_outputs: List[Path] = [] - reused = 0 - for src in original_sources: - out_copc = copc_dir / _copc_name_for_source(src) - expected_outputs.append(out_copc) - if out_copc.exists() and out_copc.stat().st_size > 0: - reused += 1 - continue - tasks.append((src, out_copc)) - - if reused: - print(f" Reusing {reused} existing COPC file(s)") - - if tasks: - worker_count = max(1, min(max_workers, len(tasks))) - print(f" Converting {len(tasks)} file(s) with {worker_count} worker(s)") - with ThreadPoolExecutor(max_workers=worker_count) as executor: - futures = { - executor.submit( - _convert_laz_to_copc, - src, - out, - chunk_size, - chunkwise_source_creation, - ): (src, out) - for src, out in tasks - } - for future in as_completed(futures): - src, out = futures[future] - success, message = future.result() - if not success: - raise RuntimeError(f"LAZ/LAS→COPC conversion failed: {src} ({message})") - print(f" ✓ {src.name} -> {out.name} [{message}]") - else: - print(" All COPC sources already exist") - - missing_outputs = [path for path in expected_outputs if not path.exists() or path.stat().st_size == 0] - if missing_outputs: - raise RuntimeError( - "Missing expected COPC file(s): " - + ", ".join(path.name for path in missing_outputs[:5]) - ) - - total_size_gb = sum(f.stat().st_size for f in expected_outputs) / (1024 ** 3) - print(f" ✓ COPC source preparation complete: {len(expected_outputs)} file(s), {total_size_gb:.2f} GB total") - return copc_dir, original_sources, expected_outputs + print(f" ⚠ Error processing {Path(src_file).name}: {e}") + return [] def create_tiles( @@ -1323,17 +314,17 @@ def create_tiles( threads: int = 5, max_parallel: int = 5, chunk_size: int = 20_000_000, - tile_bounds_json: Optional[Path] = None, ) -> List[Path]: """ - Create overlapping tiles from source point cloud files (two-phase). + Create overlapping tiles from source LAZ/LAS files (two-phase). Phase 1 – Distribute: each source file is read exactly once (in chunks) and cropped points are written as per-tile LAZ part files. This avoids the previous O(sources × tiles) read pattern. Phase 2 – Finalise: each tile's part files are merged and converted to - COPC format with untwine. Fully parallelised across tiles. + COPC format (tries untwine, falls back to PDAL). Fully parallelised + across tiles. Args: tindex_file: Path to tindex GeoPackage @@ -1343,15 +334,13 @@ def create_tiles( threads: Threads used per process for LAZ chunk decompression (LazrsParallel/Rayon) max_parallel: Maximum parallel workers for each phase chunk_size: Points per chunk when reading source files (smaller = less peak RAM) - tile_bounds_json: Optional tile_bounds_tindex.json path used to skip - tiles whose buffered crop contains points but whose core contains none. Returns: List of created tile paths """ print() print("=" * 60) - print("Creating tiles (two-phase)") + print("Step 3: Creating tiles (two-phase)") print("=" * 60) # Create directories @@ -1375,18 +364,6 @@ def create_tiles( if not all_tiles: raise ValueError("No tile jobs found") - core_bounds_by_label = _load_core_bounds_from_tile_bounds_json(tile_bounds_json) - - # Skip tiles whose COPC output already exists - pending_tiles: Dict[str, Tuple[float, float, float, float]] = {} - already_done = 0 - for label, bounds in all_tiles.items(): - final_tile = tiles_dir / f"{label}.copc.laz" - if final_tile.exists() and final_tile.stat().st_size > 0: - already_done += 1 - else: - pending_tiles[label] = bounds - # ── Source files & bounds ──────────────────────────────────────────── source_files = get_source_files_from_tindex(tindex_file) if not source_files: @@ -1396,13 +373,40 @@ def create_tiles( bounds_by_basename = ( {Path(p).name: b for p, b in source_bounds.items()} if source_bounds else {} ) + crs_reference = _first_crs_source([Path(p) for p in source_files]) + + # Skip tiles whose COPC output already exists + pending_tiles: Dict[str, Tuple[float, float, float, float]] = {} + already_done = 0 + for label, bounds in all_tiles.items(): + final_tile = tiles_dir / f"{label}.copc.laz" + if final_tile.exists() and final_tile.stat().st_size > 0: + valid_existing_crs = True + if crs_reference is not None: + preserved_geotiff, geotiff_message = _append_source_geotiff_projection_evlrs( + crs_reference, final_tile + ) + if not preserved_geotiff: + print(f" Existing tile {final_tile.name} GeoTIFF preservation failed: {geotiff_message}") + valid_existing_crs = False + valid_existing_crs, crs_message = _copc_preserves_source_crs( + crs_reference, final_tile + ) if valid_existing_crs else (False, geotiff_message) + if not valid_existing_crs: + print(f" Existing tile {final_tile.name} CRS validation failed: {crs_message}") + try: + final_tile.unlink(missing_ok=True) + except OSError: + pass + if valid_existing_crs: + already_done += 1 + else: + pending_tiles[label] = bounds + else: + pending_tiles[label] = bounds print(f" Source files: {len(source_files)}") print(f" Total tiles: {len(all_tiles)} ({already_done} already done, {len(pending_tiles)} pending)") - if core_bounds_by_label: - print(" Core occupancy check: enabled") - else: - print(" Core occupancy check: unavailable (no tile_bounds_tindex.json)") print(f" Workers: {max_parallel}") if not pending_tiles: @@ -1417,7 +421,7 @@ def create_tiles( overlapping = [] for label, tb in pending_tiles.items(): if fb is None or _bounds_overlap(fb, tb): - overlapping.append((label, tb, core_bounds_by_label.get(label))) + overlapping.append((label, tb)) if overlapping: distribute_tasks.append((source_idx, src_file, overlapping, tiles_dir, threads, chunk_size)) @@ -1427,7 +431,6 @@ def create_tiles( print() tile_point_counts: Dict[str, int] = {} - tile_core_point_counts: Dict[str, int] = {} with ProcessPoolExecutor(max_workers=max_parallel) as executor: futures = { executor.submit(_distribute_source_file, task): Path(task[1]).name @@ -1436,43 +439,22 @@ def create_tiles( for future in as_completed(futures): src_name = futures[future] try: - results, read_mode = future.result() - for label, count, core_count in results: + results = future.result() + for label, count in results: tile_point_counts[label] = tile_point_counts.get(label, 0) + count - tile_core_point_counts[label] = ( - tile_core_point_counts.get(label, 0) + core_count - ) if results: - total_pts = sum(c for _, c, _ in results) - total_core_pts = sum(c for _, _, c in results) - print(f" ✓ {src_name}: {total_pts:,} pts → {len(results)} tile(s) [{read_mode}]") - if core_bounds_by_label: - print(f" core ownership: {total_core_pts:,} pts") + total_pts = sum(c for _, c in results) + print(f" ✓ {src_name}: {total_pts:,} pts → {len(results)} tile(s)") else: - print(f" - {src_name}: no overlapping data [{read_mode}]") + print(f" - {src_name}: no overlapping data") except Exception as e: print(f" ✗ {src_name}: {e}") # ── Phase 2: Finalise ─────────────────────────────────────────────── - finalize_tasks = [] - buffer_only_skipped = 0 - for label, tile_bounds in pending_tiles.items(): - total_count = tile_point_counts.get(label, 0) - has_core_bounds = label in core_bounds_by_label - core_count = tile_core_point_counts.get( - label, - total_count if not has_core_bounds else 0, - ) - if has_core_bounds and total_count > 0 and core_count == 0: - buffer_only_skipped += 1 - shutil.rmtree(tiles_dir / label, ignore_errors=True) - print( - f" - {label}: skipped buffer-only tile " - f"({total_count:,} buffered pts, 0 core pts)", - flush=True, - ) - continue - finalize_tasks.append((label, tiles_dir, log_dir, tile_bounds)) + finalize_tasks = [ + (label, tiles_dir, log_dir, pending_tiles.get(label)) + for label in pending_tiles + ] print() print( @@ -1482,7 +464,7 @@ def create_tiles( successful = 0 failed = 0 - skipped = buffer_only_skipped + skipped = 0 with ProcessPoolExecutor(max_workers=max_parallel) as executor: futures = { @@ -1491,17 +473,11 @@ def create_tiles( } for future in as_completed(futures): label, success, message = future.result() - final_tile = tiles_dir / f"{label}.copc.laz" - pts = _read_copc_point_count(final_tile) - if pts is None: - pts = tile_point_counts.get(label, 0) + pts = tile_point_counts.get(label, 0) if success: if "Already exists" in message or "No data" in message: skipped += 1 - if "No data" in message: - print(f" - {label}: {message}") - else: - print(f" - {label}: {message} ({pts:,} pts)") + print(f" - {label}: {message}") else: successful += 1 print(f" ✓ {label}: {message} ({pts:,} pts)") @@ -1515,141 +491,89 @@ def create_tiles( return list(tiles_dir.glob("*.copc.laz")) -def _convert_laz_to_copc_pdal(input_laz: Path, output_copc: Path) -> bool: - """Convert a single LAZ file to COPC using PDAL.""" - pipeline = { - "pipeline": [ - {"type": "readers.las", "filename": str(input_laz)}, - { - "type": "writers.copc", - "filename": str(output_copc), - "forward": "all", - "extra_dims": "all", - }, - ] - } - with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as f: - json.dump(pipeline, f) - pipeline_file = Path(f.name) - try: - pdal_cmd = get_pdal_path() - r = subprocess.run( - [pdal_cmd, "pipeline", str(pipeline_file)], - capture_output=True, text=True, check=False, - ) - return r.returncode == 0 and output_copc.exists() and output_copc.stat().st_size > 0 - finally: - if pipeline_file.exists(): - pipeline_file.unlink() - - def run_tiling_pipeline( input_dir: Path, output_dir: Path, tile_length: float = 100, tile_buffer: float = 5, + grid_offset: float = 1.0, num_workers: int = 4, threads: int = 5, max_tile_procs: int = 5, - dimension_reduction: bool = True, + dimension_reduction: bool = True, # Ignored (kept for API compatibility) tiling_threshold: float = None, chunk_size: int = 2_000_000, - chunkwise_copc_source_creation: bool = False, ) -> Path: """ Run the complete tiling pipeline. Steps: - 1. Convert source LAZ/LAS inputs to COPC, preserving all dimensions - 2. Build spatial index (tindex) from COPC source files - 3. Calculate tile bounds - 4. Create overlapping COPC tiles + 1. Build spatial index (tindex) from input LAZ/LAS files + 2. Calculate tile bounds + 3. Create overlapping tiles (laspy crop, COPC conversion via untwine/PDAL) - If input folder contains a single file below tiling_threshold, the already - converted COPC source is returned directly for subsampling. + If input folder contains a single file below tiling_threshold, converts it to + COPC and returns that directory for direct subsampling. Args: input_dir: Directory containing input LAZ/LAS files output_dir: Base output directory tile_length: Tile size in meters tile_buffer: Buffer overlap in meters - num_workers: Worker count used for source COPC conversion + grid_offset: Offset from min coordinates + num_workers: Unused (kept for API compatibility) threads: Threads per PDAL writer max_tile_procs: Maximum parallel tile processes - dimension_reduction: - Kept for API compatibility with the caller. COPC conversion always - preserves all dimensions; dimension reduction only applies later in - the subsampling stage. + dimension_reduction: Ignored (kept for API compatibility) tiling_threshold: File size threshold in MB. If single file below this, skip tiling - chunk_size: Points per chunk when reading source data in Phase 1 (smaller = less peak RAM) + chunk_size: Points per chunk when reading LAZ/LAS in Phase 1 (smaller = less peak RAM) Returns: Path to tiles directory (or copc_single directory if tiling was skipped) """ print("=" * 60) - print("3DTrees Tiling Pipeline (COPC-first)") + print("3DTrees Tiling Pipeline (laspy + PDAL)") print("=" * 60) - untwine_cmd = get_untwine_path(require=True) print(f"Input: {input_dir}") print(f"Output: {output_dir}") print(f"Tile size: {tile_length}m with {tile_buffer}m buffer") - print("Source normalization: COPC with all dimensions preserved") - print(f"COPC writer: untwine ({untwine_cmd})") - print( - "Subsampling dimension policy: " - f"{'standard dims only later' if dimension_reduction else 'keep all dims later'}" - ) - print( - "Source COPC conversion: " - + ( - f"chunkwise staging enabled ({chunk_size:,} pts/chunk)" - if chunkwise_copc_source_creation - else "direct untwine" - ) - ) print() tiles_dir = output_dir / f"tiles_{int(tile_length)}m" - source_copc_dir = output_dir / "original_copc" log_dir = output_dir / "logs" log_dir.mkdir(parents=True, exist_ok=True) tindex_file = output_dir / f"tindex_{int(tile_length)}m.gpkg" # Check if we should skip tiling (single small file) should_skip_tiling = False - input_source_files = list_point_cloud_files(input_dir, include_copc=False) - if not input_source_files: - input_source_files = sorted(input_dir.glob("*.copc.laz")) if tiling_threshold is not None: - if len(input_source_files) == 1: - original_size_mb = input_source_files[0].stat().st_size / (1024 * 1024) + input_files = _tiling_input_files(input_dir) + + if len(input_files) == 1: + original_size_mb = input_files[0].stat().st_size / (1024 * 1024) if original_size_mb < tiling_threshold: should_skip_tiling = True print("=" * 60) print("Tiling Threshold Check") print("=" * 60) - print(f" Single file detected: {input_source_files[0].name}") + print(f" Single file detected: {input_files[0].name}") print(f" Original file size: {original_size_mb:.2f} MB") print(f" Threshold: {tiling_threshold} MB") - print(" Decision: Will skip tile generation after COPC normalization") + print(" Decision: Will skip tiling and use a COPC source for subsampling") print("=" * 60) print() - # Step 1: Normalize input sources to COPC so downstream steps can use COPC-aware reads. - source_copc_dir, original_sources, copc_sources = ensure_copc_sources( - input_dir=input_dir, - copc_dir=source_copc_dir, - max_workers=num_workers, - chunk_size=chunk_size, - chunkwise_source_creation=chunkwise_copc_source_creation, - ) + # Validate input + source_files = _tiling_input_files(input_dir) + if not source_files: + raise ValueError(f"No LAZ/LAS files found in {input_dir}") - # Step 2: Build tindex from COPC source files - tindex_file = build_tindex(source_copc_dir, tindex_file) + # Step 1: Build tindex from input LAZ/LAS + tindex_file = build_tindex(input_dir, tindex_file) - # Step 3: Calculate tile bounds + # Step 2: Calculate tile bounds jobs_file, bounds_json, env = calculate_tile_bounds( - tindex_file, tile_length, tile_buffer, output_dir + tindex_file, tile_length, tile_buffer, output_dir, grid_offset ) # Symlink tindex for Galaxy if needed @@ -1664,39 +588,57 @@ def run_tiling_pipeline( tindex_file, bounds_json, output_dir / "overview_copc_tiles.png" ) - # Check if we should skip tiling. - # Done AFTER tindex/bounds/plot so those outputs are always available for merge. + # Check if we should skip tiling (single small file) + # Done AFTER tindex/bounds/plot so those outputs are always available for merge if should_skip_tiling: print() print("=" * 60) print("Skipping Tiling (Single Small File)") print("=" * 60) + source_file = source_files[0] copc_single_dir = output_dir / "copc_single" copc_single_dir.mkdir(parents=True, exist_ok=True) - source_copc = copc_sources[0] - out_copc = copc_single_dir / source_copc.name - if not out_copc.exists() or out_copc.stat().st_size == 0: - print(" Reusing normalized COPC source...") - shutil.copy2(source_copc, out_copc) - print(f" ✓ Prepared {out_copc.name}") + if source_file.name.lower().endswith(".copc.laz"): + out_copc = copc_single_dir / source_file.name + if not out_copc.exists() or out_copc.stat().st_size == 0: + print(" Reusing uploaded COPC without conversion...") + shutil.copy2(source_file, out_copc) + print(f" ✓ Copied {out_copc.name}") + else: + print(f" Using existing {out_copc.name}") + print(" Returning COPC directory for direct subsampling") + print("=" * 60) + return copc_single_dir + + out_copc = copc_single_dir / f"{source_file.stem}.copc.laz" + rebuild_copc = not out_copc.exists() or out_copc.stat().st_size == 0 + if not rebuild_copc: + preserved_geotiff, geotiff_message = _append_source_geotiff_projection_evlrs( + source_file, out_copc + ) + valid_crs, crs_message = _copc_preserves_source_crs(source_file, out_copc) + if not preserved_geotiff or not valid_crs: + if not preserved_geotiff: + print(f" Existing COPC GeoTIFF preservation failed: {geotiff_message}") + print(f" Existing COPC CRS validation failed: {crs_message}") + print(" Rebuilding COPC from source LAZ...") + try: + out_copc.unlink(missing_ok=True) + except OSError: + pass + rebuild_copc = True + if rebuild_copc: + print(" Converting LAZ to COPC...") + if not _convert_laz_to_copc(source_file, out_copc): + raise RuntimeError(f"LAZ→COPC conversion failed: {source_file}") + print(f" ✓ Created {out_copc.name}") else: print(f" Using existing {out_copc.name}") - - rewritten_extent = rewrite_tile_bounds_json_for_single_file_skip(bounds_json, out_copc) - print( - " Rewrote tile_bounds_tindex.json for single-file skip " - f"to bounds x=[{rewritten_extent['minx']:.3f}, {rewritten_extent['maxx']:.3f}] " - f"y=[{rewritten_extent['miny']:.3f}, {rewritten_extent['maxy']:.3f}]" - ) - plot_tiles_and_copc.plot_extents( - tindex_file, bounds_json, output_dir / "overview_copc_tiles.png" - ) - print(" Regenerated overview_copc_tiles.png with corrected single-tile bounds") print(f" Returning COPC directory for direct subsampling") print("=" * 60) return copc_single_dir - # Step 4: Create tiles from the COPC source tindex + # Step 3: Create tiles tile_files = create_tiles( tindex_file, jobs_file, @@ -1705,15 +647,13 @@ def run_tiling_pipeline( threads, max_tile_procs, chunk_size, - tile_bounds_json=bounds_json, ) print() print("=" * 60) print("Tiling Pipeline Complete") print("=" * 60) - print(f" Original source files: {len(original_sources) if original_sources else len(copc_sources)}") - print(f" COPC source files: {len(copc_sources)}") + print(f" Source files: {len(source_files)}") print(f" Tiles created: {len(tile_files)}") print(f" Tiles directory: {tiles_dir}") @@ -1723,31 +663,31 @@ def run_tiling_pipeline( def main(): """CLI entry point.""" parser = argparse.ArgumentParser( - description="3DTrees Tiling Pipeline - COPC-first tiling from LAZ/LAS input", + description="3DTrees Tiling Pipeline - laspy + PDAL tiling from LAZ/LAS input", formatter_class=argparse.RawDescriptionHelpFormatter, ) - + parser.add_argument( "--input_dir", "-i", type=Path, required=True, - help="Input directory containing LAZ/LAS files" + help="Input directory containing LAZ files" ) - + parser.add_argument( "--output_dir", "-o", type=Path, required=True, help="Output directory for all stages" ) - + parser.add_argument( "--tile_length", type=float, default=TILE_PARAMS.get('tile_length', 100), help=f"Tile size in meters (default: {TILE_PARAMS.get('tile_length', 100)})" ) - + parser.add_argument( "--tile_buffer", type=float, @@ -1761,14 +701,14 @@ def main(): default=TILE_PARAMS.get('workers', 4), help=f"Number of parallel workers (default: {TILE_PARAMS.get('workers', 4)})" ) - + parser.add_argument( "--threads", type=int, default=TILE_PARAMS.get('threads', 5), help=f"Threads per COPC writer (default: {TILE_PARAMS.get('threads', 5)})" ) - + parser.add_argument( "--max_tile_procs", type=int, @@ -1782,12 +722,12 @@ def main(): help="Points per chunk when reading LAZ/LAS (default: 2_000_000; smaller = less peak RAM)", ) args = parser.parse_args() - + # Validate input if not args.input_dir.exists(): print(f"Error: Input directory does not exist: {args.input_dir}") sys.exit(1) - + # Run pipeline try: tiles_dir = run_tiling_pipeline( diff --git a/src/merge_deduplication.py b/src/merge_deduplication.py new file mode 100644 index 0000000..6de8ded --- /dev/null +++ b/src/merge_deduplication.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Point deduplication helpers for SmartTile merged products.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import numpy as np + + +def deduplicate_points( + points: np.ndarray, + instances: np.ndarray, + extra_dims: Dict[str, np.ndarray], + tolerance: float = 0.01, + grid_size: float = 50.0, +) -> Tuple[np.ndarray, np.ndarray, Dict[str, np.ndarray]]: + """Remove duplicate points from overlapping tiles. + + Duplicate keys are computed on a tolerance grid. When duplicate points exist, + the point with the higher instance ID is kept. + """ + n_points = len(points) + scale = 1.0 / tolerance + + min_coords = points.min(axis=0) + grid_indices = ((points[:, :2] - min_coords[:2]) / grid_size).astype(np.int32) + + max_grid_y = grid_indices[:, 1].max() + 1 + cell_keys = grid_indices[:, 0] * max_grid_y + grid_indices[:, 1] + + rounded = np.floor(points * scale).astype(np.int64) + point_hash = rounded[:, 0] + rounded[:, 1] * 73856093 + rounded[:, 2] * 19349669 + + sort_order = np.lexsort((-instances, point_hash, cell_keys)) + + sorted_cell_keys = cell_keys[sort_order] + sorted_point_hash = point_hash[sort_order] + + is_duplicate = np.zeros(n_points, dtype=bool) + is_duplicate[1:] = (sorted_cell_keys[1:] == sorted_cell_keys[:-1]) & ( + sorted_point_hash[1:] == sorted_point_hash[:-1] + ) + + keep_mask = np.ones(n_points, dtype=bool) + keep_mask[sort_order[is_duplicate]] = False + + unique_points = points[keep_mask] + unique_instances = instances[keep_mask] + unique_extras = {name: arr[keep_mask] for name, arr in extra_dims.items()} + + return unique_points, unique_instances, unique_extras diff --git a/src/merge_instance_ids.py b/src/merge_instance_ids.py new file mode 100644 index 0000000..b3e0f13 --- /dev/null +++ b/src/merge_instance_ids.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""Global instance ID helpers for SmartTile merge stages.""" + +from __future__ import annotations + +from typing import Tuple + + +TILE_OFFSET = 100000 + + +def global_id(tile_idx: int, local_id: int) -> int: + """Return the unique global ID for one local tile instance.""" + return tile_idx * TILE_OFFSET + local_id + + +def local_id(gid: int) -> Tuple[int, int]: + """Return tile index and local instance ID from a global instance ID.""" + return gid // TILE_OFFSET, gid % TILE_OFFSET diff --git a/src/merge_instance_matching.py b/src/merge_instance_matching.py new file mode 100644 index 0000000..0abc570 --- /dev/null +++ b/src/merge_instance_matching.py @@ -0,0 +1,650 @@ +#!/usr/bin/env python3 +"""Instance ID assignment, cross-tile matching, and orphan recovery for SmartTile merge.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Dict, List, Optional, Set, Tuple + +import numpy as np + +from merge_instance_ids import global_id +from merge_overlap import compute_ff3d_overlap_ratios +from merge_orphan_recovery import recover_orphaned_instances +from merge_tile_loading import TileData +from tile_spatial import ( + compute_centroids_vectorized, + find_overlap_region, + find_spatial_neighbors, + get_border_region_mask, +) +from union_find import UnionFind + + +@dataclass +class InstanceMergeResult: + """Outputs needed by downstream merge stages after instance matching.""" + + global_to_merged: Dict[int, int] + merged_instance_sources: Dict[int, List[int]] + tile_idx_to_name: Dict[int, str] + + +def assign_and_match_instances( + tiles: List[TileData], + tile_boundaries: Dict[str, Tuple[float, float, float, float]], + neighbors_by_tile: Dict[str, Dict[str, Optional[str]]], + kept_instances_per_tile: Dict[str, Set[int]], + filtered_instances_per_tile: Dict[str, Set[int]], + buffer_direction_per_tile: Dict[str, Dict[int, Optional[str]]], + buffer: float, + border_zone_width: float, + overlap_threshold: float, + correspondence_tolerance: float, + num_threads: int, + debug_instance_ids: Optional[Set[int]] = None, + match_all_instances: bool = False, + verbose: bool = False, +) -> InstanceMergeResult: + """Assign global instance IDs, match instances across tiles, and recover orphans.""" + # Stage 2: Assign Global Instance IDs + # ========================================================================= + print(f"\n{'=' * 60}") + print("Stage 2: Assigning global instance IDs") + print(f"{'=' * 60}") + + # Initialize Union-Find and track instance sizes + uf = UnionFind() + instance_sizes = {} # global_id -> point count + + for tile_idx, tile in enumerate(tiles): + print(f" Processing tile {tile_idx + 1}/{len(tiles)}: {tile.name} ({len(tile.points):,} points)...") + kept_instances = kept_instances_per_tile[tile.name] + + unique_inst, inst_counts = np.unique(tile.instances, return_counts=True) + + for i, local_inst in enumerate(unique_inst): + if local_inst <= 0 or local_inst not in kept_instances: + continue + + gid = global_id(tile_idx, local_inst) + size = int(inst_counts[i]) + uf.make_set(gid, size) + instance_sizes[gid] = size + + print(f" Total global instances: {len(instance_sizes)}") + print(f" ✓ Stage 2 completed: Assigned global IDs to {len(instance_sizes)} instances") + + # Helper functions for border matching + def get_opposite_direction(direction: str) -> str: + """Get opposite direction.""" + opposites = {"east": "west", "west": "east", "north": "south", "south": "north"} + return opposites.get(direction, direction) + + def log_instance_pair_analysis( + inst_id_a: int, + inst_id_b: int, + tile_a_name: str, + tile_b_name: str, + direction: str, + bbox_a: Tuple[float, float, float, float], + bbox_b: Tuple[float, float, float, float], + overlap_ratio: float, + overlap_threshold: float, + bbox_overlaps: bool, + centroid_a: np.ndarray, + centroid_b: np.ndarray, + size_a: int, + size_b: int, + matched: bool, + ): + """Log detailed analysis of an instance pair for debugging.""" + print(f"\n{'='*60}") + print(f"DEBUG: Instance Pair Analysis") + print(f"{'='*60}") + print(f"Instance {inst_id_a} ({tile_a_name}) <-> Instance {inst_id_b} ({tile_b_name})") + print(f"Direction: {tile_a_name} ({direction}) <-> {tile_b_name} ({get_opposite_direction(direction)})") + print(f"\nInstance {inst_id_a}:") + print(f" Tile: {tile_a_name}") + print(f" Point count: {size_a:,}") + print(f" Centroid: ({centroid_a[0]:.2f}, {centroid_a[1]:.2f}, {centroid_a[2]:.2f})") + print(f" BBox: ({bbox_a[0]:.2f}, {bbox_a[1]:.2f}) x ({bbox_a[2]:.2f}, {bbox_a[3]:.2f})") + print(f"\nInstance {inst_id_b}:") + print(f" Tile: {tile_b_name}") + print(f" Point count: {size_b:,}") + print(f" Centroid: ({centroid_b[0]:.2f}, {centroid_b[1]:.2f}, {centroid_b[2]:.2f})") + print(f" BBox: ({bbox_b[0]:.2f}, {bbox_b[1]:.2f}) x ({bbox_b[2]:.2f}, {bbox_b[3]:.2f})") + centroid_dist = np.linalg.norm(centroid_a - centroid_b) + print(f"\nCentroid distance: {centroid_dist:.2f}m") + print(f"BBox overlaps (10cm tolerance): {'YES' if bbox_overlaps else 'NO'}") + print(f"FF3D overlap ratio: {overlap_ratio:.4f}") + print(f"Overlap threshold: {overlap_threshold:.4f}") + print(f"Match result: {'MATCHED' if matched else 'NOT MATCHED'}") + if not matched: + reasons = [] + if not bbox_overlaps: + reasons.append("BBox doesn't overlap (within 10cm)") + if overlap_ratio < overlap_threshold: + reasons.append(f"Overlap ratio {overlap_ratio:.4f} < threshold {overlap_threshold:.4f}") + if reasons: + print(f" Reasons: {', '.join(reasons)}") + print(f"{'='*60}\n") + + def bboxes_overlap(bbox_a: Tuple[float, float, float, float], bbox_b: Tuple[float, float, float, float], tolerance: float = 0.1) -> bool: + """ + Check if two bounding boxes overlap or are within tolerance distance. + + Args: + bbox_a: (minx, maxx, miny, maxy) of first bounding box + bbox_b: (minx, maxx, miny, maxy) of second bounding box + tolerance: Maximum distance between boxes to still consider them (default: 0.1m = 10cm) + + Returns: + True if boxes overlap or are within tolerance distance + """ + minx_a, maxx_a, miny_a, maxy_a = bbox_a + minx_b, maxx_b, miny_b, maxy_b = bbox_b + + # Check if boxes overlap (original check) + if not (maxx_a < minx_b or minx_a > maxx_b or maxy_a < miny_b or miny_a > maxy_b): + return True + + # Check if boxes are within tolerance distance (almost touching) + # Compute gaps in X and Y dimensions + # If boxes don't overlap, find the minimum separation + x_gap = 0.0 + if maxx_a < minx_b: + x_gap = minx_b - maxx_a # A is to the left of B + elif maxx_b < minx_a: + x_gap = minx_a - maxx_b # B is to the left of A + # else: they overlap in X, x_gap = 0 + + y_gap = 0.0 + if maxy_a < miny_b: + y_gap = miny_b - maxy_a # A is below B + elif maxy_b < miny_a: + y_gap = miny_a - maxy_b # B is below A + # else: they overlap in Y, y_gap = 0 + + # Minimum separation is the diagonal distance between closest corners + # For non-overlapping boxes: √(x_gap² + y_gap²) + # But if boxes overlap in one dimension, we use the gap in the other dimension + separation = np.sqrt(x_gap * x_gap + y_gap * y_gap) + + return separation <= tolerance + + # ========================================================================= + # Stage 3: Border Region Instance Matching (or All Instance Matching) + # ========================================================================= + # Note: Cross-tile matching is optimized - each tile pair is checked exactly once + # using `for j in range(i + 1, len(tiles))`, avoiding duplicate A->B and B->A checks. + stage_name = "All Instance Matching" if match_all_instances else "Border Region Instance Matching" + print(f"\n{'=' * 60}") + print(f"Stage 3: {stage_name}") + print(f"{'=' * 60}") + + # Instance tracking for debugging + instance_tracking = {} # (tile_name, local_inst_id) -> tracking info + if debug_instance_ids: + print(f" Debug mode enabled for instances: {sorted(debug_instance_ids)}") + # Initialize tracking for all instances in all tiles + for tile_idx, tile in enumerate(tiles): + unique_instances = np.unique(tile.instances[tile.instances > 0]) + for local_inst in unique_instances: + gid = global_id(tile_idx, local_inst) + if local_inst in debug_instance_ids: + instance_tracking[(tile.name, local_inst)] = { + "tile_name": tile.name, + "local_id": local_inst, + "global_id": gid, + "filtered_in_stage1": local_inst not in kept_instances_per_tile[tile.name], + "in_border_region": False, + "border_direction": None, + "compared_with": [], + "matched_with": None + } + + if match_all_instances: + print(f" Finding all instances (matching all instances, not just border region)...") + else: + print(f" Finding border region instances (centroids in buffer to buffer+{border_zone_width}m zone)...") + + # Find instances to match (border region or all instances) + border_instances = {} # tile_name -> {instance_id: {'centroid': [...], 'points': [...], 'boundary': [...]}} + + # Build tile name to index mapping + tile_name_to_idx = {tile.name: idx for idx, tile in enumerate(tiles)} + + for tile_idx, tile in enumerate(tiles): + print(f" Processing tile {tile_idx + 1}/{len(tiles)}: {tile.name} ({len(tile.points):,} points)...") + tile_name = tile.name + # Use neighbors from JSON graph when available; fall back to spatial neighbors only + # if this tile was somehow not present in the JSON mapping (should not happen). + if tile_name in neighbors_by_tile: + neighbors = neighbors_by_tile[tile_name] + else: + neighbors = find_spatial_neighbors(tile.boundary, tile_name, tile_boundaries, tolerance=buffer) + kept_instances = kept_instances_per_tile[tile_name] + + neighbor_names = [n for n in neighbors.values() if n is not None] + print(f" Neighbors: {', '.join(neighbor_names) if neighbor_names else 'none'}") + if verbose: + for direction, neighbor_name in neighbors.items(): + if neighbor_name is not None: + neighbor_boundary = tile_boundaries.get(neighbor_name) + if neighbor_boundary: + overlap = find_overlap_region(tile.boundary, neighbor_boundary) + if overlap: + ov_minx, ov_maxx, ov_miny, ov_maxy = overlap + ov_width = ov_maxx - ov_minx + ov_height = ov_maxy - ov_miny + print(f" {direction.upper()} {neighbor_name}: overlap {ov_width:.1f}m x {ov_height:.1f}m") + + min_x, max_x, min_y, max_y = tile.boundary + border_zone_end = buffer + border_zone_width # border_zone_width beyond buffer + + # Define border region boundaries (buffer to buffer+border_zone_width from edges with neighbors) + # Inner edge of border region (end of buffer zone) + border_inner_min_x = min_x + (buffer if neighbors["west"] is not None else 0) + border_inner_max_x = max_x - (buffer if neighbors["east"] is not None else 0) + border_inner_min_y = min_y + (buffer if neighbors["south"] is not None else 0) + border_inner_max_y = max_y - (buffer if neighbors["north"] is not None else 0) + + # Outer edge of border region (buffer+border_zone_width from tile edge) + border_outer_min_x = min_x + (border_zone_end if neighbors["west"] is not None else 0) + border_outer_max_x = max_x - (border_zone_end if neighbors["east"] is not None else 0) + border_outer_min_y = min_y + (border_zone_end if neighbors["south"] is not None else 0) + border_outer_max_y = max_y - (border_zone_end if neighbors["north"] is not None else 0) + + border_instances[tile_name] = {} + + if match_all_instances: + # Collect ALL kept instances (not just border region) + all_unique_insts = kept_instances - {0} # All kept instances except ground + + if len(all_unique_insts) == 0: + print(f" No instances to match in {tile.name}") + continue + + # Compute centroids for all instances + print(f" Computing centroids for {len(all_unique_insts)} instances (all instances)...") + all_centroids = compute_centroids_vectorized(tile.points, tile.instances) + instance_centroids = { + inst_id: all_centroids[inst_id] + for inst_id in all_unique_insts + if inst_id in all_centroids + } + + instance_count = 0 + + # For each instance, extract full points (no direction filtering) + for inst_id in all_unique_insts: + if inst_id not in instance_centroids: + continue + + centroid = instance_centroids[inst_id] + + # Extract full instance points + inst_mask = tile.instances == inst_id + inst_points = tile.points[inst_mask] + + # Compute instance bounding box + inst_minx = inst_points[:, 0].min() + inst_maxx = inst_points[:, 0].max() + inst_miny = inst_points[:, 1].min() + inst_maxy = inst_points[:, 1].max() + + # Use "all" as direction to indicate this is not border-specific + border_instances[tile_name][inst_id] = { + 'centroid': centroid, + 'points': inst_points, + 'boundary': (inst_minx, inst_maxx, inst_miny, inst_maxy), + 'direction': 'all', # Special direction for all-instance matching + 'tile_idx': tile_idx + } + instance_count += 1 + + # Update tracking for debug instances + if debug_instance_ids and inst_id in debug_instance_ids: + key = (tile_name, inst_id) + if key in instance_tracking: + instance_tracking[key]["in_border_region"] = True + instance_tracking[key]["border_direction"] = 'all' + print(f" DEBUG: Instance {inst_id} included in all-instance matching") + + print(f" Found {instance_count} instances in {tile.name} (all instances)") + else: + pass + + border_mask = get_border_region_mask( + tile.points, tile.boundary, buffer, border_zone_end, neighbors + ) + border_points = tile.points[border_mask] + border_inst_ids = tile.instances[border_mask] + + # Get unique instances in border region (much smaller set than all instances) + border_unique_insts = set(np.unique(border_inst_ids)) - {0} + border_unique_insts &= kept_instances # Only kept instances + + if len(border_unique_insts) == 0: + continue + + if verbose: + print(f" Computing centroids for {len(border_unique_insts)} border instances...") + all_centroids = compute_centroids_vectorized(tile.points, tile.instances) + border_centroids = { + inst_id: all_centroids[inst_id] + for inst_id in border_unique_insts + if inst_id in all_centroids + } + + border_count = 0 + + # For each border instance, determine direction and extract full points + for inst_id in border_unique_insts: + if inst_id not in border_centroids: + continue + + centroid = border_centroids[inst_id] + cx, cy = centroid[0], centroid[1] + + # Determine border direction based on centroid position + border_direction = None + if neighbors["west"] is not None and cx < min_x + border_zone_end: + border_direction = "west" + elif neighbors["east"] is not None and cx > max_x - border_zone_end: + border_direction = "east" + elif neighbors["south"] is not None and cy < min_y + border_zone_end: + border_direction = "south" + elif neighbors["north"] is not None and cy > max_y - border_zone_end: + border_direction = "north" + + if border_direction is None: + continue + + # Extract full instance points (from original tile, not just border region) + inst_mask = tile.instances == inst_id + inst_points = tile.points[inst_mask] + + # Compute instance bounding box + inst_minx = inst_points[:, 0].min() + inst_maxx = inst_points[:, 0].max() + inst_miny = inst_points[:, 1].min() + inst_maxy = inst_points[:, 1].max() + + border_instances[tile_name][inst_id] = { + 'centroid': centroid, + 'points': inst_points, + 'boundary': (inst_minx, inst_maxx, inst_miny, inst_maxy), + 'direction': border_direction, + 'tile_idx': tile_idx + } + border_count += 1 + + # Update tracking for debug instances + if debug_instance_ids and inst_id in debug_instance_ids: + key = (tile_name, inst_id) + if key in instance_tracking: + instance_tracking[key]["in_border_region"] = True + instance_tracking[key]["border_direction"] = border_direction + print(f" DEBUG: Instance {inst_id} in border region ({border_direction})") + + if border_count > 0: + print(f" {tile.name}: {border_count} border instances") + + # Match instances between neighbor tiles + total_border_insts = sum(len(insts) for insts in border_instances.values()) + tiles_with_border = len([t for t in border_instances if border_instances[t]]) + if match_all_instances: + print(f" Found {total_border_insts} instances across {tiles_with_border} tiles (all instances)") + else: + print(f" Found {total_border_insts} border region instances across {tiles_with_border} tiles") + print(f" Processing tile pairs...") + + # Track which global IDs have already been matched to avoid duplicate checks + matched_gids = set() + + border_matches = 0 + total_bbox_checks = 0 + total_ff3d_computations = 0 + tiles_processed = 0 + + for i in range(len(tiles)): + tile_a = tiles[i] + # Use JSON-based neighbors when available + if tile_a.name in neighbors_by_tile: + neighbors_a = neighbors_by_tile[tile_a.name] + else: + neighbors_a = find_spatial_neighbors(tile_a.boundary, tile_a.name, tile_boundaries) + + for direction, neighbor_name in neighbors_a.items(): + if neighbor_name is None: + continue + + # Find neighbor tile index + tile_b_idx = tile_name_to_idx.get(neighbor_name) + if tile_b_idx is None: + continue + + tile_b = tiles[tile_b_idx] + + # Get instances from both tiles + if match_all_instances: + # Match ALL instances between neighbor tiles (no direction filtering) + border_insts_a = border_instances.get(tile_a.name, {}) + border_insts_b = border_instances.get(tile_b.name, {}) + else: + # Original logic: only match border instances in specific directions + border_insts_a = { + inst_id: data for inst_id, data in border_instances.get(tile_a.name, {}).items() + if data['direction'] == direction + } + border_insts_b = { + inst_id: data for inst_id, data in border_instances.get(tile_b.name, {}).items() + if data['direction'] == get_opposite_direction(direction) + } + + if not border_insts_a or not border_insts_b: + continue + + # Progress: Show which tile pair is being processed + matches_before = border_matches + if match_all_instances: + print(f" Checking {tile_a.name} <-> {tile_b.name} ({direction} neighbors): " + f"{len(border_insts_a)} vs {len(border_insts_b)} instances", end=" ... ") + else: + print(f" Checking {tile_a.name} ({direction}) <-> {tile_b.name} ({get_opposite_direction(direction)}): " + f"{len(border_insts_a)} vs {len(border_insts_b)} border instances", end=" ... ") + + # Build list of candidate instances from tile B (not already matched) + candidates_b = [] + for inst_id_b, data_b in border_insts_b.items(): + gid_b = global_id(tile_b_idx, inst_id_b) + if gid_b not in matched_gids: + candidates_b.append((inst_id_b, gid_b, data_b)) + + if not candidates_b: + continue + + # For each instance in tile A, check overlap with all candidate instances in tile B + for inst_id_a, data_a in border_insts_a.items(): + gid_a = global_id(i, inst_id_a) + + # Skip if already matched + if gid_a in matched_gids: + continue + + bbox_a = data_a['boundary'] + + # Check each candidate in tile B + for inst_id_b, gid_b, data_b in candidates_b: + # Skip if already matched + if gid_b in matched_gids: + continue + + bbox_b = data_b['boundary'] + + # Quick bounding box overlap/nearby check (within 10cm tolerance) + total_bbox_checks += 1 + bbox_overlaps = bboxes_overlap(bbox_a, bbox_b, tolerance=0.1) + + # Check if we should debug this pair + should_debug = ( + debug_instance_ids is not None and + (inst_id_a in debug_instance_ids or inst_id_b in debug_instance_ids) + ) + + if should_debug: + print(f"\n DEBUG: Checking pair {inst_id_a} <-> {inst_id_b}") + print(f" BBox overlap check: {'PASS' if bbox_overlaps else 'FAIL'}") + + if not bbox_overlaps: + if should_debug: + print(f" Skipping: BBox doesn't overlap (within 10cm tolerance)") + continue + + # Now compute expensive FF3D overlap ratio + total_ff3d_computations += 1 + points_a = data_a['points'] + points_b = data_b['points'] + instances_a = np.full(len(points_a), inst_id_a, dtype=np.int32) + instances_b = np.full(len(points_b), inst_id_b, dtype=np.int32) + + overlap_ratios_dict, size_a, size_b = compute_ff3d_overlap_ratios( + instances_a, instances_b, points_a, points_b, correspondence_tolerance + ) + + overlap_ratio = overlap_ratios_dict.get((inst_id_a, inst_id_b), 0.0) + + # Debug logging for instance pairs + if should_debug: + centroid_a = data_a['centroid'] + centroid_b = data_b['centroid'] + size_a_val = size_a.get(inst_id_a, 0) + size_b_val = size_b.get(inst_id_b, 0) + + # Update tracking + key_a = (tile_a.name, inst_id_a) + key_b = (tile_b.name, inst_id_b) + if key_a in instance_tracking: + instance_tracking[key_a]["compared_with"].append({ + "tile": tile_b.name, + "instance": inst_id_b, + "overlap_ratio": overlap_ratio, + "matched": overlap_ratio >= overlap_threshold + }) + if key_b in instance_tracking: + instance_tracking[key_b]["compared_with"].append({ + "tile": tile_a.name, + "instance": inst_id_a, + "overlap_ratio": overlap_ratio, + "matched": overlap_ratio >= overlap_threshold + }) + + log_instance_pair_analysis( + inst_id_a, inst_id_b, + tile_a.name, tile_b.name, direction, + bbox_a, bbox_b, + overlap_ratio, overlap_threshold, + bbox_overlaps, + centroid_a, centroid_b, + size_a_val, size_b_val, + overlap_ratio >= overlap_threshold + ) + + if overlap_ratio >= overlap_threshold: + # Merge via Union-Find + root = uf.union(gid_a, gid_b) + matched_gids.add(gid_a) + matched_gids.add(gid_b) + border_matches += 1 + + # Update tracking for matched instances + if debug_instance_ids: + key_a = (tile_a.name, inst_id_a) + key_b = (tile_b.name, inst_id_b) + if key_a in instance_tracking: + instance_tracking[key_a]["matched_with"] = (tile_b.name, inst_id_b) + if key_b in instance_tracking: + instance_tracking[key_b]["matched_with"] = (tile_a.name, inst_id_a) + + if verbose: + print(f" ✓ Match: {tile_a.name}:{inst_id_a} <-> {tile_b.name}:{inst_id_b} (overlap: {overlap_ratio:.3f})") + + # Progress: Show results for this tile pair + matches_this_pair = border_matches - matches_before + if matches_this_pair > 0: + print(f"{matches_this_pair} match(es) found") + else: + print("no matches") + tiles_processed += 1 + + # Periodic progress update every 10 tile pairs + if tiles_processed % 10 == 0: + print(f" Progress: {tiles_processed} tile pairs processed, {border_matches} total matches so far...") + + if match_all_instances: + print(f" Matched {border_matches} instance pairs (all instances)") + else: + print(f" Matched {border_matches} border region instance pairs") + print(f" Performance: {total_bbox_checks} bbox checks, {total_ff3d_computations} FF3D computations") + print(f" ✓ Stage 3 completed: {stage_name} done") + + # Print instance tracking summary for debug instances + if debug_instance_ids and instance_tracking: + print(f"\n{'='*60}") + print("Instance Tracking Summary") + print(f"{'='*60}") + for (tile_name, local_id), info in sorted(instance_tracking.items()): + print(f"\nInstance {local_id} (Tile: {tile_name}):") + print(f" Global ID: {info['global_id']}") + print(f" Filtered in Stage 1: {'YES' if info['filtered_in_stage1'] else 'NO'}") + print(f" In border region: {'YES' if info['in_border_region'] else 'NO'}") + if info['in_border_region']: + print(f" Border direction: {info['border_direction']}") + print(f" Compared with {len(info['compared_with'])} instance(s):") + for comp in info['compared_with']: + print(f" - {comp['tile']}:{comp['instance']} (overlap: {comp['overlap_ratio']:.4f}, matched: {comp['matched']})") + if info['matched_with']: + print(f" Matched with: {info['matched_with'][0]}:{info['matched_with'][1]}") + else: + print(f" Matched with: NONE") + print(f"{'='*60}\n") + + # Get connected components + components = uf.get_components() + print(f" Connected components: {len(components)}") + print(f" ✓ Instance matching completed: {len(components)} merged instance groups") + + # Create mapping from global ID to final merged ID + global_to_merged = {} + merged_instance_sources = {} # merged_id -> list of source global IDs (for CSV tracking) + + for merged_id, (root, members) in enumerate(components.items(), start=1): + merged_instance_sources[merged_id] = list(members) + + if len(members) > 1 and verbose: + print(f" Merged ID {merged_id} created from {len(members)} global IDs: {sorted(members)}") + + for gid in members: + global_to_merged[gid] = merged_id + + tile_idx_to_name = recover_orphaned_instances( + tiles=tiles, + kept_instances_per_tile=kept_instances_per_tile, + filtered_instances_per_tile=filtered_instances_per_tile, + buffer_direction_per_tile=buffer_direction_per_tile, + neighbors_by_tile=neighbors_by_tile, + global_to_merged=global_to_merged, + merged_instance_sources=merged_instance_sources, + buffer=buffer, + border_zone_width=border_zone_width, + num_threads=num_threads, + verbose=verbose, + ) + + return InstanceMergeResult( + global_to_merged=global_to_merged, + merged_instance_sources=merged_instance_sources, + tile_idx_to_name=tile_idx_to_name, + ) diff --git a/src/merge_loaded_cloud.py b/src/merge_loaded_cloud.py new file mode 100644 index 0000000..3ab2969 --- /dev/null +++ b/src/merge_loaded_cloud.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Load and mutate already-merged SmartTile point clouds.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Dict, Tuple + +import laspy +import numpy as np + +from instance_labels import validate_prediction_instance_labels +from merge_small_instances import merge_small_volume_instances +from point_cloud_metadata import extra_bytes_params_from_dimension_info + + +def load_merged_file( + merged_file: Path, + chunk_size: int = 1_000_000, +) -> Tuple[np.ndarray, Dict[str, np.ndarray], Dict[str, laspy.ExtraBytesParams]]: + """Load merged point coordinates and all non-XYZ dimensions from a LAZ file.""" + print(f"Loading existing merged file: {merged_file}") + + try: + with laspy.open(str(merged_file), laz_backend=laspy.LazBackend.LazrsParallel) as f: + n_points = f.header.point_count + points = np.empty((n_points, 3), dtype=np.float64) + all_dims: Dict[str, np.ndarray] = {} + extra_dim_params: Dict[str, laspy.ExtraBytesParams] = {} + offset = 0 + for chunk in f.chunk_iterator(chunk_size): + chunk_len = len(chunk) + end = offset + chunk_len + points[offset:end, 0] = chunk.x + points[offset:end, 1] = chunk.y + points[offset:end, 2] = chunk.z + for dim_name in f.header.point_format.dimension_names: + if dim_name in ("X", "Y", "Z"): + continue + arr = getattr(chunk, dim_name, None) + if arr is not None: + if dim_name not in all_dims: + all_dims[dim_name] = np.zeros(n_points, dtype=arr.dtype) + all_dims[dim_name][offset:end] = arr + for dim in f.header.point_format.extra_dimensions: + if dim.name not in extra_dim_params: + extra_dim_params[dim.name] = extra_bytes_params_from_dimension_info(dim) + if dim.name not in all_dims: + all_dims[dim.name] = np.zeros(n_points, dtype=dim.dtype) + all_dims[dim.name][offset:end] = getattr(chunk, dim.name) + offset = end + + print(f" Loaded {len(points):,} points") + if all_dims: + print(f" Dimensions from merged: {', '.join(sorted(all_dims.keys()))}") + + return points, all_dims, extra_dim_params + except Exception as exc: + raise ValueError(f"Error loading merged file {merged_file}: {exc}") from exc + + +def reassign_small_instances_in_dims( + points: np.ndarray, + all_dims: Dict[str, np.ndarray], + instance_dimension: str, + min_cluster_size: int = 250, + hull_point_threshold: int = 5000, + max_volume_for_merge: float = 5.0, + max_search_radius: float = 5.0, + num_threads: int = 1, + verbose: bool = False, +) -> Dict[str, int]: + """Reassign small instances in one loaded segmented/merged point cloud dimension.""" + if instance_dimension not in all_dims: + raise ValueError( + f"Cannot pre-remap reassign instances: dimension '{instance_dimension}' " + f"not found. Available dimensions: {', '.join(sorted(all_dims.keys()))}" + ) + + original_instances = np.asarray(all_dims[instance_dimension]) + validate_prediction_instance_labels( + original_instances, + instance_dimension, + Path(""), + ) + + reassigned_instances = original_instances.astype(np.int64, copy=True) + before_unique = np.unique(reassigned_instances[reassigned_instances > 0]) + + reassigned_instances, _ = merge_small_volume_instances( + points, + reassigned_instances, + min_points_for_hull_check=hull_point_threshold, + min_cluster_size=min_cluster_size, + max_volume_for_merge=max_volume_for_merge, + max_search_radius=max_search_radius, + num_threads=num_threads, + verbose=verbose, + ) + + changed_points = int(np.count_nonzero(reassigned_instances != original_instances)) + after_unique = np.unique(reassigned_instances[reassigned_instances > 0]) + all_dims[instance_dimension] = reassigned_instances.astype(original_instances.dtype, copy=False) + + return { + "changed_points": changed_points, + "instances_before": int(len(before_unique)), + "instances_after": int(len(after_unique)), + "instances_removed": int(len(before_unique) - len(after_unique)), + } diff --git a/src/merge_original_dimensions.py b/src/merge_original_dimensions.py new file mode 100644 index 0000000..6d626a3 --- /dev/null +++ b/src/merge_original_dimensions.py @@ -0,0 +1,222 @@ +#!/usr/bin/env python3 +"""Add original point-cloud dimensions back onto merged SmartTile products.""" + +from __future__ import annotations + +import gc +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Dict, Optional + +import laspy +import numpy as np +from scipy.spatial import cKDTree + +from dimension_transfer import plan_dimension_transfer +from point_cloud_metadata import ( + extra_bytes_params_from_dimension_info, + point_cloud_files, +) + + +def add_original_dimensions_to_merged( + merged_laz: Path, + original_input_dir: Path, + output_path: Path, + tolerance: float = 0.1, + retile_buffer: float = 2.0, + distance_threshold: Optional[float] = None, + num_threads: int = 4, +) -> None: + """Enrich a merged point cloud with dimensions from original input files.""" + if output_path.resolve() == Path(merged_laz).resolve(): + raise ValueError("output_path must differ from merged_laz to avoid overwriting input") + + original_files = point_cloud_files(original_input_dir) + if not original_files: + print(" No original input files found; skipping merged-with-originals output.", flush=True) + return + + print(f"\n{'=' * 60}", flush=True) + print("Adding original-file dimensions to merged point cloud", flush=True) + print(f"{'=' * 60}", flush=True) + + merged = laspy.read(str(merged_laz), laz_backend=laspy.LazBackend.LazrsParallel) + n_merged = len(merged.points) + merged_points = np.column_stack([merged.x, merged.y, merged.z]) + merged_dim_names = set(merged.point_format.dimension_names) + for dim in merged.point_format.extra_dimensions: + merged_dim_names.add(dim.name) + + skip_core = {"X", "Y", "Z"} + orig_dims: Dict[str, np.dtype] = {} + orig_extra_dim_info: Dict[str, object] = {} + for orig_path in original_files: + with laspy.open(str(orig_path), laz_backend=laspy.LazBackend.LazrsParallel) as f: + pf = f.header.point_format + pt_dtype = None + try: + one = f.read_points(1) + if one is not None and one.size > 0: + arr = getattr(one, "array", one) + dt = getattr(arr, "dtype", None) + if dt is not None and getattr(dt, "names", None) is not None: + pt_dtype = dt + if one is not None and one.size > 0: + for dim_name in pf.dimension_names: + if dim_name in skip_core or dim_name in orig_dims: + continue + dim_view = getattr(one, dim_name, None) + if dim_view is not None and hasattr(dim_view, "dtype"): + orig_dims[dim_name] = np.dtype(dim_view.dtype) + elif pt_dtype is not None and dim_name in pt_dtype.names: + orig_dims[dim_name] = pt_dtype.fields[dim_name][0] + else: + one = None + except Exception: + one = None + if one is None or pt_dtype is None: + for dim_name in pf.dimension_names: + if dim_name in skip_core or dim_name in orig_dims: + continue + orig_dims[dim_name] = np.float64 + for dim in pf.extra_dimensions: + if dim.name in skip_core: + continue + if dim.name not in orig_dims: + orig_dims[dim.name] = dim.dtype + if dim.name not in orig_extra_dim_info: + orig_extra_dim_info[dim.name] = dim + + transfer_plan = plan_dimension_transfer( + orig_dims, + merged_dim_names, + lambda name: getattr(merged, name, None), + skip=skip_core, + ) + dims_to_add = transfer_plan.output_dtypes + orig_dim_to_read = transfer_plan.output_to_source + + if not dims_to_add: + print(" No dimensions to add or replace from originals; writing copy of merged file.", flush=True) + merged.write(str(output_path), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) + del merged + gc.collect() + return + + for dim_name in sorted(transfer_plan.add_new.keys()): + print(f" Adding dimension: {dim_name}", flush=True) + for dim_name in sorted(transfer_plan.overwrite.keys()): + print(f" Replacing dimension (was empty/constant): {dim_name}", flush=True) + for orig_name, out_name in sorted(transfer_plan.renamed.items()): + print(f" Adding dimension from originals (collision with merged): {orig_name} -> {out_name}", flush=True) + + spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer + max_dist = distance_threshold if distance_threshold is not None else spatial_buffer + + best_dist = np.full(n_merged, np.inf, dtype=np.float64) + new_arrays: Dict[str, np.ndarray] = { + name: np.zeros(n_merged, dtype=dtype) + for name, dtype in dims_to_add.items() + } + + def process_one_original(orig_path: Path): + try: + orig_las = laspy.read(str(orig_path), laz_backend=laspy.LazBackend.LazrsParallel) + except Exception: + return None + bounds = ( + orig_las.header.x_min, + orig_las.header.x_max, + orig_las.header.y_min, + orig_las.header.y_max, + ) + orig_points = np.column_stack([orig_las.x, orig_las.y, orig_las.z]) + mask = ( + (merged_points[:, 0] >= bounds[0] - spatial_buffer) + & (merged_points[:, 0] <= bounds[1] + spatial_buffer) + & (merged_points[:, 1] >= bounds[2] - spatial_buffer) + & (merged_points[:, 1] <= bounds[3] + spatial_buffer) + ) + merged_idx = np.where(mask)[0] + if len(merged_idx) == 0: + del orig_las + return None + tree = cKDTree(orig_points) + distances, orig_idx = tree.query(merged_points[merged_idx], k=1, workers=1) + if distances.ndim == 2: + distances = distances[:, 0] + orig_idx = orig_idx[:, 0] + orig_dim_arrays = {} + for out_name, orig_name in orig_dim_to_read.items(): + arr = getattr(orig_las, orig_name, None) + if arr is not None: + orig_dim_arrays[out_name] = np.asarray(arr) + del orig_las + return (merged_idx, distances, orig_idx, orig_dim_arrays) + + if num_threads > 1: + print(f" Processing {len(original_files)} original files with {num_threads} workers...", flush=True) + with ThreadPoolExecutor(max_workers=num_threads) as executor: + results = list(executor.map(process_one_original, original_files)) + else: + results = [process_one_original(path) for path in original_files] + + for result in results: + if result is None: + continue + merged_idx, distances, orig_idx, orig_dim_arrays = result + within_max = distances <= max_dist + better = distances < best_dist[merged_idx] + accept = within_max & better + if np.any(accept): + acc_merged = merged_idx[accept] + acc_orig = orig_idx[accept] + best_dist[acc_merged] = distances[accept] + for dim_name, arr_np in orig_dim_arrays.items(): + new_arrays[dim_name][acc_merged] = arr_np[acc_orig] + gc.collect() + + for name, arr in new_arrays.items(): + arr = np.asarray(arr) + vmin, vmax = float(np.min(arr)), float(np.max(arr)) + n_nonzero = int(np.count_nonzero(arr)) + if vmin == vmax or n_nonzero == 0: + print(f" Warning: {name} has no variation (min=max={vmin}, non-zero={n_nonzero})", flush=True) + else: + print(f" {name}: min={vmin}, max={vmax}, non-zero={n_nonzero}", flush=True) + + extra_params = [] + for name, dtype in transfer_plan.add_new.items(): + if name in orig_extra_dim_info: + extra_params.append( + extra_bytes_params_from_dimension_info(orig_extra_dim_info[name], name=name) + ) + else: + extra_params.append(laspy.ExtraBytesParams(name=name, type=dtype)) + for orig_name, out_name in transfer_plan.renamed.items(): + dtype = transfer_plan.output_dtypes[out_name] + if orig_name is not None and orig_name in orig_extra_dim_info: + extra_params.append( + extra_bytes_params_from_dimension_info(orig_extra_dim_info[orig_name], name=out_name) + ) + else: + extra_params.append(laspy.ExtraBytesParams(name=out_name, type=dtype)) + if extra_params: + merged.add_extra_dims(extra_params) + + for name, arr in new_arrays.items(): + arr = np.asarray(arr) + try: + target_dtype = getattr(merged.points, name).dtype + if arr.dtype != target_dtype: + arr = arr.astype(target_dtype) + except (AttributeError, KeyError, TypeError): + pass + setattr(merged, name, arr) + + output_path.parent.mkdir(parents=True, exist_ok=True) + merged.write(str(output_path), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) + del merged + gc.collect() + print(f" Saved merged with original dimensions: {output_path}", flush=True) diff --git a/src/merge_orphan_recovery.py b/src/merge_orphan_recovery.py new file mode 100644 index 0000000..29d5678 --- /dev/null +++ b/src/merge_orphan_recovery.py @@ -0,0 +1,249 @@ +#!/usr/bin/env python3 +"""Recover filtered tile instances that are not covered by neighboring tiles.""" + +from __future__ import annotations + +from concurrent.futures import ThreadPoolExecutor +from typing import Dict, List, Optional, Set, Tuple + +import numpy as np +from scipy.spatial import cKDTree + +from merge_instance_ids import global_id +from merge_tile_loading import TileData +from tile_spatial import compute_centroids_vectorized + + +def _instance_is_in_border( + centroid: np.ndarray, + boundary: Tuple[float, float, float, float], + neighbors: Dict[str, Optional[str]], + buffer: float, + border_zone_end: float, +) -> bool: + min_x, max_x, min_y, max_y = boundary + cx, cy = centroid[0], centroid[1] + bi_min_x = min_x + (buffer if neighbors.get("west") is not None else 0) + bi_max_x = max_x - (buffer if neighbors.get("east") is not None else 0) + bi_min_y = min_y + (buffer if neighbors.get("south") is not None else 0) + bi_max_y = max_y - (buffer if neighbors.get("north") is not None else 0) + bo_min_x = min_x + (border_zone_end if neighbors.get("west") is not None else 0) + bo_max_x = max_x - (border_zone_end if neighbors.get("east") is not None else 0) + bo_min_y = min_y + (border_zone_end if neighbors.get("south") is not None else 0) + bo_max_y = max_y - (border_zone_end if neighbors.get("north") is not None else 0) + + return ( + (neighbors.get("west") is not None and bi_min_x <= cx <= bo_min_x) + or (neighbors.get("east") is not None and bo_max_x <= cx <= bi_max_x) + or (neighbors.get("south") is not None and bi_min_y <= cy <= bo_min_y) + or (neighbors.get("north") is not None and bo_max_y <= cy <= bi_max_y) + ) + + +def _build_instance_bboxes( + tiles: List[TileData], + kept_instances_per_tile: Dict[str, Set[int]], + filtered_instances_per_tile: Dict[str, Set[int]], + neighbors_by_tile: Dict[str, Dict[str, Optional[str]]], + buffer: float, + border_zone_width: float, +) -> Dict[str, Dict[int, Tuple[np.ndarray, np.ndarray]]]: + """Compute bboxes only for filtered instances and kept border instances.""" + border_zone_end = buffer + border_zone_width + instance_bboxes: Dict[str, Dict[int, Tuple[np.ndarray, np.ndarray]]] = {} + + for tile in tiles: + centroids = compute_centroids_vectorized(tile.points, tile.instances) + kept_instances = kept_instances_per_tile[tile.name] + neighbors = neighbors_by_tile.get(tile.name) or { + "east": None, + "west": None, + "north": None, + "south": None, + } + kept_in_border = { + inst_id + for inst_id in kept_instances + if inst_id in centroids + and _instance_is_in_border( + centroids[inst_id], + tile.boundary, + neighbors, + buffer, + border_zone_end, + ) + } + need_bbox = filtered_instances_per_tile[tile.name] | kept_in_border + + sort_idx = np.argsort(tile.instances) + sorted_inst = tile.instances[sort_idx] + sorted_points = tile.points[sort_idx] + unique_inst, first_idx, counts = np.unique( + sorted_inst, + return_index=True, + return_counts=True, + ) + bboxes = {} + for i, inst_id in enumerate(unique_inst): + if inst_id <= 0 or inst_id not in need_bbox: + continue + start = first_idx[i] + end = start + counts[i] + pts = sorted_points[start:end] + bboxes[inst_id] = (pts.min(axis=0), pts.max(axis=0)) + instance_bboxes[tile.name] = bboxes + + return instance_bboxes + + +def recover_orphaned_instances( + tiles: List[TileData], + kept_instances_per_tile: Dict[str, Set[int]], + filtered_instances_per_tile: Dict[str, Set[int]], + buffer_direction_per_tile: Dict[str, Dict[int, Optional[str]]], + neighbors_by_tile: Dict[str, Dict[str, Optional[str]]], + global_to_merged: Dict[int, int], + merged_instance_sources: Dict[int, List[int]], + buffer: float, + border_zone_width: float, + num_threads: int, + verbose: bool = False, +) -> Dict[int, str]: + """Recover filtered instances that are not covered by neighboring kept instances.""" + print("\n Checking for orphaned filtered instances...") + + tile_idx_to_name = {idx: tile.name for idx, tile in enumerate(tiles)} + instance_bboxes = _build_instance_bboxes( + tiles, + kept_instances_per_tile, + filtered_instances_per_tile, + neighbors_by_tile, + buffer, + border_zone_width, + ) + + print(" Building spatial index of kept instances (border region only)...") + kept_instance_data = [] + for tile_idx, tile in enumerate(tiles): + tile_bboxes = instance_bboxes[tile.name] + kept_instances = kept_instances_per_tile[tile.name] + for inst_id in kept_instances: + if inst_id not in tile_bboxes: + continue + bbox_min, bbox_max = tile_bboxes[inst_id] + center_x = (bbox_min[0] + bbox_max[0]) / 2.0 + center_y = (bbox_min[1] + bbox_max[1]) / 2.0 + kept_instance_data.append((center_x, center_y, tile_idx, inst_id, bbox_min, bbox_max)) + + search_radius = 5.0 + if kept_instance_data: + centers = np.array([(x, y) for x, y, _, _, _, _ in kept_instance_data]) + kept_tree = cKDTree(centers) + else: + kept_tree = None + print(" Warning: No kept instances found for spatial indexing") + + neighbor_trees: Dict[Tuple[int, int], cKDTree] = {} + for _, _, check_tile_idx, neighbor_inst, _, _ in kept_instance_data: + cache_key = (check_tile_idx, neighbor_inst) + if cache_key in neighbor_trees: + continue + check_tile = tiles[check_tile_idx] + neighbor_mask = check_tile.instances == neighbor_inst + neighbor_points = check_tile.points[neighbor_mask] + if len(neighbor_points) > 0: + neighbor_trees[cache_key] = cKDTree(neighbor_points[:, :2]) + + overlap_tolerance = 1.0 + + def check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: + tile_idx, local_inst = item + tile = tiles[tile_idx] + tile_name = tile.name + tile_bboxes = instance_bboxes[tile_name] + if local_inst <= 0 or local_inst not in tile_bboxes: + return (tile_idx, local_inst, True) + if buffer_direction_per_tile[tile_name].get(local_inst) is None: + return (tile_idx, local_inst, True) + fmin, fmax = tile_bboxes[local_inst] + orphan_center = ((fmin[0] + fmax[0]) / 2.0, (fmin[1] + fmax[1]) / 2.0) + inst_mask = tile.instances == local_inst + filtered_points = tile.points[inst_mask] + if len(filtered_points) == 0: + return (tile_idx, local_inst, True) + neighbor_has_tree = False + if kept_tree is not None: + nearby_indices = kept_tree.query_ball_point(orphan_center, r=search_radius) + for idx in nearby_indices: + _, _, check_tile_idx, neighbor_inst, nmin, nmax = kept_instance_data[idx] + if check_tile_idx == tile_idx: + continue + if ( + fmax[0] < nmin[0] - overlap_tolerance + or fmin[0] > nmax[0] + overlap_tolerance + or fmax[1] < nmin[1] - overlap_tolerance + or fmin[1] > nmax[1] + overlap_tolerance + ): + continue + neighbor_tree = neighbor_trees.get((check_tile_idx, neighbor_inst)) + if neighbor_tree is None: + continue + distances, _ = neighbor_tree.query(filtered_points[:, :2], k=1) + fraction_within = np.sum(distances <= overlap_tolerance) / len(filtered_points) + if fraction_within > 0.50: + neighbor_has_tree = True + break + return (tile_idx, local_inst, neighbor_has_tree) + + orphan_candidates: List[Tuple[int, int]] = [] + for tile_idx, tile in enumerate(tiles): + filtered_instances = filtered_instances_per_tile[tile.name] + buffer_directions = buffer_direction_per_tile[tile.name] + tile_bboxes = instance_bboxes[tile.name] + for local_inst in filtered_instances: + if local_inst <= 0 or local_inst not in tile_bboxes: + continue + if buffer_directions.get(local_inst) is None: + continue + if np.sum(tile.instances == local_inst) == 0: + continue + orphan_candidates.append((tile_idx, local_inst)) + + next_merged_id = max(global_to_merged.values()) + 1 if global_to_merged else 1 + recovered_count = 0 + skipped_covered = 0 + orphan_parallel_workers = max(1, min(num_threads, len(orphan_candidates) or 1)) + + if orphan_candidates: + print( + f" Checking {len(orphan_candidates)} orphan candidates " + f"with {orphan_parallel_workers} workers...", + flush=True, + ) + with ThreadPoolExecutor(max_workers=orphan_parallel_workers) as executor: + orphan_results = list(executor.map(check_one_orphan_covered, orphan_candidates)) + for tile_idx, local_inst, covered in orphan_results: + if covered: + skipped_covered += 1 + continue + tile = tiles[tile_idx] + gid = global_id(tile_idx, local_inst) + global_to_merged[gid] = next_merged_id + merged_instance_sources[next_merged_id] = [gid] + if verbose: + print( + " Recovered orphan - " + f"global_id={gid} (tile={tile.name}, local={local_inst}) " + f"-> merged_id={next_merged_id}" + ) + kept_instances_per_tile[tile.name].add(local_inst) + next_merged_id += 1 + recovered_count += 1 + + if recovered_count > 0 or skipped_covered > 0: + print(f" Recovered {recovered_count} orphaned instances") + print(f" Skipped {skipped_covered} instances (neighbor has overlapping tree)") + else: + print(" No orphaned instances found") + + return tile_idx_to_name diff --git a/src/merge_overlap.py b/src/merge_overlap.py new file mode 100644 index 0000000..87fd78e --- /dev/null +++ b/src/merge_overlap.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""Instance-overlap algorithms for SmartTile tile merging.""" + +from __future__ import annotations + +from typing import Dict, Tuple + +import numpy as np + + +def compute_ff3d_overlap_ratios( + instances_a: np.ndarray, + instances_b: np.ndarray, + points_a: np.ndarray, + points_b: np.ndarray, + correspondence_tolerance: float = 0.1, +) -> Tuple[Dict[Tuple[int, int], float], Dict[int, int], Dict[int, int]]: + """Compute FF3D-style overlap ratios between instance pairs. + + The metric is `max(intersection / size_a, intersection / size_b)`. Point + correspondence is approximated by hashing points onto the requested + tolerance grid, which keeps the overlap check linear after sorting the + lookup cloud. + """ + unique_a, counts_a = np.unique(instances_a[instances_a > 0], return_counts=True) + unique_b, counts_b = np.unique(instances_b[instances_b > 0], return_counts=True) + size_a = dict(zip(unique_a, counts_a)) + size_b = dict(zip(unique_b, counts_b)) + + scale = 1.0 / correspondence_tolerance + + grid_b = np.floor(points_b * scale).astype(np.int64) + hash_b = grid_b[:, 0] + grid_b[:, 1] * 73856093 + grid_b[:, 2] * 19349669 + + grid_a = np.floor(points_a * scale).astype(np.int64) + hash_a = grid_a[:, 0] + grid_a[:, 1] * 73856093 + grid_a[:, 2] * 19349669 + + sort_idx_b = np.argsort(hash_b) + sorted_hash_b = hash_b[sort_idx_b] + sorted_inst_b = instances_b[sort_idx_b] + + unique_hash_b, first_idx = np.unique(sorted_hash_b, return_index=True) + unique_inst_b = sorted_inst_b[first_idx] + + insert_pos = np.searchsorted(unique_hash_b, hash_a) + insert_pos_clamped = np.clip(insert_pos, 0, len(unique_hash_b) - 1) + matches_mask = unique_hash_b[insert_pos_clamped] == hash_a + + matched_inst_b = np.zeros(len(hash_a), dtype=instances_b.dtype) + matched_inst_b[matches_mask] = unique_inst_b[insert_pos_clamped[matches_mask]] + + valid_mask = matches_mask & (instances_a > 0) & (matched_inst_b > 0) + valid_inst_a = instances_a[valid_mask] + valid_inst_b = matched_inst_b[valid_mask] + + if len(valid_inst_a) > 0: + max_inst = max(instances_a.max(), instances_b.max()) + 1 + pair_keys = valid_inst_a.astype(np.int64) * max_inst + valid_inst_b.astype(np.int64) + unique_pairs, pair_counts = np.unique(pair_keys, return_counts=True) + intersection_counts = { + (int(key // max_inst), int(key % max_inst)): count + for key, count in zip(unique_pairs, pair_counts) + } + else: + intersection_counts = {} + + overlap_ratios = {} + for (inst_a, inst_b), intersection in intersection_counts.items(): + ratio_a = intersection / size_a[inst_a] if size_a.get(inst_a, 0) > 0 else 0 + ratio_b = intersection / size_b[inst_b] if size_b.get(inst_b, 0) > 0 else 0 + overlap_ratios[(inst_a, inst_b)] = max(ratio_a, ratio_b) + + return overlap_ratios, size_a, size_b diff --git a/src/merge_small_instances.py b/src/merge_small_instances.py new file mode 100644 index 0000000..e433bd6 --- /dev/null +++ b/src/merge_small_instances.py @@ -0,0 +1,308 @@ +#!/usr/bin/env python3 +"""Small-instance reassignment for SmartTile merged point clouds.""" + +from __future__ import annotations + +from concurrent.futures import ProcessPoolExecutor +from typing import Optional, Tuple + +import numpy as np +from scipy.spatial import cKDTree + + +def _compute_hull_wrapper(args): + """Compute one convex-hull volume for process-pool execution.""" + from scipy.spatial import ConvexHull + + points, bbox_volume = args + try: + hull = ConvexHull(points) + return (hull.volume, True) + except Exception: + return (bbox_volume, False) + + +def merge_small_volume_instances( + points: np.ndarray, + instances: np.ndarray, + min_points_for_hull_check: int = 1000, + min_cluster_size: int = 300, + max_volume_for_merge: float = 4.0, + max_search_radius: float = 5.0, + num_threads: int = 1, + verbose: bool = True, + presorted_points: Optional[np.ndarray] = None, + presorted_instances: Optional[np.ndarray] = None, + presorted_unique_inst: Optional[np.ndarray] = None, + presorted_first_idx: Optional[np.ndarray] = None, + presorted_inst_counts: Optional[np.ndarray] = None, +) -> Tuple[np.ndarray, int]: + """Reassign small/noisy instances to the nearest large instance. + + Only the instance ID array is modified. Extra dimensions remain attached to + their original points. + """ + use_presorted = ( + presorted_points is not None + and presorted_instances is not None + and presorted_unique_inst is not None + and presorted_first_idx is not None + and presorted_inst_counts is not None + ) + + if use_presorted: + sorted_points = presorted_points + sorted_instances = presorted_instances + unique_inst = presorted_unique_inst + first_idx = presorted_first_idx + inst_counts = presorted_inst_counts + print(f" {len(unique_inst):,} unique instances (pre-sorted).", flush=True) + else: + total_points = len(instances) + nonzero_mask = instances > 0 + nonzero_count = nonzero_mask.sum() + + instances_to_sort = instances[nonzero_mask] + points_to_sort = points[nonzero_mask] + + print( + f" Sorting {nonzero_count:,} instance points (of {total_points:,} total)...", + flush=True, + ) + sort_idx = np.argsort(instances_to_sort) + sorted_instances = instances_to_sort[sort_idx] + sorted_points = points_to_sort[sort_idx] + + unique_inst, first_idx, inst_counts = np.unique( + sorted_instances, + return_index=True, + return_counts=True, + ) + print(f" Found {len(unique_inst):,} unique instances.", flush=True) + + hull_candidates = [] + small_volume_instances = [] + small_point_count_instances = [] + large_instances = [] + bbox_skipped_count = 0 + + total_instances = len(unique_inst) + print(f" Categorizing {total_instances:,} instances...", flush=True) + + for idx, (inst_id, start, count) in enumerate(zip(unique_inst, first_idx, inst_counts)): + count = int(count) + + if total_instances >= 1000 and idx % 1000 == 0 and idx > 0: + print(f" {idx:,}/{total_instances:,} instances processed...", flush=True) + + end = start + count + if count >= min_points_for_hull_check: + centroid = sorted_points[start:end].mean(axis=0) + large_instances.append((inst_id, count, centroid)) + continue + + bbox_volume = np.prod( + sorted_points[start:end].max(axis=0) - sorted_points[start:end].min(axis=0) + ) + + if bbox_volume >= max_volume_for_merge * 4.0: + centroid = sorted_points[start:end].mean(axis=0) + if count < min_cluster_size: + small_point_count_instances.append((inst_id, count, centroid)) + bbox_skipped_count += 1 + if verbose: + print( + f" Instance {inst_id}: {count} pts, bbox {bbox_volume:.2f} m3 - " + f"REDISTRIBUTE (sparse, < {min_cluster_size} pts)" + ) + else: + large_instances.append((inst_id, count, centroid)) + bbox_skipped_count += 1 + if verbose: + print( + f" Instance {inst_id}: {count} pts, bbox {bbox_volume:.2f} m3 - " + "keeping (bbox too large, enough points)" + ) + continue + + hull_candidates.append((inst_id, count, start, end, bbox_volume)) + + if verbose: + print( + f" Categorized instances: {len(large_instances):,} large, " + f"{bbox_skipped_count:,} skipped (large bbox), " + f"{len(hull_candidates):,} need hull computation", + flush=True, + ) + + if len(hull_candidates) > 0: + if verbose: + print( + f" Computing centroids and convex hulls for {len(hull_candidates):,} " + f"instances (< {min_points_for_hull_check} points)...", + flush=True, + ) + if num_threads > 1: + print( + f" Using {num_threads} workers (--workers={num_threads}) " + "for parallel processing...", + flush=True, + ) + + hull_args = [] + hull_centroids = [] + for _inst_id, _count, start, end, bbox_volume in hull_candidates: + pts = sorted_points[start:end] + hull_args.append((pts, bbox_volume)) + hull_centroids.append(pts.mean(axis=0)) + + use_parallel = num_threads > 1 and len(hull_candidates) > 10 + if use_parallel: + batch_size = max(100, len(hull_args) // 20) + hull_results = [] + + with ProcessPoolExecutor(max_workers=num_threads) as executor: + for batch_idx in range(0, len(hull_args), batch_size): + batch = hull_args[batch_idx:batch_idx + batch_size] + batch_results = list(executor.map(_compute_hull_wrapper, batch)) + hull_results.extend(batch_results) + + if verbose: + progress = min(100.0, len(hull_results) * 100.0 / len(hull_candidates)) + print( + f" Hull progress: {len(hull_results):,}/" + f"{len(hull_candidates):,} ({progress:.1f}%)...", + flush=True, + ) + else: + if verbose and num_threads == 1: + print(" Using sequential computation (--workers=1 or <10 candidates)...", flush=True) + hull_results = [] + for idx, args in enumerate(hull_args): + hull_results.append(_compute_hull_wrapper(args)) + + if verbose and (idx % 100 == 0 or idx == len(hull_args) - 1): + progress = (idx + 1) * 100.0 / len(hull_candidates) + print( + f" Hull progress: {idx + 1:,}/" + f"{len(hull_candidates):,} ({progress:.1f}%)...", + flush=True, + ) + + if verbose: + print(" Processing hull results and categorizing instances...", flush=True) + + for (inst_id, count, _start, _end, _bbox_volume), (volume, hull_success), centroid in zip( + hull_candidates, + hull_results, + hull_centroids, + ): + if verbose and not hull_success: + print(f" Instance {inst_id}: hull computation failed, using bbox volume") + + if volume < max_volume_for_merge: + small_volume_instances.append((inst_id, count, volume, centroid)) + if verbose: + print( + f" Instance {inst_id}: {count} pts, {volume:.2f} m3 - " + f"SMALL (< {max_volume_for_merge} m3) - merge" + ) + elif count < min_cluster_size: + small_point_count_instances.append((inst_id, count, centroid)) + if verbose: + print( + f" Instance {inst_id}: {count} pts, {volume:.2f} m3 - " + f"REDISTRIBUTE (< {min_cluster_size} pts)" + ) + else: + large_instances.append((inst_id, count, centroid)) + if verbose: + print( + f" Instance {inst_id}: {count} pts, {volume:.2f} m3 - " + "keeping (volume ok, enough points)" + ) + + all_small_instances = small_volume_instances + [ + (inst_id, count, 0.0, centroid) + for inst_id, count, centroid in small_point_count_instances + ] + + if len(all_small_instances) == 0: + print(" No small instances to merge/redistribute", flush=True) + if bbox_skipped_count > 0: + print( + f" Skipped {bbox_skipped_count} instances using bounding box filter " + f"(bbox >= {max_volume_for_merge * 4.0:.1f} m3)", + flush=True, + ) + return (instances, bbox_skipped_count) + + if len(large_instances) == 0: + print(" No large instances to merge into", flush=True) + if bbox_skipped_count > 0: + print( + f" Skipped {bbox_skipped_count} instances using bounding box filter " + f"(bbox >= {max_volume_for_merge * 4.0:.1f} m3)", + flush=True, + ) + return (instances, bbox_skipped_count) + + print( + f" Found {len(small_volume_instances)} small-volume instances " + f"(< {max_volume_for_merge} m3) to merge", + flush=True, + ) + if len(small_point_count_instances) > 0: + print( + f" Found {len(small_point_count_instances)} small point-count instances " + f"(< {min_cluster_size} pts, volume >= {max_volume_for_merge} m3) to redistribute", + flush=True, + ) + if bbox_skipped_count > 0: + print( + f" Skipped {bbox_skipped_count} instances using bounding box filter " + f"(bbox >= {max_volume_for_merge * 4.0:.1f} m3) - saved convex hull computation", + flush=True, + ) + + large_ids = [x[0] for x in large_instances] + large_coords = np.array([x[2] for x in large_instances]) + tree = cKDTree(large_coords) + + max_inst = instances.max() + 1 + inst_to_target = np.arange(max_inst, dtype=np.int32) + + small_centroids = np.array([centroid for _, _, _, centroid in all_small_instances]) + distances, indices = tree.query(small_centroids) + + total_merged = 0 + for i, (inst_id, count, _volume, _centroid) in enumerate(all_small_instances): + distance = distances[i] + idx = indices[i] + + if distance > max_search_radius: + if verbose: + print( + f" x Cluster {inst_id} ({count} pts) - " + f"no target within {max_search_radius}m" + ) + continue + + target_inst = large_ids[idx] + inst_to_target[inst_id] = target_inst + total_merged += count + if verbose: + print(f" + Cluster {inst_id} ({count} pts) -> Instance {target_inst} (dist: {distance:.1f}m)") + + valid_mask = (instances > 0) & (instances < max_inst) + instances[valid_mask] = inst_to_target[instances[valid_mask]] + + print( + f" Merged/redistributed {total_merged:,} points from " + f"{len(all_small_instances)} small instances " + f"({len(small_volume_instances)} small-volume + " + f"{len(small_point_count_instances)} small point-count)", + flush=True, + ) + + return (instances, bbox_skipped_count) diff --git a/src/merge_tile_loading.py b/src/merge_tile_loading.py new file mode 100644 index 0000000..f4ca53f --- /dev/null +++ b/src/merge_tile_loading.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 +"""Tile loading helpers for SmartTile merge.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Dict, Optional, Set, Tuple + +import laspy +import numpy as np + +from instance_labels import validate_prediction_instance_labels +from point_cloud_metadata import point_cloud_source_key +from tile_spatial import compute_tile_bounds, filter_by_centroid_in_buffer + + +@dataclass +class TileData: + """Container for tile point-cloud data used during merge.""" + + name: str + points: np.ndarray + instances: np.ndarray + boundary: Tuple[float, float, float, float] + extra_dims: Dict[str, np.ndarray] = field(default_factory=dict) + + +def merge_tile_name(filepath: Path) -> str: + """Return the logical tile id shared by LAZ/LAS/COPC merge inputs.""" + tile_name = point_cloud_source_key(filepath) + for suffix in ["_segmented_remapped", "_segmented", "_remapped"]: + tile_name = tile_name.replace(suffix, "") + return tile_name + + +def load_tile( + filepath: Path, + all_tiles: Dict[str, Tuple[float, float, float, float]], + buffer: float, + neighbors_by_tile: Optional[Dict[str, Dict[str, Optional[str]]]] = None, + chunk_size: int = 1_000_000, + instance_dimension: str = "PredInstance", +) -> Optional[Tuple[TileData, Set[int], Set[int], Dict[int, str]]]: + """Load one segmented tile and compute buffer-filter metadata.""" + print(f"Loading {filepath.name}...") + + try: + with laspy.open(str(filepath), laz_backend=laspy.LazBackend.Lazrs) as reader: + n_points = reader.header.point_count + header_extra_dims = {dim.name: dim for dim in reader.header.point_format.extra_dimensions} + has_instance_dim = instance_dimension in header_extra_dims + has_tree_id = "treeID" in header_extra_dims + + points = np.empty((n_points, 3), dtype=np.float64) + instances = np.zeros(n_points, dtype=np.int32) + extra_dims: Dict[str, np.ndarray] = {} + for dim in reader.header.point_format.extra_dimensions: + if dim.name == instance_dimension or (not has_instance_dim and dim.name == "treeID"): + continue + extra_dims[dim.name] = np.zeros(n_points, dtype=dim.dtype) + + offset = 0 + for chunk in reader.chunk_iterator(chunk_size): + chunk_len = len(chunk) + end = offset + chunk_len + + points[offset:end, 0] = chunk.x + points[offset:end, 1] = chunk.y + points[offset:end, 2] = chunk.z + + if has_instance_dim: + instances[offset:end] = getattr(chunk, instance_dimension) + elif has_tree_id: + instances[offset:end] = chunk.treeID + + for dim_name in extra_dims: + extra_dims[dim_name][offset:end] = getattr(chunk, dim_name) + + offset = end + except Exception as exc: + print(f" Error loading {filepath}: {exc}") + return None + + if not has_instance_dim and not has_tree_id: + print(f" Warning: No instance attribute ({instance_dimension}/treeID) found in {filepath}") + elif has_instance_dim: + validate_prediction_instance_labels(instances, instance_dimension, filepath) + else: + validate_prediction_instance_labels(instances, "treeID", filepath) + + boundary = compute_tile_bounds(points) + tile_name = merge_tile_name(filepath) + + neighbors_for_tile = neighbors_by_tile.get(tile_name) if neighbors_by_tile is not None else None + instances_to_remove, instance_buffer_direction = filter_by_centroid_in_buffer( + points, + instances, + boundary, + tile_name, + all_tiles, + buffer, + precomputed_neighbors=neighbors_for_tile, + ) + kept_instances = set(np.unique(instances)) - instances_to_remove - {0} + + print( + f" {len(points):,} points, {len(kept_instances)} instances kept, " + f"{len(instances_to_remove)} filtered" + ) + + return ( + TileData( + name=tile_name, + points=points, + instances=instances, + boundary=boundary, + extra_dims=extra_dims, + ), + instances_to_remove, + kept_instances, + instance_buffer_direction, + ) + + +def load_tile_wrapper(args): + """ProcessPool wrapper for load_tile.""" + filepath, tile_boundaries, buffer, neighbors_by_tile, instance_dimension = args + return load_tile(filepath, tile_boundaries, buffer, neighbors_by_tile, instance_dimension=instance_dimension) diff --git a/src/merge_tiles.py b/src/merge_tiles.py index f91cdce..c276b53 100644 --- a/src/merge_tiles.py +++ b/src/merge_tiles.py @@ -21,4316 +21,58 @@ --output-tiles-dir /path/to/output_tiles """ -import argparse import gc -import os import sys -import json -import math -import re -import shutil -import subprocess -import tempfile import numpy as np import laspy from pathlib import Path from typing import Dict, List, Tuple, Set, Optional -from scipy.spatial import cKDTree -from collections import defaultdict -from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed -from dataclasses import dataclass, field +from concurrent.futures import ProcessPoolExecutor +from instance_labels import ( + MERGED_OUTPUT_SCALES, + cast_instances_for_output, + instance_extra_bytes_params, + validate_merged_output_contract, +) +from point_cloud_outputs import ( + merged_product_header, + write_loaded_point_cloud, +) +from merge_deduplication import deduplicate_points +from merge_instance_ids import global_id +from merge_instance_matching import assign_and_match_instances +from merge_loaded_cloud import ( + load_merged_file, + reassign_small_instances_in_dims, +) +from merge_original_dimensions import add_original_dimensions_to_merged +from merge_small_instances import merge_small_volume_instances +from merge_tile_loading import ( + load_tile, + load_tile_wrapper as _load_tile_wrapper, + merge_tile_name, +) +from point_cloud_metadata import point_cloud_files +from output_remap import ( + remap_to_original_input_files, + retile_to_original_files, +) +from tile_spatial import ( + get_tile_bounds_from_header, +) +from tile_bounds_graph import ( + build_neighbor_graph_from_bounds_json, + match_tiles_to_json_bounds, +) # Force unbuffered output for real-time progress feedback # (especially important when running in Docker/containers) sys.stdout.reconfigure(line_buffering=True) -# ============================================================================= -# R/lidR → laspy dimension name mapping -# ============================================================================= -# The standardization JSON (collection_summary.json from tool_standard) can mix -# R/lidR names, LAS 1.2 names, LAS 1.4 names, and already-normalized laspy -# names. Extra-byte dimensions (e.g. Amplitude, Reflectance, Deviation) are -# preserved as-is and fall through unchanged via the .get() default. -# -# In particular, LAS 1.2 `ScanAngleRank` / laspy `scan_angle_rank` and LAS 1.4 -# `ScanAngle` / laspy `scan_angle` should all resolve to the same canonical -# dimension used by the remap path: `scan_angle`. - -_DIMENSION_NAME_ALIASES = { - "Intensity": "intensity", - "intensity": "intensity", - "ReturnNumber": "return_number", - "return_number": "return_number", - "NumberOfReturns": "number_of_returns", - "number_of_returns": "number_of_returns", - "ScanDirectionFlag": "scan_direction_flag", - "scan_direction_flag": "scan_direction_flag", - "EdgeOfFlightline": "edge_of_flight_line", - "edge_of_flight_line": "edge_of_flight_line", - "Classification": "classification", - "classification": "classification", - "ScannerChannel": "scanner_channel", - "scanner_channel": "scanner_channel", - "Synthetic_flag": "synthetic", - "synthetic": "synthetic", - "Keypoint_flag": "key_point", - "key_point": "key_point", - "Withheld_flag": "withheld", - "withheld": "withheld", - "Overlap_flag": "overlap", - "overlap": "overlap", - "ScanAngle": "scan_angle", - "scan_angle": "scan_angle", - "ScanAngleRank": "scan_angle", - "scan_angle_rank": "scan_angle", - "UserData": "user_data", - "user_data": "user_data", - "PointSourceID": "point_source_id", - "point_source_id": "point_source_id", - "gpstime": "gps_time", - "gps_time": "gps_time", - "R": "red", - "red": "red", - "G": "green", - "green": "green", - "B": "blue", - "blue": "blue", -} - -_RGB_STANDARD_DIMS = ("red", "green", "blue") -_COPC_READER_FALLBACK_WARNED: Set[str] = set() - - -def get_pdal_path() -> str: - """Get the path to pdal executable.""" - pdal_path = shutil.which("pdal") - return pdal_path if pdal_path else "pdal" - - -def has_standard_rgb_dims(dim_names) -> bool: - """Return True when all LAS RGB dimensions are present.""" - return all(name in dim_names for name in _RGB_STANDARD_DIMS) - - -def point_format_has_standard_rgb(point_format_id: int) -> bool: - """Return True if the point format includes standard RGB fields.""" - try: - return has_standard_rgb_dims(set(laspy.PointFormat(point_format_id).dimension_names)) - except Exception: - return False - - -def point_format_with_standard_rgb(point_format_id: int) -> int: - """Promote an RGB-less point format to the RGB-capable sibling when possible.""" - if point_format_has_standard_rgb(point_format_id): - return point_format_id - return { - 0: 2, - 1: 3, - 4: 5, - 6: 7, - 9: 10, - }.get(point_format_id, point_format_id) - - -def list_pointcloud_files(input_dir: Path) -> List[Path]: - """List LAS/LAZ/COPC files, preferring COPC over matching plain LAZ.""" - files = sorted(input_dir.glob("*.laz")) + sorted(input_dir.glob("*.las")) - if not files: - return [] - - by_key: Dict[str, Path] = {} - for path in sorted(files): - key = path.name[:-9] if path.name.endswith(".copc.laz") else path.stem - existing = by_key.get(key) - if existing is None or path.name.endswith(".copc.laz"): - by_key[key] = path - return sorted(by_key.values()) - - -def load_standardization_dims(json_path: Path) -> Set[str]: - """Load reference_attribute_names from a collection_summary.json, - convert R/lidR names to laspy names, and return as a set (minus X, Y, Z). - - Dimensions that are constant/all-zero across the entire collection - (variance == 0 or absent from global_attribute_stats) are excluded - automatically, since transferring them would be pointless. - """ - with open(json_path) as f: - data = json.load(f) - collection = data["collection"] - ref_names = collection["reference_attribute_names"] - - # Build set of attribute names that have actual variation - global_stats = collection.get("global_attribute_stats", []) - has_variation = set() - for stat in global_stats: - name = stat.get("name", "") - variance = stat.get("variance", 0) - if variance > 0: - has_variation.add(name) - - result = set() - skipped = [] - for n in ref_names: - if n in ("X", "Y", "Z"): - continue - laspy_name = _DIMENSION_NAME_ALIASES.get(n, n) - if has_variation and n not in has_variation: - skipped.append(n) - continue - result.add(laspy_name) - - if skipped: - print(f" Standardization: skipping {len(skipped)} constant/zero dims: {skipped}", flush=True) - - return result - - -# ============================================================================= -# Data Classes -# ============================================================================= - - -@dataclass -class TileData: - """Container for tile point cloud data.""" - - name: str - points: np.ndarray - instances: np.ndarray - boundary: Tuple[float, float, float, float] # min_x, max_x, min_y, max_y - extra_dims: Dict[str, np.ndarray] = field(default_factory=dict) - - -def normalize_tile_id(stem: str) -> str: - """Return the base tile id by extracting the c##_r## pattern if present.""" - m = re.search(r"c\d+_r\d+", stem) - if m: - return m.group(0) - return re.sub( - r"(?:_segmented_remapped|_segmented|_remapped|_results|_subsampled_[\d.]+(?:cm|m))+$", - "", - stem, - ) - - -def extra_bytes_params_from_dimension_info( - dim_info, - name: Optional[str] = None, -) -> laspy.ExtraBytesParams: - """Build ExtraBytesParams from laspy DimensionInfo while preserving metadata.""" - return laspy.ExtraBytesParams( - name=name or dim_info.name, - type=dim_info.dtype, - description=getattr(dim_info, "description", "") or "", - offsets=getattr(dim_info, "offsets", None), - scales=getattr(dim_info, "scales", None), - no_data=getattr(dim_info, "no_data", None), - ) - - -def _next_available_suffix(base: str, used: set) -> str: - """Return base_1, base_2, ... first not in used. Used to avoid losing dimensions on name collision.""" - for i in range(1, 10000): - cand = f"{base}_{i}" - if cand not in used: - return cand - return f"{base}_9999" - - -def _suffixes_for_collision(base: str, used: set) -> tuple[str, str]: - """Return (name_1, name_2) for original vs merged, per dimension. Suffix is 1/2 per base name, not global.""" - cand_1 = f"{base}_1" - cand_2 = f"{base}_2" - out_1 = cand_1 if cand_1 not in used else _next_available_suffix(base, used) - used.add(out_1) - out_2 = cand_2 if cand_2 not in used else _next_available_suffix(base, used) - used.add(out_2) - return (out_1, out_2) - - -# ============================================================================= -# Union-Find Data Structure -# ============================================================================= - - -class UnionFind: - """ - Union-Find (Disjoint Set) data structure for grouping matched instances. - Tracks instance sizes for species ID preservation. - """ - - def __init__(self): - self.parent = {} - self.rank = {} - self.size = {} # Track size for species preservation - - def make_set(self, x, size: int = 0): - """Create a new set containing only x.""" - if x not in self.parent: - self.parent[x] = x - self.rank[x] = 0 - self.size[x] = size - - def find(self, x) -> int: - """Find the root of the set containing x with path compression.""" - if x not in self.parent: - self.make_set(x) - if self.parent[x] != x: - self.parent[x] = self.find(self.parent[x]) - return self.parent[x] - - def union(self, x, y) -> int: - """ - Merge the sets containing x and y. - Returns the root of the merged set (the larger one by size). - """ - root_x, root_y = self.find(x), self.find(y) - if root_x == root_y: - return root_x - - # Union by size (larger becomes root for species preservation) - if self.size.get(root_x, 0) >= self.size.get(root_y, 0): - self.parent[root_y] = root_x - self.size[root_x] = self.size.get(root_x, 0) + self.size.get(root_y, 0) - return root_x - else: - self.parent[root_x] = root_y - self.size[root_y] = self.size.get(root_x, 0) + self.size.get(root_y, 0) - return root_y - - def get_components(self) -> Dict[int, List[int]]: - """Get all connected components as {root: [members]}.""" - components = defaultdict(list) - for x in self.parent: - root = self.find(x) - components[root].append(x) - return dict(components) - - -# ============================================================================= -# Stage 1: Load and Filter -# ============================================================================= - - -def compute_tile_bounds(points: np.ndarray) -> Tuple[float, float, float, float]: - """Get the XY bounding box of a point cloud.""" - return ( - points[:, 0].min(), - points[:, 0].max(), - points[:, 1].min(), - points[:, 1].max(), - ) - - -def get_tile_bounds_from_header(filepath: Path) -> Optional[Tuple[float, float, float, float]]: - """ - Get spatial bounds of a tile from its header (without loading points). - - Args: - filepath: Path to LAZ file - - Returns: - Tuple of (minx, maxx, miny, maxy) or None on error - """ - try: - with laspy.open(str(filepath), laz_backend=laspy.LazBackend.LazrsParallel) as las: - return (las.header.x_min, las.header.x_max, las.header.y_min, las.header.y_max) - except Exception: - return None - - -# ============================================================================= -# Neighbor graph based on tile_bounds_tindex.json -# ============================================================================= - - -def build_neighbor_graph_from_bounds_json( - tile_bounds_json: Path, - bounds_field: str = "bounds", -) -> Tuple[List[Tuple[float, float, float, float]], List[Tuple[float, float]], List[Dict[str, Optional[int]]]]: - """ - Build a neighbor graph from tile_bounds_tindex.json. - - Args: - tile_bounds_json: Path to the tile bounds JSON. - bounds_field: Preferred bounds field to use for matching/centers. - Falls back to ``bounds`` when the requested field is absent. - - Returns: - json_bounds: list of (minx, maxx, miny, maxy) for each JSON tile - centers: list of (cx, cy) centers for each JSON tile - neighbors_idx: list of dicts {dir -> neighbor_index or None} for each JSON tile - """ - if not tile_bounds_json.exists(): - raise FileNotFoundError(f"tile_bounds_tindex.json not found: {tile_bounds_json}") - - with tile_bounds_json.open() as f: - data = json.load(f) - - tiles = data.get("tiles", []) - if not tiles: - raise ValueError(f"No tiles found in tile bounds JSON: {tile_bounds_json}") - tile_buffer = float(data.get("tile_buffer", 0.0)) - - json_bounds: List[Tuple[float, float, float, float]] = [] - centers: List[Tuple[float, float]] = [] - - for tile in tiles: - if bounds_field in tile: - bx, by = tile[bounds_field] - elif bounds_field == "planned_bounds" and "core" in tile: - core = tile["core"] - bx = [float(core[0][0]) - tile_buffer, float(core[0][1]) + tile_buffer] - by = [float(core[1][0]) - tile_buffer, float(core[1][1]) + tile_buffer] - elif "bounds" in tile: - bx, by = tile["bounds"] - else: - raise ValueError( - f"Tile entry missing bounds field '{bounds_field}' and fallback 'bounds': {tile}" - ) - minx, maxx = float(bx[0]), float(bx[1]) - miny, maxy = float(by[0]), float(by[1]) - json_bounds.append((minx, maxx, miny, maxy)) - cx = (minx + maxx) * 0.5 - cy = (miny + maxy) * 0.5 - centers.append((cx, cy)) - - n = len(json_bounds) - neighbors_idx: List[Dict[str, Optional[int]]] = [ - {"east": None, "west": None, "north": None, "south": None} for _ in range(n) - ] - - # Prefer grid-based neighbor detection when col/row are present (deterministic, correct - # even when bounds are cropped so center_x/center_y differ only slightly between tiles). - col_row_to_idx: Dict[Tuple[int, int], int] = {} - for i, tile in enumerate(tiles): - if "col" in tile and "row" in tile: - col_row_to_idx[(int(tile["col"]), int(tile["row"]))] = i - - if col_row_to_idx: - for i, tile in enumerate(tiles): - if "col" not in tile or "row" not in tile: - continue - c, r = int(tile["col"]), int(tile["row"]) - neighbors_idx[i]["east"] = col_row_to_idx.get((c + 1, r)) - neighbors_idx[i]["west"] = col_row_to_idx.get((c - 1, r)) - neighbors_idx[i]["north"] = col_row_to_idx.get((c, r + 1)) - neighbors_idx[i]["south"] = col_row_to_idx.get((c, r - 1)) - else: - # Fallback: geometry-based neighbor detection (center + overlap) - for i in range(n): - minx_i, maxx_i, miny_i, maxy_i = json_bounds[i] - cx_i, cy_i = centers[i] - - best_east = None # (distance, j) - best_west = None - best_north = None - best_south = None - - for j in range(n): - if i == j: - continue - - minx_j, maxx_j, miny_j, maxy_j = json_bounds[j] - cx_j, cy_j = centers[j] - - overlap_y = not (maxy_i <= miny_j or maxy_j <= miny_i) - overlap_x = not (maxx_i <= minx_j or maxx_j <= minx_i) - - if cx_j > cx_i and overlap_y: - dx = cx_j - cx_i - if best_east is None or dx < best_east[0]: - best_east = (dx, j) - - if cx_j < cx_i and overlap_y: - dx = cx_i - cx_j - if best_west is None or dx < best_west[0]: - best_west = (dx, j) - - if cy_j > cy_i and overlap_x: - dy = cy_j - cy_i - if best_north is None or dy < best_north[0]: - best_north = (dy, j) - - if cy_j < cy_i and overlap_x: - dy = cy_i - cy_j - if best_south is None or dy < best_south[0]: - best_south = (dy, j) - - if best_east is not None: - neighbors_idx[i]["east"] = best_east[1] - if best_west is not None: - neighbors_idx[i]["west"] = best_west[1] - if best_north is not None: - neighbors_idx[i]["north"] = best_north[1] - if best_south is not None: - neighbors_idx[i]["south"] = best_south[1] - - return json_bounds, centers, neighbors_idx - - -def _match_tiles_to_json_bounds( - tile_boundaries: Dict[str, Tuple[float, float, float, float]], - json_bounds: List[Tuple[float, float, float, float]], - centers: List[Tuple[float, float]], - json_labels: Optional[List[str]] = None, -) -> Tuple[Dict[str, int], Dict[int, str]]: - """ - Match loaded tiles (from LAS headers) to JSON tiles using a stepwise strategy: - 1) bounds-based matching (L1 distance) within tolerance - 2) centroid-based matching (Euclidean distance) within tolerance - - Tolerance is increased stepwise until all tiles are matched or max tolerance reached. - Raises ValueError if any tile remains unmatched. - """ - tile_items = list(tile_boundaries.items()) - tile_to_json: Dict[str, int] = {} - json_to_tile: Dict[int, str] = {} - used_json: Set[int] = set() - - # Single-tile shortcut: 1 file and 1 JSON entry -> pair directly - if len(tile_items) == 1 and len(json_bounds) == 1: - name = tile_items[0][0] - tile_to_json[name] = 0 - json_to_tile[0] = name - return tile_to_json, json_to_tile - - # Fast path: canonical tile labels like c00_r00 can be matched directly to - # JSON row/col labels when available. - if json_labels is not None and len(json_labels) == len(json_bounds): - label_to_json = { - label: idx for idx, label in enumerate(json_labels) if label is not None - } - for name, _ in tile_items: - json_idx = label_to_json.get(name) - if json_idx is None or json_idx in used_json: - continue - tile_to_json[name] = json_idx - json_to_tile[json_idx] = name - used_json.add(json_idx) - if len(tile_to_json) == len(tile_boundaries): - return tile_to_json, json_to_tile - - # Tolerance schedule in meters - tolerance_steps = [0.1, 0.5, 1.0, 2.0, 5.0] - - for tol in tolerance_steps: - # Phase 1: bounds-based matching - for name, bounds in tile_items: - if name in tile_to_json: - continue - minx_a, maxx_a, miny_a, maxy_a = bounds - - best_j = None - best_l1 = None - for j, (minx_b, maxx_b, miny_b, maxy_b) in enumerate(json_bounds): - if j in used_json: - continue - if ( - abs(minx_a - minx_b) <= tol - and abs(maxx_a - maxx_b) <= tol - and abs(miny_a - miny_b) <= tol - and abs(maxy_a - maxy_b) <= tol - ): - l1 = ( - abs(minx_a - minx_b) - + abs(maxx_a - maxx_b) - + abs(miny_a - miny_b) - + abs(maxy_a - maxy_b) - ) - if best_l1 is None or l1 < best_l1: - best_l1 = l1 - best_j = j - - if best_j is not None: - tile_to_json[name] = best_j - json_to_tile[best_j] = name - used_json.add(best_j) - - # Phase 2: centroid-based matching for remaining tiles - for name, bounds in tile_items: - if name in tile_to_json: - continue - minx_a, maxx_a, miny_a, maxy_a = bounds - cx_a = (minx_a + maxx_a) * 0.5 - cy_a = (miny_a + maxy_a) * 0.5 - - best_j = None - best_dist = None - for j, (cx_b, cy_b) in enumerate(centers): - if j in used_json: - continue - dx = cx_b - cx_a - dy = cy_b - cy_a - dist = math.hypot(dx, dy) - if dist <= tol and (best_dist is None or dist < best_dist): - best_dist = dist - best_j = j - - if best_j is not None: - tile_to_json[name] = best_j - json_to_tile[best_j] = name - used_json.add(best_j) - - # If all tiles matched, we can stop early - if len(tile_to_json) == len(tile_boundaries): - break - - if len(tile_to_json) != len(tile_boundaries): - unmatched = sorted(set(tile_boundaries.keys()) - set(tile_to_json.keys())) - raise ValueError( - "Failed to match all tiles to entries in tile_bounds_tindex.json. " - f"Unmatched tiles: {', '.join(unmatched)}" - ) - - return tile_to_json, json_to_tile - - -def find_spatial_neighbors( - tile_boundary: Tuple[float, float, float, float], - tile_name: str, - all_tiles: Dict[str, Tuple[float, float, float, float]], # name -> boundary - tolerance: float = 1.0 -) -> Dict[str, Optional[str]]: - """ - Find neighboring tiles based on actual spatial overlaps. - - Detects neighbors by checking if tiles overlap spatially (not just aligned edges). - This handles cases where tiles extend into each other by buffer meters. - - Args: - tile_boundary: (minx, maxx, miny, maxy) of the current tile - tile_name: Name of the current tile - all_tiles: Dictionary mapping tile names to their boundaries - tolerance: Minimum overlap distance to consider tiles as neighbors (default: 1.0m) - - Returns: - Dictionary with 'east', 'west', 'north', 'south' -> neighbor_name or None - """ - minx_a, maxx_a, miny_a, maxy_a = tile_boundary - tile_width_a = maxx_a - minx_a - tile_height_a = maxy_a - miny_a - - neighbors = { - "east": None, - "west": None, - "north": None, - "south": None - } - - # Track overlaps for each direction to pick the best neighbor - east_overlaps = [] # (overlap_area, other_name) - west_overlaps = [] - north_overlaps = [] - south_overlaps = [] - - for other_name, (minx_b, maxx_b, miny_b, maxy_b) in all_tiles.items(): - if other_name == tile_name: - continue - - # Check for actual spatial overlap (not just edge alignment) - overlap = find_overlap_region(tile_boundary, (minx_b, maxx_b, miny_b, maxy_b)) - if overlap is None: - continue - - overlap_minx, overlap_maxx, overlap_miny, overlap_maxy = overlap - overlap_width = overlap_maxx - overlap_minx - overlap_height = overlap_maxy - overlap_miny - overlap_area = overlap_width * overlap_height - - # Determine which edge(s) the overlap is on - # Check if overlap is significant enough (at least tolerance meters) - - # East neighbor: other tile extends to the right (east) of this tile - # The overlap should be on the right side of this tile - # Neighbor must be mostly to the east (right) AND its left edge must overlap/near this tile's right edge - # This ensures we only match true edge neighbors, not tiles that are far away - if minx_b > minx_a and minx_b <= maxx_a + tolerance and overlap_width >= tolerance: - # Check if there's vertical overlap - if not (maxy_b < miny_a or miny_b > maxy_a): - # Calculate alignment: same row = high alignment - # For same row, the vertical overlap should span most of BOTH tiles' heights - # OR if overlap spans most of the smaller tile (indicates same row with size mismatch) - tile_height_b = maxy_b - miny_b - overlap_height_ratio_a = overlap_height / tile_height_a if tile_height_a > 0 else 0.0 - overlap_height_ratio_b = overlap_height / tile_height_b if tile_height_b > 0 else 0.0 - # High alignment if: - # 1. Both tiles have >80% overlap (perfect alignment) - # 2. Exact Y bounds match (perfect alignment) - # 3. Smaller tile has >80% overlap (indicates same row with size mismatch) - max_ratio = max(overlap_height_ratio_a, overlap_height_ratio_b) - y_alignment = 1.0 if (overlap_height_ratio_a > 0.8 and overlap_height_ratio_b > 0.8) or (miny_b == miny_a and maxy_b == maxy_a) or (max_ratio > 0.8) else 0.5 - # Edge alignment: check if Y edges align (for same-row detection) - # Tiles in same row should have miny or maxy very close (within 0.1m tolerance) - edge_tolerance = 0.1 - bottom_edge_align = abs(miny_a - miny_b) < edge_tolerance - top_edge_align = abs(maxy_a - maxy_b) < edge_tolerance - # Higher score if BOTH edges align (perfect same row), medium if one aligns - edge_alignment = 2.0 if (bottom_edge_align and top_edge_align) else (1.0 if (bottom_edge_align or top_edge_align) else 0.0) - # Store (overlap_area, y_alignment, minx_b - minx_a, edge_alignment, other_name) - # Priority: alignment (same row), then overlap area, then distance, then edge alignment - east_overlaps.append((overlap_area, y_alignment, minx_b - minx_a, edge_alignment, other_name)) - - # West neighbor: other tile extends to the left (west) of this tile - # The overlap should be on the left side of this tile - # Neighbor must be mostly to the west (left) AND its right edge must overlap/near this tile's left edge - # This ensures we only match true edge neighbors, not tiles contained within or diagonal overlaps - if minx_b < minx_a and maxx_b >= minx_a - tolerance and overlap_width >= tolerance: - # Check if there's vertical overlap - if not (maxy_b < miny_a or miny_b > maxy_a): - # Calculate alignment: same row = high alignment - # For same row, the vertical overlap should span most of BOTH tiles' heights - # OR if overlap spans most of the smaller tile (indicates same row with size mismatch) - tile_height_b = maxy_b - miny_b - overlap_height_ratio_a = overlap_height / tile_height_a if tile_height_a > 0 else 0.0 - overlap_height_ratio_b = overlap_height / tile_height_b if tile_height_b > 0 else 0.0 - # High alignment if: - # 1. Both tiles have >80% overlap (perfect alignment) - # 2. Exact Y bounds match (perfect alignment) - # 3. Smaller tile has >80% overlap (indicates same row with size mismatch) - max_ratio = max(overlap_height_ratio_a, overlap_height_ratio_b) - y_alignment = 1.0 if (overlap_height_ratio_a > 0.8 and overlap_height_ratio_b > 0.8) or (miny_b == miny_a and maxy_b == maxy_a) or (max_ratio > 0.8) else 0.5 - # Edge alignment: check if Y edges align (for same-row detection) - # Tiles in same row should have miny or maxy very close (within 0.1m tolerance) - edge_tolerance = 0.1 - bottom_edge_align = abs(miny_a - miny_b) < edge_tolerance - top_edge_align = abs(maxy_a - maxy_b) < edge_tolerance - # Higher score if BOTH edges align (perfect same row), medium if one aligns - edge_alignment = 2.0 if (bottom_edge_align and top_edge_align) else (1.0 if (bottom_edge_align or top_edge_align) else 0.0) - # Store (overlap_area, y_alignment, minx_a - minx_b, edge_alignment, other_name) - # Priority: alignment (same row), then overlap area, then distance, then edge alignment - west_overlaps.append((overlap_area, y_alignment, minx_a - minx_b, edge_alignment, other_name)) - - # North neighbor: other tile extends above (north) of this tile - # The overlap should be on the top side of this tile - # Neighbor must be mostly to the north (above) AND its bottom edge must overlap/near this tile's top edge - # This ensures we only match true edge neighbors, not tiles that are far away - if miny_b > miny_a and miny_b <= maxy_a + tolerance and overlap_height >= tolerance: - # Check if there's horizontal overlap - if not (maxx_b < minx_a or minx_b > maxx_a): - # Calculate alignment: same column = high alignment - # For same column, the horizontal overlap should span most of BOTH tiles' widths - # OR if overlap spans most of the smaller tile (indicates same column with size mismatch) - tile_width_b = maxx_b - minx_b - overlap_width_ratio_a = overlap_width / tile_width_a if tile_width_a > 0 else 0.0 - overlap_width_ratio_b = overlap_width / tile_width_b if tile_width_b > 0 else 0.0 - # High alignment if: - # 1. Both tiles have >80% overlap (perfect alignment) - # 2. Exact X bounds match (perfect alignment) - # 3. Smaller tile has >80% overlap (indicates same column with size mismatch) - max_ratio = max(overlap_width_ratio_a, overlap_width_ratio_b) - x_alignment = 1.0 if (overlap_width_ratio_a > 0.8 and overlap_width_ratio_b > 0.8) or (minx_b == minx_a and maxx_b == maxx_a) or (max_ratio > 0.8) else 0.5 - # Edge alignment: check if X edges align (for same-column detection) - # Tiles in same column should have minx or maxx very close (within 0.1m tolerance) - edge_tolerance = 0.1 - left_edge_align = abs(minx_a - minx_b) < edge_tolerance - right_edge_align = abs(maxx_a - maxx_b) < edge_tolerance - # Higher score if BOTH edges align (perfect same column), medium if one aligns - edge_alignment = 2.0 if (left_edge_align and right_edge_align) else (1.0 if (left_edge_align or right_edge_align) else 0.0) - # Store (overlap_area, x_alignment, miny_b - miny_a, edge_alignment, other_name) - # Priority: alignment (same column), then overlap area, then distance, then edge alignment - north_overlaps.append((overlap_area, x_alignment, miny_b - miny_a, edge_alignment, other_name)) - - # South neighbor: other tile extends below (south) of this tile - # The overlap should be on the bottom side of this tile - # Neighbor must be mostly to the south (below) AND its top edge must overlap/near this tile's bottom edge - # This ensures we only match true edge neighbors, not tiles contained within or diagonal overlaps - if miny_b < miny_a and maxy_b >= miny_a - tolerance and overlap_height >= tolerance: - # Check if there's horizontal overlap - if not (maxx_b < minx_a or minx_b > maxx_a): - # Calculate alignment: same column = high alignment - # For same column, the horizontal overlap should span most of BOTH tiles' widths - # OR if overlap spans most of the smaller tile (indicates same column with size mismatch) - tile_width_b = maxx_b - minx_b - overlap_width_ratio_a = overlap_width / tile_width_a if tile_width_a > 0 else 0.0 - overlap_width_ratio_b = overlap_width / tile_width_b if tile_width_b > 0 else 0.0 - # High alignment if: - # 1. Both tiles have >80% overlap (perfect alignment) - # 2. Exact X bounds match (perfect alignment) - # 3. Smaller tile has >80% overlap (indicates same column with size mismatch) - max_ratio = max(overlap_width_ratio_a, overlap_width_ratio_b) - x_alignment = 1.0 if (overlap_width_ratio_a > 0.8 and overlap_width_ratio_b > 0.8) or (minx_b == minx_a and maxx_b == maxx_a) or (max_ratio > 0.8) else 0.5 - # Edge alignment: check if X edges align (for same-column detection) - # Tiles in same column should have minx or maxx very close (within 0.1m tolerance) - edge_tolerance = 0.1 - left_edge_align = abs(minx_a - minx_b) < edge_tolerance - right_edge_align = abs(maxx_a - maxx_b) < edge_tolerance - # Higher score if BOTH edges align (perfect same column), medium if one aligns - edge_alignment = 2.0 if (left_edge_align and right_edge_align) else (1.0 if (left_edge_align or right_edge_align) else 0.0) - # Store (overlap_area, x_alignment, miny_a - miny_b, edge_alignment, other_name) - # Priority: alignment (same column), then overlap area, then distance, then edge alignment - south_overlaps.append((overlap_area, x_alignment, miny_a - miny_b, edge_alignment, other_name)) - - # Pick the neighbor with the best alignment first, then largest overlap, then closest distance, then best edge alignment - # Alignment (same row/column) is prioritized over overlap area to avoid diagonal neighbors - if east_overlaps: - best_east = max(east_overlaps, key=lambda x: (x[1], x[3], x[0], -x[2])) # Alignment, edge alignment, overlap, min distance - neighbors["east"] = best_east[4] # other_name is now at index 4 - # Debug logging - if len(east_overlaps) > 1: - print(f" DEBUG {tile_name} east neighbor: selected {best_east[4]} from {len(east_overlaps)} candidates (overlap: {best_east[0]:.2f} m², alignment: {best_east[1]:.1f}, distance: {best_east[2]:.2f}m)") - if west_overlaps: - best_west = max(west_overlaps, key=lambda x: (x[1], x[3], x[0], -x[2])) # Alignment, edge alignment, overlap, min distance - neighbors["west"] = best_west[4] # other_name is now at index 4 - # Debug logging - if len(west_overlaps) > 1: - print(f" DEBUG {tile_name} west neighbor: selected {best_west[4]} from {len(west_overlaps)} candidates (overlap: {best_west[0]:.2f} m², alignment: {best_west[1]:.1f}, distance: {best_west[2]:.2f}m)") - if north_overlaps: - best_north = max(north_overlaps, key=lambda x: (x[1], x[3], x[0], -x[2])) # Alignment, edge alignment, overlap, min distance - neighbors["north"] = best_north[4] # other_name is now at index 4 - # Debug logging - if len(north_overlaps) > 1: - print(f" DEBUG {tile_name} north neighbor: selected {best_north[4]} from {len(north_overlaps)} candidates (overlap: {best_north[0]:.2f} m², alignment: {best_north[1]:.1f}, distance: {best_north[2]:.2f}m)") - if south_overlaps: - best_south = max(south_overlaps, key=lambda x: (x[1], x[3], x[0], -x[2])) # Alignment, edge alignment, overlap, min distance - neighbors["south"] = best_south[4] # other_name is now at index 4 - # Debug logging - if len(south_overlaps) > 1: - print(f" DEBUG {tile_name} south neighbor: selected {best_south[4]} from {len(south_overlaps)} candidates (overlap: {best_south[0]:.2f} m², alignment: {best_south[1]:.1f}, distance: {best_south[2]:.2f}m)") - - return neighbors - - -def filter_by_centroid_in_buffer( - points: np.ndarray, - instances: np.ndarray, - boundary: Tuple[float, float, float, float], - tile_name: str, - all_tiles: Dict[str, Tuple[float, float, float, float]], - buffer: float = 10.0, - precomputed_neighbors: Optional[Dict[str, Optional[str]]] = None, -) -> Tuple[Set[int], Dict[int, str]]: - """ - Find instances whose centroid is in the buffer zone on overlapping edges. - Uses vectorized centroid computation for efficiency. - - Args: - points: Nx3 array of point coordinates - instances: Array of instance IDs - boundary: (min_x, max_x, min_y, max_y) of the tile - tile_name: Name of the tile (e.g., "c00_r00") - all_tiles: Dictionary mapping tile names to their boundaries for neighbor detection - buffer: Buffer distance from inner edges (also used as minimum overlap to consider) - - Returns: - Tuple of: - - Set of instance IDs to REMOVE (centroid in buffer zone) - - Dict mapping instance ID to buffer direction ('east', 'west', 'north', 'south') - For instances in multiple buffers (corners), uses priority: west > south > east > north - """ - min_x, max_x, min_y, max_y = boundary - - # Determine which edges have neighbors using either precomputed neighbors - # from tile_bounds_tindex.json or spatial bounds as a fallback. - if precomputed_neighbors is not None: - neighbors = { - "east": precomputed_neighbors.get("east"), - "west": precomputed_neighbors.get("west"), - "north": precomputed_neighbors.get("north"), - "south": precomputed_neighbors.get("south"), - } - else: - neighbors = find_spatial_neighbors(boundary, tile_name, all_tiles, tolerance=buffer) - - # Define buffer zone boundaries (only on edges with neighbors) - # Simple approach: buffer meters from each edge that has a neighbor - buf_min_x = min_x + (buffer if neighbors["west"] is not None else 0) - buf_max_x = max_x - (buffer if neighbors["east"] is not None else 0) - buf_min_y = min_y + (buffer if neighbors["south"] is not None else 0) - buf_max_y = max_y - (buffer if neighbors["north"] is not None else 0) - - # Vectorized centroid computation: O(n log n) instead of O(n * k) - # Where n = number of points, k = number of instances - # This provides ~100-250x speedup for typical tiles with many instances - centroids = compute_centroids_vectorized(points, instances) - - # Find instances to remove and track their buffer direction - instances_to_remove = set() - instance_buffer_direction = {} # inst_id -> direction - - for inst_id, centroid in centroids.items(): - if inst_id <= 0: - continue - - cx, cy = centroid[0], centroid[1] - - # Check if centroid is in buffer zone (any direction with a neighbor) - in_west_buffer = neighbors["west"] is not None and cx < buf_min_x - in_east_buffer = neighbors["east"] is not None and cx > buf_max_x - in_south_buffer = neighbors["south"] is not None and cy < buf_min_y - in_north_buffer = neighbors["north"] is not None and cy > buf_max_y - - if in_west_buffer or in_east_buffer or in_south_buffer or in_north_buffer: - instances_to_remove.add(inst_id) - # Priority for corner cases: west/south (don't recover) > east/north (recover) - # If in west or south buffer, neighbor with lower col/row should have it - if in_west_buffer: - instance_buffer_direction[inst_id] = "west" - elif in_south_buffer: - instance_buffer_direction[inst_id] = "south" - elif in_east_buffer: - instance_buffer_direction[inst_id] = "east" - else: # in_north_buffer - instance_buffer_direction[inst_id] = "north" - - return instances_to_remove, instance_buffer_direction - - -def load_tile( - filepath: Path, - all_tiles: Dict[str, Tuple[float, float, float, float]], - buffer: float, - neighbors_by_tile: Optional[Dict[str, Dict[str, Optional[str]]]] = None, - chunk_size: int = 1_000_000, - instance_dimension: str = "PredInstance", -) -> Optional[Tuple[TileData, Set[int], Set[int], Dict[int, str]]]: - """ - Load a LAZ tile using chunked reading for memory efficiency. - - Args: - filepath: Path to the LAZ file - all_tiles: Dictionary mapping tile names to their boundaries for neighbor detection - buffer: Buffer distance for filtering - chunk_size: Number of points to read per chunk (default 1M) - instance_dimension: Name of the instance dimension (default: PredInstance, fallback: treeID) - - Returns: - Tuple of (TileData, instances_to_remove, kept_instances, instance_buffer_direction) or None if loading fails - """ - print(f"Loading {filepath.name}...") - - try: - with laspy.open(str(filepath), laz_backend=laspy.LazBackend.Lazrs) as f: - n_points = f.header.point_count - - header_extra_dims = {dim.name: dim for dim in f.header.point_format.extra_dimensions} - has_instance_dim = instance_dimension in header_extra_dims - has_tree_id = "treeID" in header_extra_dims - - # Pre-allocate arrays - points = np.empty((n_points, 3), dtype=np.float64) - instances = np.zeros(n_points, dtype=np.int32) - - # Pre-allocate generic extra dims (all except the instance dimension) - extra_dims: Dict[str, np.ndarray] = {} - for dim in f.header.point_format.extra_dimensions: - if dim.name == instance_dimension or (not has_instance_dim and dim.name == "treeID"): - continue - extra_dims[dim.name] = np.zeros(n_points, dtype=dim.dtype) - - # Read in chunks to reduce peak memory - offset = 0 - for chunk in f.chunk_iterator(chunk_size): - chunk_len = len(chunk) - end = offset + chunk_len - - points[offset:end, 0] = chunk.x - points[offset:end, 1] = chunk.y - points[offset:end, 2] = chunk.z - - if has_instance_dim: - instances[offset:end] = getattr(chunk, instance_dimension) - elif has_tree_id: - instances[offset:end] = chunk.treeID - - for dim_name in extra_dims: - extra_dims[dim_name][offset:end] = getattr(chunk, dim_name) - - offset = end - - except Exception as e: - print(f" Error loading {filepath}: {e}") - return None - - if not has_instance_dim and not has_tree_id: - print(f" Warning: No instance attribute ({instance_dimension}/treeID) found in {filepath}") - - tile_name = normalize_tile_id(filepath.stem) - - boundary = all_tiles.get(tile_name, compute_tile_bounds(points)) - - neighbors_for_tile = None - if neighbors_by_tile is not None: - neighbors_for_tile = neighbors_by_tile.get(tile_name) - - instances_to_remove, instance_buffer_direction = filter_by_centroid_in_buffer( - points, instances, boundary, tile_name, all_tiles, buffer, precomputed_neighbors=neighbors_for_tile - ) - - kept_instances = set(np.unique(instances)) - instances_to_remove - {0} - - print( - f" {len(points):,} points, {len(kept_instances)} instances kept, {len(instances_to_remove)} filtered" - ) - - return ( - TileData( - name=tile_name, - points=points, - instances=instances, - boundary=boundary, - extra_dims=extra_dims, - ), - instances_to_remove, - kept_instances, - instance_buffer_direction, - ) - - -# ============================================================================= -# Helper function for multiprocessing (must be at module level for pickling) -# ============================================================================= - - -def _load_tile_wrapper(args): - """ - Wrapper function for load_tile to make it pickleable for ProcessPoolExecutor. - - Args: - args: Tuple of (filepath, tile_boundaries, buffer, neighbors_by_tile, instance_dimension) - - Returns: - Result from load_tile() - """ - filepath, tile_boundaries, buffer, neighbors_by_tile, instance_dimension = args - return load_tile(filepath, tile_boundaries, buffer, neighbors_by_tile, instance_dimension=instance_dimension) - - -def _compute_hull_wrapper(args): - """ - Wrapper function for convex hull computation to make it pickleable for ProcessPoolExecutor. - - Args: - args: Tuple of (points, bbox_volume) - - Returns: - Tuple of (volume, success) where success is True if hull computation succeeded - """ - from scipy.spatial import ConvexHull - - points, bbox_volume = args - try: - hull = ConvexHull(points) - return (hull.volume, True) - except Exception: - # If hull fails, use bounding box volume as fallback - return (bbox_volume, False) - - -# ============================================================================= -# Stage 4: Deduplicate (used in Stage 4: Merge and Deduplicate) -# ============================================================================= - - -def deduplicate_points( - points: np.ndarray, - instances: np.ndarray, - extra_dims: Dict[str, np.ndarray], - tolerance: float = 0.01, - grid_size: float = 50.0, -) -> Tuple[np.ndarray, np.ndarray, Dict[str, np.ndarray]]: - """ - Remove duplicate points from overlapping tiles using grid-based processing. - When duplicates exist, keep the one with higher instance ID. - - Uses spatial grid cells to reduce memory usage - instead of sorting billions - of points at once, we process smaller cells independently. - - Args: - points: Nx3 array of point coordinates - instances: Array of instance IDs - extra_dims: Dict of extra dimension name -> array (passenger data) - tolerance: Distance tolerance (default 1cm) - grid_size: Size of spatial grid cells in meters (default 50m) - - Returns: - Tuple of (unique_points, unique_instances, unique_extra_dims) - """ - n_points = len(points) - scale = 1.0 / tolerance - - min_coords = points.min(axis=0) - grid_indices = ((points[:, :2] - min_coords[:2]) / grid_size).astype(np.int32) - - max_grid_y = grid_indices[:, 1].max() + 1 - cell_keys = grid_indices[:, 0] * max_grid_y + grid_indices[:, 1] - - rounded = np.floor(points * scale).astype(np.int64) - - point_hash = rounded[:, 0] + rounded[:, 1] * 73856093 + rounded[:, 2] * 19349669 - - sort_order = np.lexsort((-instances, point_hash, cell_keys)) - - sorted_cell_keys = cell_keys[sort_order] - sorted_point_hash = point_hash[sort_order] - - is_duplicate = np.zeros(n_points, dtype=bool) - is_duplicate[1:] = (sorted_cell_keys[1:] == sorted_cell_keys[:-1]) & ( - sorted_point_hash[1:] == sorted_point_hash[:-1] - ) - - keep_mask = np.ones(n_points, dtype=bool) - keep_mask[sort_order[is_duplicate]] = False - - unique_points = points[keep_mask] - unique_instances = instances[keep_mask] - unique_extras = {name: arr[keep_mask] for name, arr in extra_dims.items()} - - return unique_points, unique_instances, unique_extras - - -# ============================================================================= -# Stage 3: FF3D Instance Matching (Border Region Instance Matching) -# ============================================================================= - - -def find_overlap_region( - bounds_a: Tuple[float, float, float, float], - bounds_b: Tuple[float, float, float, float], -) -> Optional[Tuple[float, float, float, float]]: - """Find the overlap region between two bounding boxes.""" - minx_a, maxx_a, miny_a, maxy_a = bounds_a - minx_b, maxx_b, miny_b, maxy_b = bounds_b - - overlap_minx = max(minx_a, minx_b) - overlap_maxx = min(maxx_a, maxx_b) - overlap_miny = max(miny_a, miny_b) - overlap_maxy = min(maxy_a, maxy_b) - - if overlap_minx < overlap_maxx and overlap_miny < overlap_maxy: - return (overlap_minx, overlap_maxx, overlap_miny, overlap_maxy) - return None - - -def compute_ff3d_overlap_ratios( - instances_a: np.ndarray, - instances_b: np.ndarray, - points_a: np.ndarray, - points_b: np.ndarray, - correspondence_tolerance: float = 0.1, -) -> Dict[Tuple[int, int], float]: - """ - Compute FF3D-style overlap ratios between all instance pairs. - - Only counts points as "same point" if they're within correspondence_tolerance - (should be small, ~10cm, to only match actual duplicate points from overlapping tiles). - - FF3D metric: max(intersection/size_a, intersection/size_b) - More lenient than IoU for asymmetric overlaps. - - Returns: - Dictionary mapping (inst_a, inst_b) pairs to their overlap ratio - - Note: Uses hash-based grid matching instead of KDTree for O(n) instead of O(n log n). - """ - # Vectorized: Count points per instance using np.unique - unique_a, counts_a = np.unique(instances_a[instances_a > 0], return_counts=True) - unique_b, counts_b = np.unique(instances_b[instances_b > 0], return_counts=True) - size_a = dict(zip(unique_a, counts_a)) - size_b = dict(zip(unique_b, counts_b)) - - # Grid-based matching: O(n) instead of KDTree O(n log n) - # Round points to grid cells based on tolerance - scale = 1.0 / correspondence_tolerance - - # Create grid keys for points_b (the lookup table) - grid_b = np.floor(points_b * scale).astype(np.int64) - # Combine x, y, z into single hash key using large primes - hash_b = grid_b[:, 0] + grid_b[:, 1] * 73856093 + grid_b[:, 2] * 19349669 - - # Create grid keys for points_a - grid_a = np.floor(points_a * scale).astype(np.int64) - hash_a = grid_a[:, 0] + grid_a[:, 1] * 73856093 + grid_a[:, 2] * 19349669 - - # Fully vectorized: Get unique grid cells from B with their instance IDs - # Sort by hash to enable searchsorted lookup - sort_idx_b = np.argsort(hash_b) - sorted_hash_b = hash_b[sort_idx_b] - sorted_inst_b = instances_b[sort_idx_b] - - # Get first occurrence of each unique hash (deduplicate grid cells) - unique_hash_b, first_idx = np.unique(sorted_hash_b, return_index=True) - unique_inst_b = sorted_inst_b[first_idx] - - # Find matches: where hash_a exists in unique_hash_b - insert_pos = np.searchsorted(unique_hash_b, hash_a) - - # Clamp to valid range and check for actual matches - insert_pos_clamped = np.clip(insert_pos, 0, len(unique_hash_b) - 1) - matches_mask = unique_hash_b[insert_pos_clamped] == hash_a - - # Get matched instance IDs - matched_inst_b = np.zeros(len(hash_a), dtype=instances_b.dtype) - matched_inst_b[matches_mask] = unique_inst_b[insert_pos_clamped[matches_mask]] - - # Filter to valid matches with positive instance IDs - valid_mask = matches_mask & (instances_a > 0) & (matched_inst_b > 0) - valid_inst_a = instances_a[valid_mask] - valid_inst_b = matched_inst_b[valid_mask] - - # Create unique pair keys and count occurrences - if len(valid_inst_a) > 0: - max_inst = max(instances_a.max(), instances_b.max()) + 1 - pair_keys = valid_inst_a.astype(np.int64) * max_inst + valid_inst_b.astype( - np.int64 - ) - unique_pairs, pair_counts = np.unique(pair_keys, return_counts=True) - - # Decode back to instance pairs - intersection_counts = {} - for key, count in zip(unique_pairs, pair_counts): - inst_a_val = int(key // max_inst) - inst_b_val = int(key % max_inst) - intersection_counts[(inst_a_val, inst_b_val)] = count - else: - intersection_counts = {} - - # Compute FF3D overlap ratio for each pair - overlap_ratios = {} - for (inst_a, inst_b), intersection in intersection_counts.items(): - ratio_a = intersection / size_a[inst_a] if size_a.get(inst_a, 0) > 0 else 0 - ratio_b = intersection / size_b[inst_b] if size_b.get(inst_b, 0) > 0 else 0 - # Use mutual overlap rather than one-sided containment. - overlap_ratios[(inst_a, inst_b)] = min(ratio_a, ratio_b) - - return overlap_ratios, size_a, size_b - - -def compute_centroids_vectorized( - points: np.ndarray, instances: np.ndarray -) -> Dict[int, np.ndarray]: - """ - Compute centroids for all instances using vectorized operations. - - Instead of looping over each instance and scanning the full array, - this uses sorting and cumulative sums for O(n log n) instead of O(n * k). - - Args: - points: Nx3 array of point coordinates - instances: Array of instance IDs - - Returns: - Dictionary mapping instance_id -> centroid (3D array) - """ - # Filter to positive instances - valid_mask = instances > 0 - valid_points = points[valid_mask] - valid_instances = instances[valid_mask] - - if len(valid_instances) == 0: - return {} - - # Sort by instance ID - sort_idx = np.argsort(valid_instances) - sorted_instances = valid_instances[sort_idx] - sorted_points = valid_points[sort_idx] - - # Find boundaries between different instances - unique_instances, first_indices, counts = np.unique( - sorted_instances, return_index=True, return_counts=True - ) - - # Compute cumulative sums for efficient mean calculation - cumsum = np.zeros((len(sorted_points) + 1, 3), dtype=np.float64) - cumsum[1:] = np.cumsum(sorted_points, axis=0) - - # Calculate centroids using cumulative sum differences - centroids = {} - for i, (inst_id, start_idx, count) in enumerate( - zip(unique_instances, first_indices, counts) - ): - end_idx = start_idx + count - centroid = (cumsum[end_idx] - cumsum[start_idx]) / count - centroids[int(inst_id)] = centroid - - return centroids - - -def get_border_region_mask( - points: np.ndarray, - boundary: Tuple[float, float, float, float], - inner_dist: float, - outer_dist: float, - neighbors: Dict[str, Optional[str]], -) -> np.ndarray: - """ - Get mask for points in the donut-shaped border region. - - Only considers edges that have neighbors (no point checking for edges without neighbors). - - Args: - points: Nx3 array of point coordinates - boundary: (min_x, max_x, min_y, max_y) tile bounds - inner_dist: Inner distance from edge (buffer zone boundary) - outer_dist: Outer distance from edge (end of border region) - neighbors: Dict with 'east', 'west', 'north', 'south' -> neighbor name or None - - Returns: - Boolean mask for points in border region - """ - min_x, max_x, min_y, max_y = boundary - x, y = points[:, 0], points[:, 1] - - # Start with no points selected - in_border = np.zeros(len(points), dtype=bool) - - # West border region: inner_dist <= (x - min_x) < outer_dist - if neighbors.get("west") is not None: - west_mask = (x >= min_x + inner_dist) & (x < min_x + outer_dist) - in_border |= west_mask - - # East border region: inner_dist <= (max_x - x) < outer_dist - if neighbors.get("east") is not None: - east_mask = (x > max_x - outer_dist) & (x <= max_x - inner_dist) - in_border |= east_mask - - # South border region: inner_dist <= (y - min_y) < outer_dist - if neighbors.get("south") is not None: - south_mask = (y >= min_y + inner_dist) & (y < min_y + outer_dist) - in_border |= south_mask - - # North border region: inner_dist <= (max_y - y) < outer_dist - if neighbors.get("north") is not None: - north_mask = (y > max_y - outer_dist) & (y <= max_y - inner_dist) - in_border |= north_mask - - return in_border - - -# ============================================================================= -# Stage 5: Small Cluster Reassignment / Volume Merging -# ============================================================================= - - -def merge_small_volume_instances( - points: np.ndarray, - instances: np.ndarray, - min_points_for_hull_check: int = 1000, - min_cluster_size: int = 300, - max_volume_for_merge: float = 4.0, - max_search_radius: float = 5.0, - num_threads: int = 1, - verbose: bool = True, - presorted_points: Optional[np.ndarray] = None, - presorted_instances: Optional[np.ndarray] = None, - presorted_unique_inst: Optional[np.ndarray] = None, - presorted_first_idx: Optional[np.ndarray] = None, - presorted_inst_counts: Optional[np.ndarray] = None, -) -> Tuple[np.ndarray, int]: - """ - Merge small-volume instances to nearest large instance by centroid distance. - - Only reassigns instance IDs. Extra dims are untouched (values stay with their points). - - Logic: - - For instances with >= min_points_for_hull_check (1000) points: Keep instance - - For instances with < min_points_for_hull_check (1000) points: - 1. Calculate convex hull volume - 2. If volume < max_volume_for_merge (4.0 m³): Merge to nearest large instance - 3. Else if point_count < min_cluster_size: Redistribute to nearest instance - 4. Else: Keep instance - - Args: - points: Nx3 array of point coordinates - instances: Array of instance IDs (modified in-place) - min_points_for_hull_check: Only compute convex hull for instances with fewer points than this (default: 1000) - min_cluster_size: Redistribute instances with fewer points than this if volume >= threshold (default: 300) - max_volume_for_merge: Merge instances with convex hull volume below this (m³) (default: 4.0) - max_search_radius: Max distance to search for target instance (m) (default: 5.0) - num_threads: Number of workers for parallel hull computation (default: 1) - verbose: Print detailed decisions (default: True) - presorted_points: Pre-sorted points array (optional, to avoid redundant sorting) - presorted_instances: Pre-sorted instances array (optional) - presorted_unique_inst: Pre-computed unique instances (optional) - presorted_first_idx: Pre-computed first indices (optional) - presorted_inst_counts: Pre-computed instance counts (optional) - - Returns: - Updated (instances, bbox_skipped_count) - """ - import sys - - # Check if pre-sorted data is provided - use_presorted = ( - presorted_points is not None and - presorted_instances is not None and - presorted_unique_inst is not None and - presorted_first_idx is not None and - presorted_inst_counts is not None - ) - - if use_presorted: - sorted_points = presorted_points - sorted_instances = presorted_instances - unique_inst = presorted_unique_inst - first_idx = presorted_first_idx - inst_counts = presorted_inst_counts - print(f" {len(unique_inst):,} unique instances (pre-sorted).", flush=True) - else: - total_points = len(instances) - nonzero_mask = instances > 0 - nonzero_count = nonzero_mask.sum() - - instances_to_sort = instances[nonzero_mask] - points_to_sort = points[nonzero_mask] - - print(f" Sorting {nonzero_count:,} instance points (of {total_points:,} total)...", flush=True) - sort_idx = np.argsort(instances_to_sort) - sorted_instances = instances_to_sort[sort_idx] - sorted_points = points_to_sort[sort_idx] - - unique_inst, first_idx, inst_counts = np.unique( - sorted_instances, return_index=True, return_counts=True - ) - print(f" Found {len(unique_inst):,} unique instances.", flush=True) - - # Categorize instances and compute volumes using sorted slices - # Optimization: Only compute convex hull for instances < min_points_for_hull_check (1000 points) - # Optimization: Only compute centroids for instances < 1000 points after bbox filtering - # First pass: collect candidates that need hull computation (after bbox filter) - hull_candidates = [] # (inst_id, count, start, end, bbox_volume) - no centroid/points yet - small_volume_instances = [] # (inst_id, point_count, volume, centroid) - small_point_count_instances = [] # (inst_id, point_count, centroid) - for redistribution - large_instances = [] # (inst_id, count, centroid) - bbox_skipped_count = 0 # Track instances skipped by bounding box filter - - total_instances = len(unique_inst) - print(f" Categorizing {total_instances:,} instances...", flush=True) - - for idx, (inst_id, start, count) in enumerate(zip(unique_inst, first_idx, inst_counts)): - count = int(count) - - if total_instances >= 1000 and idx % 1000 == 0 and idx > 0: - print(f" {idx:,}/{total_instances:,} instances processed...", flush=True) - - # Skip large instances (>= min_points_for_hull_check) - they're kept (no hull computation) - if count >= min_points_for_hull_check: - end = start + count - centroid = sorted_points[start:end].mean(axis=0) - large_instances.append((inst_id, count, centroid)) - continue - - # For instances < 1000 points: compute bbox volume without extracting points array - # This avoids memory copies for instances that will be filtered out - end = start + count - # Compute bbox volume directly from slice (no need to copy points array) - bbox_volume = np.prod( - sorted_points[start:end].max(axis=0) - sorted_points[start:end].min(axis=0) - ) - - # If bounding box volume is already too large, skip convex hull computation - # But still check point count - sparse large-bbox instances should be redistributed - if ( - bbox_volume >= max_volume_for_merge * 4.0 - ): # Conservative threshold (bbox >= 4x target) - centroid = sorted_points[start:end].mean(axis=0) - - # Check if instance has too few points - redistribute sparse noise - if count < min_cluster_size: - small_point_count_instances.append((inst_id, count, centroid)) - bbox_skipped_count += 1 - if verbose: - print( - f" Instance {inst_id}: {count} pts, bbox {bbox_volume:.2f} m³ - REDISTRIBUTE (sparse, < {min_cluster_size} pts)" - ) - else: - # Large bbox with enough points - keep as large instance - large_instances.append((inst_id, count, centroid)) - bbox_skipped_count += 1 - if verbose: - print( - f" Instance {inst_id}: {count} pts, bbox {bbox_volume:.2f} m³ - keeping (bbox too large, enough points)" - ) - continue - - # Collect candidates for hull computation (< min_points_for_hull_check, passed bbox filter) - # Store indices instead of points/centroid to avoid memory overhead - # We'll compute centroids only for instances that pass hull computation - hull_candidates.append((inst_id, count, start, end, bbox_volume)) - - if verbose: - print(f" Categorized instances: {len(large_instances):,} large, {bbox_skipped_count:,} skipped (large bbox), {len(hull_candidates):,} need hull computation", flush=True) - - # Parallel convex hull computation for candidates using --workers (num_threads) - # Now compute centroids and extract points only for hull candidates - if len(hull_candidates) > 0: - if verbose: - print(f" Computing centroids and convex hulls for {len(hull_candidates):,} instances (< {min_points_for_hull_check} points)...", flush=True) - if num_threads > 1: - print(f" Using {num_threads} workers (--workers={num_threads}) for parallel processing...", flush=True) - - # Extract points and compute centroids only for hull candidates - # This avoids memory overhead for instances filtered by bbox - hull_args = [] - hull_centroids = [] - for inst_id, count, start, end, bbox_volume in hull_candidates: - pts = sorted_points[start:end] # Only extract points for hull candidates - centroid = pts.mean(axis=0) # Only compute centroids for hull candidates - hull_args.append((pts, bbox_volume)) - hull_centroids.append(centroid) - - # Use --workers (num_threads) for parallelization - # Parallelize if num_threads > 1 and we have enough candidates - use_parallel = num_threads > 1 and len(hull_candidates) > 10 - if use_parallel: - # Parallel computation using ProcessPoolExecutor with progress updates - batch_size = max(100, len(hull_args) // 20) # ~20 progress updates - hull_results = [] - - with ProcessPoolExecutor(max_workers=num_threads) as executor: - for batch_idx in range(0, len(hull_args), batch_size): - batch = hull_args[batch_idx:batch_idx + batch_size] - batch_results = list(executor.map(_compute_hull_wrapper, batch)) - hull_results.extend(batch_results) - - if verbose: - progress = min(100.0, (len(hull_results) * 100.0 / len(hull_candidates))) - print(f" Hull progress: {len(hull_results):,}/{len(hull_candidates):,} ({progress:.1f}%)...", flush=True) - else: - # Sequential computation for small batches or single thread with progress updates - if verbose and num_threads == 1: - print(f" Using sequential computation (--workers=1 or <10 candidates)...", flush=True) - hull_results = [] - for idx, args in enumerate(hull_args): - result = _compute_hull_wrapper(args) - hull_results.append(result) - - if verbose and (idx % 100 == 0 or idx == len(hull_args) - 1): - progress = (idx + 1) * 100.0 / len(hull_candidates) - print(f" Hull progress: {idx + 1:,}/{len(hull_candidates):,} ({progress:.1f}%)...", flush=True) - - # Process hull computation results - if verbose: - print(f" Processing hull results and categorizing instances...", flush=True) - - for (inst_id, count, start, end, bbox_volume), (volume, hull_success), centroid in zip(hull_candidates, hull_results, hull_centroids): - if verbose and not hull_success: - print(f" Instance {inst_id}: hull computation failed, using bbox volume") - - # Check volume threshold - if volume < max_volume_for_merge: - # Small volume - merge to nearest instance - small_volume_instances.append((inst_id, count, volume, centroid)) - if verbose: - print( - f" Instance {inst_id}: {count} pts, {volume:.2f} m³ - SMALL (< {max_volume_for_merge} m³) - merge" - ) - else: - # Volume >= threshold - check point count for redistribution - if count < min_cluster_size: - # Small point count - redistribute to nearest instance - small_point_count_instances.append((inst_id, count, centroid)) - if verbose: - print( - f" Instance {inst_id}: {count} pts, {volume:.2f} m³ - REDISTRIBUTE (< {min_cluster_size} pts)" - ) - else: - # Keep instance - large_instances.append((inst_id, count, centroid)) - if verbose: - print( - f" Instance {inst_id}: {count} pts, {volume:.2f} m³ - keeping (volume ok, enough points)" - ) - - # Combine small volume and small point count instances for merging/redistribution - all_small_instances = small_volume_instances + [(inst_id, count, 0.0, centroid) - for inst_id, count, centroid in small_point_count_instances] - - if len(all_small_instances) == 0: - print(f" No small instances to merge/redistribute", flush=True) - if bbox_skipped_count > 0: - print( - f" Skipped {bbox_skipped_count} instances using bounding box filter (bbox >= {max_volume_for_merge * 4.0:.1f} m³)", flush=True - ) - return (instances, bbox_skipped_count) - - if len(large_instances) == 0: - print(f" No large instances to merge into", flush=True) - if bbox_skipped_count > 0: - print( - f" Skipped {bbox_skipped_count} instances using bounding box filter (bbox >= {max_volume_for_merge * 4.0:.1f} m³)", flush=True - ) - return (instances, bbox_skipped_count) - - print( - f" Found {len(small_volume_instances)} small-volume instances (< {max_volume_for_merge} m³) to merge", flush=True - ) - if len(small_point_count_instances) > 0: - print( - f" Found {len(small_point_count_instances)} small point-count instances (< {min_cluster_size} pts, volume >= {max_volume_for_merge} m³) to redistribute", flush=True - ) - if bbox_skipped_count > 0: - print( - f" Skipped {bbox_skipped_count} instances using bounding box filter (bbox >= {max_volume_for_merge * 4.0:.1f} m³) - saved convex hull computation", flush=True - ) - - # Build KD-tree from large instance centroids (already computed) - large_ids = [x[0] for x in large_instances] - large_sizes = {x[0]: x[1] for x in large_instances} - large_coords = np.array([x[2] for x in large_instances]) - tree = cKDTree(large_coords) - - # Build lookup table for vectorized reassignment (instance IDs only) - max_inst = instances.max() + 1 - inst_to_target = np.arange(max_inst, dtype=np.int32) # Default: map to self - - # Determine targets for small instances - if len(all_small_instances) > 0: - small_centroids = np.array( - [centroid for _, _, _, centroid in all_small_instances] - ) - distances, indices = tree.query(small_centroids) - - total_merged = 0 - for i, (inst_id, count, volume, centroid) in enumerate(all_small_instances): - distance = distances[i] - idx = indices[i] - - if distance > max_search_radius: - if verbose: - print(f" ✗ Cluster {inst_id} ({count} pts) - no target within {max_search_radius}m") - continue - - target_inst = large_ids[idx] - inst_to_target[inst_id] = target_inst - - total_merged += count - if verbose: - print(f" ✓ Cluster {inst_id} ({count} pts) → Instance {target_inst} (dist: {distance:.1f}m)") - else: - total_merged = 0 - - # Vectorized reassignment - only instance IDs; extra dims stay with their points - valid_mask = (instances > 0) & (instances < max_inst) - instances[valid_mask] = inst_to_target[instances[valid_mask]] - - print( - f" Merged/redistributed {total_merged:,} points from {len(all_small_instances)} small instances " - f"({len(small_volume_instances)} small-volume + {len(small_point_count_instances)} small point-count)", flush=True - ) - - return (instances, bbox_skipped_count) - - -# ============================================================================= -# Load Existing Merged File -# ============================================================================= - - -def load_merged_file( - merged_file: Path, - chunk_size: int = 1_000_000, -) -> Tuple[np.ndarray, Dict[str, np.ndarray], Dict[str, laspy.ExtraBytesParams]]: - """ - Load merged point cloud data from an existing LAZ file. - All dimensions except X,Y,Z are returned in one dict (standard + extra; no special "instance" dimension). - Used by remap_to_original_input_files so that standard dims (intensity, etc.) can be transferred too. - - Args: - merged_file: Path to the merged LAZ file - chunk_size: Number of points to read per chunk (default 1M) - - Returns: - Tuple of (points, all_dims, extra_dim_params) where all_dims contains every - dimension in the file except X,Y,Z and extra_dim_params preserves metadata - for extra dimensions present in the merged file. - """ - print(f"Loading existing merged file: {merged_file}") - - try: - with laspy.open( - str(merged_file), laz_backend=laspy.LazBackend.LazrsParallel - ) as f: - n_points = f.header.point_count - points = np.empty((n_points, 3), dtype=np.float64) - all_dims: Dict[str, np.ndarray] = {} - extra_dim_params: Dict[str, laspy.ExtraBytesParams] = {} - offset = 0 - for chunk in f.chunk_iterator(chunk_size): - chunk_len = len(chunk) - end = offset + chunk_len - points[offset:end, 0] = chunk.x - points[offset:end, 1] = chunk.y - points[offset:end, 2] = chunk.z - # Standard dimensions (except X,Y,Z) - for dim_name in f.header.point_format.dimension_names: - if dim_name in ("X", "Y", "Z"): - continue - arr = getattr(chunk, dim_name, None) - if arr is not None: - if dim_name not in all_dims: - all_dims[dim_name] = np.zeros(n_points, dtype=arr.dtype) - all_dims[dim_name][offset:end] = arr - # Extra dimensions - for dim in f.header.point_format.extra_dimensions: - if dim.name not in extra_dim_params: - extra_dim_params[dim.name] = extra_bytes_params_from_dimension_info(dim) - if dim.name not in all_dims: - all_dims[dim.name] = np.zeros(n_points, dtype=dim.dtype) - all_dims[dim.name][offset:end] = getattr(chunk, dim.name) - offset = end - - print(f" Loaded {len(points):,} points") - if all_dims: - print(f" Dimensions from merged: {', '.join(sorted(all_dims.keys()))}") - - return points, all_dims, extra_dim_params - - except Exception as e: - raise ValueError(f"Error loading merged file {merged_file}: {e}") - - -def _stream_merged_subset( - merged_file: Path, - bounds: Tuple[float, float, float, float], - spatial_buffer: float, - chunk_size: int = 1_000_000, - dim_filter: Optional[Set[str]] = None, -) -> Tuple[np.ndarray, Dict[str, np.ndarray], Dict[str, laspy.ExtraBytesParams]]: - """Stream a merged LAZ file and collect only points within *bounds* + buffer. - - Returns the same format as ``load_merged_file`` — (points, all_dims, - extra_dim_params) — but only for the spatial subset. Peak memory is - bounded by the subset size, not the full file. - - Args: - merged_file: Path to merged LAZ. - bounds: (xmin, xmax, ymin, ymax) of the region of interest. - spatial_buffer: Extra padding in metres around *bounds*. - chunk_size: Points per streaming chunk. - dim_filter: If provided, only collect these dimension names (plus - XYZ which are always collected). Reduces memory when callers - only need a subset of dimensions (e.g. remap needs only - 3DTrees dims, not all standard dims). - """ - xmin, xmax, ymin, ymax = bounds - x_lo, x_hi = xmin - spatial_buffer, xmax + spatial_buffer - y_lo, y_hi = ymin - spatial_buffer, ymax + spatial_buffer - - pts_x: List[np.ndarray] = [] - pts_y: List[np.ndarray] = [] - pts_z: List[np.ndarray] = [] - dim_lists: Dict[str, List[np.ndarray]] = defaultdict(list) - extra_dim_params: Dict[str, laspy.ExtraBytesParams] = {} - - with laspy.open(str(merged_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: - n_merged = reader.header.point_count - std_dim_names = [ - n for n in reader.header.point_format.dimension_names - if n not in ("X", "Y", "Z") - and (dim_filter is None or n in dim_filter) - ] - for dim in reader.header.point_format.extra_dimensions: - extra_dim_params[dim.name] = extra_bytes_params_from_dimension_info(dim) - - extra_dim_names_to_read = [ - n for n in extra_dim_params - if dim_filter is None or n in dim_filter - ] - - scanned = 0 - kept_total = 0 - n_chunks_est = (n_merged + chunk_size - 1) // chunk_size if chunk_size else 1 - for chunk in reader.chunk_iterator(chunk_size): - cx = np.asarray(chunk.x) - cy = np.asarray(chunk.y) - mask = (cx >= x_lo) & (cx <= x_hi) & (cy >= y_lo) & (cy <= y_hi) - scanned += len(cx) - if not np.any(mask): - # Print progress every ~20% of chunks - pct = scanned / n_merged * 100 if n_merged else 100 - if int(pct) % 20 < int(len(cx) / n_merged * 100) + 1: - print(f" {pct:.0f}%", end="", flush=True) - continue - kept = int(mask.sum()) - kept_total += kept - pts_x.append(cx[mask]) - pts_y.append(cy[mask]) - pts_z.append(np.asarray(chunk.z)[mask]) - for dim_name in std_dim_names: - arr = getattr(chunk, dim_name, None) - if arr is not None: - dim_lists[dim_name].append(np.asarray(arr)[mask]) - for dim_name in extra_dim_names_to_read: - arr = getattr(chunk, dim_name, None) - if arr is not None: - dim_lists[dim_name].append(np.asarray(arr)[mask]) - pct = scanned / n_merged * 100 if n_merged else 100 - if int(pct) % 20 < int(len(cx) / n_merged * 100) + 1: - print(f" {pct:.0f}%", end="", flush=True) - - if not pts_x: - return np.empty((0, 3), dtype=np.float64), {}, extra_dim_params - - points = np.column_stack([ - np.concatenate(pts_x), np.concatenate(pts_y), np.concatenate(pts_z), - ]) - all_dims = {name: np.concatenate(arrs) for name, arrs in dim_lists.items() if arrs} - return points, all_dims, extra_dim_params - - -def _write_points_with_dimensions_to_laz( - output_path: Path, - points: np.ndarray, - dims: Dict[str, np.ndarray], -) -> Path: - """Write XYZ plus named dimensions to a LAZ file.""" - if len(points) == 0: - raise ValueError("Cannot write a LAZ file from an empty point set") - - promote_rgb_to_standard = has_standard_rgb_dims(dims.keys()) - point_format_id = 7 if promote_rgb_to_standard else 6 - - header = laspy.LasHeader(point_format=point_format_id, version="1.4") - header.offsets = np.min(points, axis=0) - header.scales = np.array([0.001, 0.001, 0.001]) - - output_las = laspy.LasData(header) - output_las.x = points[:, 0] - output_las.y = points[:, 1] - output_las.z = points[:, 2] - - extra_dims_params = [] - for dim_name, dim_arr in dims.items(): - if promote_rgb_to_standard and dim_name in _RGB_STANDARD_DIMS: - continue - extra_dims_params.append(laspy.ExtraBytesParams(name=dim_name, type=dim_arr.dtype)) - if extra_dims_params: - output_las.add_extra_dims(extra_dims_params) - - for dim_name, dim_arr in dims.items(): - setattr(output_las, dim_name, dim_arr) - - output_path.parent.mkdir(parents=True, exist_ok=True) - output_las.write( - str(output_path), - do_compress=True, - laz_backend=laspy.LazBackend.LazrsParallel, - ) - del output_las - return output_path - - -def _build_spatial_slices( - bounds: Tuple[float, float, float, float], - slice_count: int, - slice_length: Optional[float] = None, - target_points: Optional[int] = None, - total_points: Optional[int] = None, -) -> List[Tuple[Tuple[float, float, float, float], str, bool]]: - """Split XY bounds into thin spatial slices along the longer axis. - - Returns ``[(slice_bounds, axis, include_upper_edge), ...]`` where - ``include_upper_edge`` is True only for the final slice along the chosen axis. - """ - xmin, xmax, ymin, ymax = bounds - x_span = max(0.0, float(xmax) - float(xmin)) - y_span = max(0.0, float(ymax) - float(ymin)) - if (slice_count <= 1 and not slice_length and not target_points) or (x_span == 0.0 and y_span == 0.0): - return [(bounds, "x", True)] - - axis = "x" if x_span >= y_span else "y" - span = x_span if axis == "x" else y_span - if span == 0.0: - return [(bounds, axis, True)] - - if target_points is not None and int(target_points) > 0 and total_points is not None and int(total_points) > 0: - n_slices = max(1, int(np.ceil(float(total_points) / float(target_points)))) - elif slice_length is not None and float(slice_length) > 0: - n_slices = max(1, int(np.ceil(span / float(slice_length)))) - else: - n_slices = max(1, int(slice_count)) - step = span / n_slices - slices: List[Tuple[Tuple[float, float, float, float], str, bool]] = [] - for idx in range(n_slices): - is_last = idx == n_slices - 1 - start = (xmin if axis == "x" else ymin) + idx * step - stop = (xmax if axis == "x" else ymax) if is_last else start + step - if axis == "x": - slice_bounds = (start, stop, ymin, ymax) - else: - slice_bounds = (xmin, xmax, start, stop) - slices.append((slice_bounds, axis, is_last)) - return slices - - -def _snap_spatial_slices_to_header_grid( - slice_specs: List[Tuple[Tuple[float, float, float, float], str, bool]], - header, -) -> List[Tuple[Tuple[float, float, float, float], str, bool]]: - """Snap slice boundaries to the point coordinate grid defined by a LAS header. - - This avoids tiny floating-point drift like ``585910.0800000001`` that can - leave exact quantized coordinates outside both neighboring slices. - """ - scales = tuple(float(v) for v in getattr(header, "scales", (0.0, 0.0, 0.0))) - offsets = tuple(float(v) for v in getattr(header, "offsets", (0.0, 0.0, 0.0))) - if len(scales) < 2 or len(offsets) < 2: - return slice_specs - - def _snap(value: float, axis_idx: int) -> float: - scale = scales[axis_idx] - offset = offsets[axis_idx] - if not np.isfinite(value) or scale <= 0: - return float(value) - raw = round((float(value) - offset) / scale) - return float(offset + raw * scale) - - snapped = [] - for slice_bounds, axis, include_upper in slice_specs: - xmin, xmax, ymin, ymax = slice_bounds - if axis == "x": - xmin = _snap(xmin, 0) - xmax = _snap(xmax, 0) - else: - ymin = _snap(ymin, 1) - ymax = _snap(ymax, 1) - snapped.append(((xmin, xmax, ymin, ymax), axis, include_upper)) - return snapped - - -def _load_copc_subset_all_dims( - path: Path, - bounds: Tuple[float, float, float, float], - halo: float = 1.0, - work_dir: Optional[Path] = None, -) -> Tuple[Optional[laspy.LasData], int]: - """Load a spatial COPC subset with all dimensions. - - Prefer ``laspy.CopcReader`` for in-memory subset reads. Fall back to the - older PDAL -> temporary LAS path if direct COPC reads fail for a file. - """ - xmin, xmax, ymin, ymax = bounds - x_lo = xmin - halo - x_hi = xmax + halo - y_lo = ymin - halo - y_hi = ymax + halo - - try: - with laspy.CopcReader.open(str(path)) as reader: - query_bounds = laspy.copc.Bounds( - mins=np.array([x_lo, y_lo], dtype=np.float64), - maxs=np.array([x_hi, y_hi], dtype=np.float64), - ) - subset = reader.spatial_query(query_bounds) - subset_count = len(subset) - return (subset, subset_count) if subset_count > 0 else (None, 0) - except Exception as exc: - warn_key = str(path) - if warn_key not in _COPC_READER_FALLBACK_WARNED: - print( - f" CopcReader fallback for {path.name}: {exc}", - flush=True, - ) - _COPC_READER_FALLBACK_WARNED.add(warn_key) - - work_root = work_dir if work_dir is not None else Path(tempfile.gettempdir()) - work_root.mkdir(parents=True, exist_ok=True) - - with tempfile.NamedTemporaryFile( - mode="w", suffix=".json", delete=False, dir=work_root, - ) as pf: - pipeline_file = Path(pf.name) - subset_file = pipeline_file.with_suffix(".las") - pipeline = { - "pipeline": [ - { - "type": "readers.copc", - "filename": str(path), - "bounds": f"([{x_lo},{x_hi}],[{y_lo},{y_hi}])", - }, - { - "type": "writers.las", - "filename": str(subset_file), - "forward": "all", - "extra_dims": "all", - "minor_version": 4, - }, - ] - } - json.dump(pipeline, pf) - - try: - result = subprocess.run( - [get_pdal_path(), "pipeline", str(pipeline_file)], - capture_output=True, - text=True, - check=False, - ) - if result.returncode != 0: - raise RuntimeError(result.stderr.strip() or result.stdout.strip() or "PDAL pipeline failed") - with laspy.open(str(subset_file), laz_backend=laspy.LazBackend.Lazrs) as reader: - if reader.header.point_count == 0: - return None, 0 - subset = laspy.read(str(subset_file), laz_backend=laspy.LazBackend.LazrsParallel) - return subset, len(subset.points) - finally: - for tmp_path in (pipeline_file, subset_file): - try: - if tmp_path.exists(): - tmp_path.unlink() - except OSError: - pass - - -# ============================================================================= -# Stage 6 (streaming): Retile merged → original tile grid -# ============================================================================= - - -def retile_to_original_files_streaming( - merged_file: Path, - original_tiles_dir: Path, - output_dir: Path, - tolerance: float = 0.1, - retile_buffer: float = 2.0, - chunk_size: int = 1_000_000, - instance_dimension: str = "PredInstance", - threedtrees_dims: Optional[List[str]] = None, - threedtrees_suffix: str = "SAT", -): - """Fully streaming retiling — neither merged nor original files loaded fully. - - For each original tile: - 1. Stream merged LAZ → collect spatial subset (for KDTree) - 2. Build KDTree from merged subset - 3. Stream original tile in chunks, query KDTree, write output chunks - - Peak RAM ≈ merged spatial subset + KDTree + one original chunk. - """ - import gc - - print(f"\n{'=' * 60}", flush=True) - print("Retiling merged results to original tile files (streaming)", flush=True) - print(f"{'=' * 60}", flush=True) - - original_files = sorted(original_tiles_dir.glob("*.laz")) - if not original_files: - original_files = sorted(original_tiles_dir.glob("*.las")) - if not original_files: - print(f" No LAZ/LAS files found in {original_tiles_dir}", flush=True) - return - - print(f" Found {len(original_files)} original tile files", flush=True) - output_dir.mkdir(parents=True, exist_ok=True) - - spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer - threedtrees_set = set(threedtrees_dims) if threedtrees_dims else {instance_dimension} - suffix = threedtrees_suffix or "SAT" - - # Get merged extra-dim metadata once - with laspy.open(str(merged_file), laz_backend=laspy.LazBackend.LazrsParallel) as mf: - merged_extra_dim_params: Dict[str, laspy.ExtraBytesParams] = { - dim.name: extra_bytes_params_from_dimension_info(dim) - for dim in mf.header.point_format.extra_dimensions - } - - tiles_to_process = [] - skipped = 0 - for orig_file in original_files: - output_name = orig_file.name.replace(".copc.laz", ".laz") - output_file = output_dir / output_name - if output_file.exists(): - skipped += 1 - else: - tiles_to_process.append((orig_file, output_file)) - - if skipped > 0: - print(f" Skipping {skipped} already processed tiles", flush=True) - if not tiles_to_process: - print(f" All tiles already processed!", flush=True) - return - - import time as _time - print(f" Processing {len(tiles_to_process)} tiles...", flush=True) - - for ti, (orig_file, output_file) in enumerate(tiles_to_process): - try: - tile_start = _time.monotonic() - print(f" [{ti+1}/{len(tiles_to_process)}] {orig_file.name}...", flush=True) - - # 1. Get original tile header info (no points loaded) - with laspy.open(str(orig_file), laz_backend=laspy.LazBackend.LazrsParallel) as f: - tile_bounds = (f.header.x_min, f.header.x_max, f.header.y_min, f.header.y_max) - n_orig = f.header.point_count - orig_pf = f.header.point_format - orig_version = f.header.version - orig_offsets = f.header.offsets - orig_scales = f.header.scales - orig_extra_dims = list(orig_pf.extra_dimensions) - - # 2. Stream merged file — collect only this tile's subset - print(f" Collecting merged spatial subset...", end="", flush=True) - local_merged_points, local_all_dims, _ = _stream_merged_subset( - merged_file, tile_bounds, spatial_buffer, chunk_size, - ) - print(f" {len(local_merged_points):,} pts", flush=True) - if len(local_merged_points) == 0: - print(f" No merged points in tile region — skipping", flush=True) - continue - - local_merged_instances = local_all_dims.pop( - instance_dimension, - np.zeros(len(local_merged_points), dtype=np.int32), - ) - local_merged_extras = local_all_dims - - # 3. Build KDTree from merged subset - print(f" Building KDTree...", end="", flush=True) - local_tree = cKDTree(local_merged_points) - del local_merged_points - print(" done", flush=True) - - # 4. Build output header - out_header = laspy.LasHeader( - point_format=orig_pf.id, version=orig_version, - ) - out_header.offsets = orig_offsets - out_header.scales = orig_scales - - # Branded names for merged dims - inst_out_name = ( - f"3DT_{instance_dimension}_{suffix}" - if instance_dimension in threedtrees_set - else instance_dimension - ) - merged_rename: Dict[str, str] = {instance_dimension: inst_out_name} - for dim_name in local_merged_extras: - if dim_name in threedtrees_set: - merged_rename[dim_name] = f"3DT_{dim_name}_{suffix}" - else: - merged_rename[dim_name] = dim_name - - # Add extra dims to output header - out_extra_names = set(out_header.point_format.dimension_names) - extra_to_add = [] - for dim in orig_extra_dims: - if dim.name not in out_extra_names: - extra_to_add.append(extra_bytes_params_from_dimension_info(dim, name=dim.name)) - out_extra_names.add(dim.name) - if inst_out_name not in out_extra_names: - extra_to_add.append(laspy.ExtraBytesParams(name=inst_out_name, type=np.int32)) - out_extra_names.add(inst_out_name) - for dim_name in local_merged_extras: - out_name = merged_rename[dim_name] - if out_name not in out_extra_names: - if dim_name in merged_extra_dim_params: - params = merged_extra_dim_params[dim_name] - extra_to_add.append(laspy.ExtraBytesParams(name=out_name, type=params.type)) - else: - extra_to_add.append(laspy.ExtraBytesParams( - name=out_name, type=local_merged_extras[dim_name].dtype, - )) - out_extra_names.add(out_name) - if extra_to_add: - out_header.add_extra_dims(extra_to_add) - - # Pre-compute which original dims to copy - orig_dim_names_to_copy = [ - d for d in orig_pf.dimension_names if d in out_extra_names - ] - - # 5. Stream original tile + write output chunk by chunk - unique_instances: set = set() - output_file.parent.mkdir(parents=True, exist_ok=True) - - with laspy.open(str(orig_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: - with laspy.open( - str(output_file), mode="w", header=out_header, - do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel, - ) as writer: - for chunk in reader.chunk_iterator(chunk_size): - n = len(chunk) - orig_pts = np.column_stack([chunk.x, chunk.y, chunk.z]) - distances, indices = local_tree.query(orig_pts, workers=-1) - del orig_pts - - point_record = laspy.ScaleAwarePointRecord.zeros(n, header=out_header) - # Copy all original dims from chunk - for dim_name in orig_dim_names_to_copy: - try: - setattr(point_record, dim_name, getattr(chunk, dim_name)) - except Exception: - pass - # Set merged dims - setattr(point_record, inst_out_name, local_merged_instances[indices]) - for dim_name, arr in local_merged_extras.items(): - setattr(point_record, merged_rename[dim_name], arr[indices]) - - writer.write_points(point_record) - - inst_chunk = local_merged_instances[indices] - inst_pos = inst_chunk[inst_chunk > 0] - if len(inst_pos) > 0: - unique_instances.update(np.unique(inst_pos).tolist()) - - del local_tree, local_merged_instances, local_merged_extras - gc.collect() - - tile_dt = _time.monotonic() - tile_start - print( - f" Done: {n_orig:,} pts, " - f"{len(unique_instances)} instances, {tile_dt:.1f}s", - flush=True, - ) - - except Exception as e: - print(f" [{ti+1}/{len(tiles_to_process)}] FAILED: {e} → {orig_file.name}", flush=True) - - print(f"\n ✓ Retiling complete: {len(tiles_to_process)} tiles processed (streaming)", flush=True) - gc.collect() - - -# ============================================================================= -# Stage 7 (streaming): Remap merged → original input files -# ============================================================================= - - -def remap_to_original_input_files_streaming( - merged_file: Path, - original_input_dir: Path, - output_dir: Path, - tolerance: float = 0.1, - retile_buffer: float = 2.0, - chunk_size: int = 1_000_000, - spatial_slices: int = 50, - spatial_chunk_length: Optional[float] = None, - spatial_target_points: Optional[int] = None, - threedtrees_dims: Optional[List[str]] = None, - threedtrees_suffix: str = "SAT", - target_dims: Optional[Set[str]] = None, -): - """Fully streaming remap — neither merged nor original files loaded fully. - - For each original input file: - 1. Stream merged LAZ → collect spatial subset (for KDTree) - 2. Build KDTree from merged subset - 3. Stream original file in chunks, query KDTree, write output chunks - - Peak RAM ≈ merged spatial subset + KDTree + one original chunk. - """ - import gc - - print(f"\n{'=' * 60}", flush=True) - print("Remapping to original input files (streaming)", flush=True) - print(f"{'=' * 60}", flush=True) - - original_files = list_pointcloud_files(original_input_dir) - if not original_files: - print(f" No original input files found in {original_input_dir}", flush=True) - return - - print(f" Found {len(original_files)} original input files", flush=True) - output_dir.mkdir(parents=True, exist_ok=True) - - if threedtrees_dims is None: - threedtrees_dims_list = ["PredInstance", "PredSemantic"] - else: - threedtrees_dims_list = list(threedtrees_dims) - threedtrees_dims_set = set(threedtrees_dims_list) - - # Get merged extra-dim metadata once - with laspy.open(str(merged_file), laz_backend=laspy.LazBackend.LazrsParallel) as mf: - merged_extra_dim_params: Dict[str, laspy.ExtraBytesParams] = { - dim.name: extra_bytes_params_from_dimension_info(dim) - for dim in mf.header.point_format.extra_dimensions - } - - files_to_process = [] - skipped = 0 - for input_file in original_files: - output_name = input_file.name.replace(".copc.laz", ".laz") - output_file = output_dir / output_name - if output_file.exists(): - skipped += 1 - else: - files_to_process.append((input_file, output_file)) - - if skipped > 0: - print(f" Skipping {skipped} already processed files", flush=True) - if not files_to_process: - print(f" All files already processed!", flush=True) - return - - print(f" Processing {len(files_to_process)} files...", flush=True) - - keep_original_dim = ( - lambda name: target_dims is None - or name in target_dims - or name in {"X", "Y", "Z", "x", "y", "z"} - ) - - total_matched = 0 - total_points = 0 - - import time as _time - stage_start = _time.monotonic() - - with tempfile.TemporaryDirectory(prefix="3dtrees_stage7_") as tmp_dir_str: - tmp_dir = Path(tmp_dir_str) - - for fi, (input_file, output_file) in enumerate(files_to_process): - try: - file_start = _time.monotonic() - print(f" [{fi+1}/{len(files_to_process)}] {input_file.name}...", flush=True) - - # 1. Get original file header info (no points loaded) - with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as f: - file_bounds = (f.header.x_min, f.header.x_max, f.header.y_min, f.header.y_max) - n_input = f.header.point_count - orig_pf = f.header.point_format - orig_version = f.header.version - orig_offsets = f.header.offsets - orig_scales = f.header.scales - orig_extra_dims = list(orig_pf.extra_dimensions) - print(f" Header: {n_input:,} pts, bounds x=[{file_bounds[0]:.1f}, {file_bounds[1]:.1f}] y=[{file_bounds[2]:.1f}, {file_bounds[3]:.1f}]", flush=True) - - branded_names = {} - for dim_name in threedtrees_dims_set: - if dim_name in merged_extra_dim_params: - branded_names[dim_name] = ( - f"3DT_{dim_name}_{threedtrees_suffix}" - if threedtrees_suffix - else f"3DT_{dim_name}" - ) - - out_header = laspy.LasHeader( - point_format=orig_pf.id, version=orig_version, - ) - out_header.offsets = orig_offsets - out_header.scales = orig_scales - - out_extra_names = set(out_header.point_format.dimension_names) - extra_to_add = [] - for dim in orig_extra_dims: - if keep_original_dim(dim.name) and dim.name not in out_extra_names: - extra_to_add.append(extra_bytes_params_from_dimension_info(dim)) - out_extra_names.add(dim.name) - for dim_name, out_name in branded_names.items(): - if out_name not in out_extra_names: - if dim_name in merged_extra_dim_params: - params = merged_extra_dim_params[dim_name] - extra_to_add.append(laspy.ExtraBytesParams(name=out_name, type=params.type)) - out_extra_names.add(out_name) - if extra_to_add: - out_header.add_extra_dims(extra_to_add) - - orig_dims_to_copy = [ - d for d in orig_pf.dimension_names - if keep_original_dim(d) and d in out_extra_names - ] - - unique_instances: set = set() - output_file.parent.mkdir(parents=True, exist_ok=True) - written = 0 - merged_dims_to_read = set(branded_names.keys()) - - with laspy.open( - str(output_file), mode="w", header=out_header, - do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel, - ) as writer: - if input_file.name.endswith(".copc.laz"): - slice_specs = _build_spatial_slices( - file_bounds, - spatial_slices, - spatial_chunk_length, - spatial_target_points, - n_input, - ) - slice_specs = _snap_spatial_slices_to_header_grid(slice_specs, in_header) - print( - f" Spatial slicing: {len(slice_specs)} slices along {slice_specs[0][1].upper()}", - flush=True, - ) - - for slice_idx, (slice_bounds, axis, include_upper) in enumerate(slice_specs, start=1): - subset, subset_count = _load_copc_subset_all_dims( - input_file, - slice_bounds, - halo=1.0, - work_dir=tmp_dir, - ) - if subset is None or subset_count == 0: - print(f" Slice {slice_idx}/{len(slice_specs)}: empty original subset", flush=True) - continue - - sx = np.asarray(subset.x) - sy = np.asarray(subset.y) - sz = np.asarray(subset.z) - if axis == "x": - upper_mask = sx <= slice_bounds[1] if include_upper else sx < slice_bounds[1] - core_mask = (sx >= slice_bounds[0]) & upper_mask & (sy >= slice_bounds[2]) & (sy <= slice_bounds[3]) - else: - upper_mask = sy <= slice_bounds[3] if include_upper else sy < slice_bounds[3] - core_mask = (sx >= slice_bounds[0]) & (sx <= slice_bounds[1]) & (sy >= slice_bounds[2]) & upper_mask - if not np.any(core_mask): - print(f" Slice {slice_idx}/{len(slice_specs)}: no core points after halo crop", flush=True) - del subset - continue - - orig_xyz = np.column_stack([sx[core_mask], sy[core_mask], sz[core_mask]]) - local_pts, local_dims, _ = _stream_merged_subset( - merged_file, - slice_bounds, - 0.0, - chunk_size, - dim_filter=merged_dims_to_read, - ) - - point_record = laspy.ScaleAwarePointRecord.zeros(len(orig_xyz), header=out_header) - for dim_name in orig_dims_to_copy: - arr = getattr(subset, dim_name, None) - if arr is not None: - setattr(point_record, dim_name, np.asarray(arr)[core_mask]) - - if len(local_pts) > 0: - local_tree = cKDTree(local_pts) - _, indices = local_tree.query(orig_xyz, workers=-1) - del local_tree - for dim_name, out_name in branded_names.items(): - if dim_name in local_dims: - vals = local_dims[dim_name][indices] - setattr(point_record, out_name, vals) - if np.issubdtype(vals.dtype, np.integer): - pos = vals[vals > 0] - if len(pos) > 0: - unique_instances.update(np.unique(pos).tolist()) - del local_pts, local_dims - - writer.write_points(point_record) - written += len(orig_xyz) - pct = written / n_input * 100 if n_input else 100 - elapsed = _time.monotonic() - file_start - rate = written / elapsed if elapsed > 0 else 0 - print( - f" Slice {slice_idx}/{len(slice_specs)}: {len(orig_xyz):,} core pts from {subset_count:,} in-bounds ({pct:.0f}%, {rate:,.0f} pts/s)", - flush=True, - ) - del point_record, orig_xyz, subset, sx, sy, sz - gc.collect() - else: - n_chunks_expected = (n_input + chunk_size - 1) // chunk_size - spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer - print( - f" Non-COPC fallback: streaming {n_input:,} pts in ~{n_chunks_expected} chunks", - flush=True, - ) - - with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: - chunk_i = 0 - for chunk in reader.chunk_iterator(chunk_size): - n = len(chunk) - orig_xyz = np.column_stack([chunk.x, chunk.y, chunk.z]) - chunk_xmin = float(np.min(orig_xyz[:, 0])) - chunk_xmax = float(np.max(orig_xyz[:, 0])) - chunk_ymin = float(np.min(orig_xyz[:, 1])) - chunk_ymax = float(np.max(orig_xyz[:, 1])) - chunk_bounds = (chunk_xmin, chunk_xmax, chunk_ymin, chunk_ymax) - - local_pts, local_dims, _ = _stream_merged_subset( - merged_file, - chunk_bounds, - spatial_buffer, - chunk_size, - dim_filter=merged_dims_to_read, - ) - - point_record = laspy.ScaleAwarePointRecord.zeros(n, header=out_header) - for dim_name in orig_dims_to_copy: - try: - setattr(point_record, dim_name, getattr(chunk, dim_name)) - except Exception: - pass - - if len(local_pts) > 0: - local_tree = cKDTree(local_pts) - _, indices = local_tree.query(orig_xyz, workers=-1) - del local_tree, local_pts - for dim_name, out_name in branded_names.items(): - if dim_name in local_dims: - vals = local_dims[dim_name][indices] - setattr(point_record, out_name, vals) - if np.issubdtype(vals.dtype, np.integer): - pos = vals[vals > 0] - if len(pos) > 0: - unique_instances.update(np.unique(pos).tolist()) - del local_dims - - writer.write_points(point_record) - written += n - chunk_i += 1 - pct = written / n_input * 100 if n_input else 100 - elapsed = _time.monotonic() - file_start - rate = written / elapsed if elapsed > 0 else 0 - print( - f" Chunk {chunk_i}/{n_chunks_expected}: {n:,} pts ({pct:.0f}%, {rate:,.0f} pts/s)", - flush=True, - ) - del orig_xyz - - gc.collect() - - total_matched += n_input - total_points += n_input - file_dt = _time.monotonic() - file_start - print( - f" Done: {n_input:,} pts, {len(unique_instances)} instances, {file_dt:.1f}s", - flush=True, - ) - - except Exception as e: - print(f" [{fi+1}/{len(files_to_process)}] FAILED: {e} → {input_file.name}", flush=True) - - overall_pct = (total_matched / total_points * 100) if total_points > 0 else 0 - print( - f"\n ✓ Remap complete: {len(files_to_process)} files, " - f"{total_matched:,}/{total_points:,} matched ({overall_pct:.1f}%)", - flush=True, - ) - - if files_to_process and total_matched > 0: - first_input, first_output = files_to_process[0] - if first_output.exists(): - _validate_common_dimensions_minmax(first_input, first_output) - - gc.collect() - - -# ============================================================================= -# Legacy compatibility shims -# ============================================================================= - - -def retile_to_original_files( - merged_points: np.ndarray, - merged_instances: np.ndarray, - merged_extra_dims: Dict[str, np.ndarray], - merged_extra_dim_params: Optional[Dict[str, laspy.ExtraBytesParams]], - original_tiles_dir: Path, - output_dir: Path, - tolerance: float = 0.1, - num_threads: int = 8, - chunk_size: int = 1_000_000, - parallel_tiles: int = 1, - retile_buffer: float = 2.0, - instance_dimension: str = "PredInstance", - threedtrees_dims: Optional[List[str]] = None, - threedtrees_suffix: str = "SAT", -): - """Compatibility shim that materializes a temp LAZ and delegates to streaming.""" - del merged_extra_dim_params, num_threads, parallel_tiles - import tempfile - - with tempfile.TemporaryDirectory(prefix="3dtrees_retile_") as tmp_dir: - merged_laz = Path(tmp_dir) / "merged_for_retile.laz" - _write_points_with_dimensions_to_laz( - merged_laz, - merged_points, - {instance_dimension: merged_instances, **merged_extra_dims}, - ) - retile_to_original_files_streaming( - merged_file=merged_laz, - original_tiles_dir=original_tiles_dir, - output_dir=output_dir, - tolerance=tolerance, - retile_buffer=retile_buffer, - chunk_size=chunk_size, - instance_dimension=instance_dimension, - threedtrees_dims=threedtrees_dims, - threedtrees_suffix=threedtrees_suffix, - ) - - -def _validate_common_dimensions_minmax(original_path: Path, output_path: Path, rel_tol: float = 1e-5) -> None: - """Compare min/max of common dimensions between original and output; warn if they differ (rounding/loss).""" - try: - orig = laspy.read(str(original_path), laz_backend=laspy.LazBackend.LazrsParallel) - out = laspy.read(str(output_path), laz_backend=laspy.LazBackend.LazrsParallel) - except Exception as e: - print(f" Validation skip: could not read files ({e})", flush=True) - return - try: - orig_names = set(orig.point_format.dimension_names) | {d.name for d in orig.point_format.extra_dimensions} - out_names = set(out.point_format.dimension_names) | {d.name for d in out.point_format.extra_dimensions} - common = orig_names & out_names - {"X", "Y", "Z"} - if not common: - return - diffs = [] - for name in sorted(common): - oa = getattr(orig, name, None) - oo = getattr(out, name, None) - if oa is None or oo is None or len(oa) != len(oo): - continue - oa, oo = np.asarray(oa), np.asarray(oo) - omin, omax = float(np.min(oa)), float(np.max(oa)) - wmin, wmax = float(np.min(oo)), float(np.max(oo)) - if np.issubdtype(oa.dtype, np.integer) and np.issubdtype(oo.dtype, np.integer): - if (omin != wmin or omax != wmax) and (int(omin) != int(wmin) or int(omax) != int(wmax)): - diffs.append((name, omin, omax, wmin, wmax)) - else: - span = max(omax - omin, 1e-12) - if abs(omin - wmin) > rel_tol * span or abs(omax - wmax) > rel_tol * span: - diffs.append((name, omin, omax, wmin, wmax)) - if diffs: - print(f" Warning: common dimensions min/max differ (output may have rounding/loss):", flush=True) - for name, omin, omax, wmin, wmax in diffs: - print(f" {name}: original [{omin}, {omax}] vs output [{wmin}, {wmax}]", flush=True) - print(f" Tip: dimensions above were not overwritten by merged where range would be lost.", flush=True) - del orig - del out - except Exception as e: - print(f" Validation skip: {e}", flush=True) - - -def remap_to_original_input_files( - merged_points: np.ndarray, - merged_extra_dims: Dict[str, np.ndarray], - merged_extra_dim_params: Optional[Dict[str, laspy.ExtraBytesParams]], - original_input_dir: Path, - output_dir: Path, - tolerance: float = 0.1, - num_threads: int = 8, - retile_buffer: float = 2.0, - threedtrees_dims: Optional[List[str]] = None, - threedtrees_suffix: str = "SAT", - target_dims: Optional[Set[str]] = None, -): - """Compatibility shim that materializes a temp LAZ and delegates to streaming.""" - del merged_extra_dim_params, num_threads - import tempfile - - with tempfile.TemporaryDirectory(prefix="3dtrees_remap_") as tmp_dir: - merged_laz = Path(tmp_dir) / "merged_for_originals.laz" - _write_points_with_dimensions_to_laz( - merged_laz, - merged_points, - merged_extra_dims, - ) - remap_to_original_input_files_streaming( - merged_file=merged_laz, - original_input_dir=original_input_dir, - output_dir=output_dir, - tolerance=tolerance, - retile_buffer=retile_buffer, - threedtrees_dims=threedtrees_dims, - threedtrees_suffix=threedtrees_suffix, - target_dims=target_dims, - ) - - -def _dims_to_fill_from_source( - source_dims: Dict[str, np.dtype], - target_dim_names: Set[str], - get_target_array, - skip: Optional[Set[str]] = None, -) -> Tuple[Dict[str, np.dtype], Dict[str, np.dtype]]: - """ - Shared logic for "which dimensions to fill from source": add new (in source not in target) - and overwrite (in target but empty/constant, use source). Used by add_original_dimensions_to_merged - and by remap_to_original_input_files. - Returns (dims_to_add_new, dims_to_overwrite), both name -> dtype. - """ - skip = skip or {"X", "Y", "Z"} - dims_to_add_new = {k: v for k, v in source_dims.items() if k not in target_dim_names and k not in skip} - dims_to_overwrite: Dict[str, np.dtype] = {} - for dim_name in (target_dim_names & set(source_dims.keys())) - skip: - arr = get_target_array(dim_name) - if arr is None: - continue - a = np.asarray(arr) - if np.min(a) == np.max(a) or np.count_nonzero(a) == 0: - dims_to_overwrite[dim_name] = source_dims[dim_name] - return dims_to_add_new, dims_to_overwrite - - -class _OriginalFileCache: - """LRU cache for original-file spatial subsets (KDTrees + dim arrays). - - Caches spatially cropped subsets keyed by file + XY window. When the - merged file is streamed chunk-by-chunk, consecutive chunks are spatially - close, so the same cropped subsets are often reused. The LRU policy evicts - the least-recently-used entry when the cache is full, keeping peak RAM - bounded. - """ - - def __init__(self, max_entries: int = 3, tmp_dir: Optional[Path] = None, chunk_size: int = 2_000_000) -> None: - from collections import OrderedDict - self._cache: 'OrderedDict[Tuple[Path, float, float, float, float], Tuple[Optional[cKDTree], Dict[str, np.ndarray]]]' = OrderedDict() - self._max = max_entries - self._tmp_dir = tmp_dir - self._chunk_size = chunk_size - - def _load_copc_subset_with_reader( - self, - path: Path, - dims_to_read: Dict[str, str], - dim_dtypes: Dict[str, np.dtype], - x_lo: float, - x_hi: float, - y_lo: float, - y_hi: float, - ) -> Tuple[Optional[np.ndarray], Dict[str, np.ndarray], float]: - """Read a bounded COPC subset directly with laspy.CopcReader.""" - import time as _time - - read_start = _time.monotonic() - with laspy.CopcReader.open(str(path)) as reader: - query_bounds = laspy.copc.Bounds( - mins=np.array([x_lo, y_lo], dtype=np.float64), - maxs=np.array([x_hi, y_hi], dtype=np.float64), - ) - points = reader.spatial_query(query_bounds) - - if points is None or len(points) == 0: - return None, {}, _time.monotonic() - read_start - - xyz = np.column_stack([ - np.asarray(points.x), - np.asarray(points.y), - np.asarray(points.z), - ]) - dim_arrays: Dict[str, np.ndarray] = {} - for out_name, orig_name in dims_to_read.items(): - arr = getattr(points, orig_name, None) - if arr is not None: - dtype = dim_dtypes.get(out_name, np.float64) - dim_arrays[out_name] = np.asarray(arr).astype(dtype, copy=False) - - return xyz, dim_arrays, _time.monotonic() - read_start - - def _load_copc_subset_with_pdal( - self, - path: Path, - dims_to_read: Dict[str, str], - dim_dtypes: Dict[str, np.dtype], - x_lo: float, - x_hi: float, - y_lo: float, - y_hi: float, - ) -> Tuple[Optional[np.ndarray], Dict[str, np.ndarray], float]: - """Read a bounded COPC subset via PDAL and return XYZ + requested dims.""" - import time as _time - - work_dir = self._tmp_dir if self._tmp_dir is not None else Path(tempfile.gettempdir()) - work_dir.mkdir(parents=True, exist_ok=True) - read_start = _time.monotonic() - - with tempfile.NamedTemporaryFile( - mode="w", suffix=".json", delete=False, dir=work_dir, - ) as pf: - pipeline_file = Path(pf.name) - subset_file = pipeline_file.with_suffix(".las") - pipeline = { - "pipeline": [ - { - "type": "readers.copc", - "filename": str(path), - "bounds": f"([{x_lo},{x_hi}],[{y_lo},{y_hi}])", - }, - { - "type": "writers.las", - "filename": str(subset_file), - "forward": "all", - "extra_dims": "all", - "minor_version": 4, - }, - ] - } - json.dump(pipeline, pf) - - try: - result = subprocess.run( - [get_pdal_path(), "pipeline", str(pipeline_file)], - capture_output=True, text=True, check=False, - ) - if result.returncode != 0: - raise RuntimeError(result.stderr.strip() or result.stdout.strip() or "PDAL pipeline failed") - - with laspy.open(str(subset_file), laz_backend=laspy.LazBackend.Lazrs) as f: - n_pts = f.header.point_count - if n_pts == 0: - return None, {}, _time.monotonic() - read_start - points = f.read() - - xyz = np.column_stack([ - np.asarray(points.x), - np.asarray(points.y), - np.asarray(points.z), - ]) - dim_arrays: Dict[str, np.ndarray] = {} - for out_name, orig_name in dims_to_read.items(): - arr = getattr(points, orig_name, None) - if arr is not None: - dtype = dim_dtypes.get(out_name, np.float64) - dim_arrays[out_name] = np.asarray(arr).astype(dtype, copy=False) - - return xyz, dim_arrays, _time.monotonic() - read_start - finally: - for tmp_path in (pipeline_file, subset_file): - try: - if tmp_path.exists(): - tmp_path.unlink() - except OSError: - pass - - @staticmethod - def _subset_key( - path: Path, - bounds: Tuple[float, float, float, float], - xy_buffer: float, - ) -> Tuple[Path, float, float, float, float]: - xmin, xmax, ymin, ymax = bounds - return ( - path, - round(xmin - xy_buffer, 3), - round(xmax + xy_buffer, 3), - round(ymin - xy_buffer, 3), - round(ymax + xy_buffer, 3), - ) - - def get( - self, - path: Path, - bounds: Tuple[float, float, float, float], - xy_buffer: float, - ) -> Optional[Tuple[Optional[cKDTree], Dict[str, np.ndarray]]]: - """Return cached subset (tree, dims) or None. Marks entry as recently used.""" - if self._max <= 0: - return None - key = self._subset_key(path, bounds, xy_buffer) - if key in self._cache: - self._cache.move_to_end(key) - return self._cache[key] - return None - - def load( - self, - path: Path, - dims_to_read: Dict[str, str], - dim_dtypes: Dict[str, np.dtype], - bounds: Tuple[float, float, float, float], - xy_buffer: float, - ) -> Tuple[Optional[cKDTree], Dict[str, np.ndarray]]: - """Stream a spatial subset of an original file and build a KDTree. - - Returns (tree, dim_arrays). If already cached, returns immediately. - """ - import time as _time - - cached = self.get(path, bounds, xy_buffer) - if cached is not None: - return cached - - load_start = _time.monotonic() - key = self._subset_key(path, bounds, xy_buffer) - x_lo, x_hi, y_lo, y_hi = key[1:] - - # Evict if at capacity - while self._max > 0 and len(self._cache) >= self._max: - self._evict_lru() - - # Read point count from header (single-threaded to avoid OpenMP conflicts) - with laspy.open(str(path), laz_backend=laspy.LazBackend.Lazrs) as f: - n_pts = f.header.point_count - - prefix = "Cache loading subset" if self._max > 0 else "Loading subset" - print( - f" {prefix}: {path.name} x=[{x_lo:.1f}, {x_hi:.1f}] " - f"y=[{y_lo:.1f}, {y_hi:.1f}] from {n_pts:,} pts... ", - end="", - flush=True, - ) - - if path.name.endswith(".copc.laz"): - try: - xyz, dim_arrays, read_dt = self._load_copc_subset_with_reader( - path, dims_to_read, dim_dtypes, x_lo, x_hi, y_lo, y_hi, - ) - if xyz is None: - total_dt = _time.monotonic() - load_start - result = (None, {}) - if self._max > 0: - self._cache[key] = result - print(f"empty COPC subset (read {read_dt:.1f}s, total {total_dt:.1f}s)", flush=True) - return result - kept_pts = len(xyz) - print(f"kept {kept_pts:,} pts via COPC reader, tree... ", end="", flush=True) - except Exception as e: - print(f"COPC reader fallback to PDAL ({e})... ", end="", flush=True) - try: - xyz, dim_arrays, read_dt = self._load_copc_subset_with_pdal( - path, dims_to_read, dim_dtypes, x_lo, x_hi, y_lo, y_hi, - ) - if xyz is None: - total_dt = _time.monotonic() - load_start - result = (None, {}) - if self._max > 0: - self._cache[key] = result - print(f"empty COPC subset (read {read_dt:.1f}s, total {total_dt:.1f}s)", flush=True) - return result - kept_pts = len(xyz) - print(f"kept {kept_pts:,} pts via PDAL fallback, tree... ", end="", flush=True) - except Exception as p_e: - print(f"PDAL fallback failed ({p_e}); falling back to streamed scan... ", end="", flush=True) - xyz = None - dim_arrays = {} - read_dt = 0.0 - else: - xyz = None - dim_arrays = {} - read_dt = 0.0 - - if xyz is None: - dim_lists: Dict[str, List[np.ndarray]] = defaultdict(list) - xyz_parts: List[np.ndarray] = [] - - # Single-pass streaming read: keep only points in the requested XY window. - read_start = _time.monotonic() - kept_pts = 0 - with laspy.open(str(path), laz_backend=laspy.LazBackend.Lazrs) as f: - for chunk in f.chunk_iterator(self._chunk_size): - cx = np.asarray(chunk.x) - cy = np.asarray(chunk.y) - mask = (cx >= x_lo) & (cx <= x_hi) & (cy >= y_lo) & (cy <= y_hi) - if not np.any(mask): - continue - kept = int(mask.sum()) - kept_pts += kept - xyz_parts.append( - np.column_stack([ - cx[mask], - cy[mask], - np.asarray(chunk.z)[mask], - ]) - ) - for out_name, orig_name in dims_to_read.items(): - arr = getattr(chunk, orig_name, None) - if arr is not None: - dim_lists[out_name].append(np.asarray(arr)[mask]) - - read_dt = _time.monotonic() - read_start - if kept_pts == 0: - total_dt = _time.monotonic() - load_start - result = (None, {}) - if self._max > 0: - self._cache[key] = result - print(f"empty subset (read {read_dt:.1f}s, total {total_dt:.1f}s)", flush=True) - return result - - xyz = np.concatenate(xyz_parts, axis=0) - dim_arrays = {} - for out_name, arrs in dim_lists.items(): - if arrs: - dtype = dim_dtypes.get(out_name, np.float64) - dim_arrays[out_name] = np.concatenate(arrs).astype(dtype, copy=False) - print(f"kept {kept_pts:,} pts, tree... ", end="", flush=True) - - tree_start = _time.monotonic() - tree = cKDTree(xyz, leafsize=32) - tree_dt = _time.monotonic() - tree_start - del xyz - - total_dt = _time.monotonic() - load_start - result = (tree, dim_arrays) - if self._max > 0: - self._cache[key] = result - print( - f"{'cached' if self._max > 0 else 'done'} (read {read_dt:.1f}s, tree {tree_dt:.1f}s, total {total_dt:.1f}s)", - flush=True, - ) - return result - - def _evict_lru(self) -> None: - """Remove the least-recently-used entry.""" - if not self._cache: - return - key, (tree, dims) = self._cache.popitem(last=False) - path = key[0] - del tree - for v in dims.values(): - del v - dims.clear() - gc.collect() - print(f" Cache evicted: {path.name} subset", flush=True) - - def clear(self) -> None: - """Release all cached entries.""" - while self._cache: - self._evict_lru() - - -def _scan_merged_dim_stats( - merged_laz: Path, - merged_dim_names: Set[str], - skip_core: Set[str], - chunk_size: int = 2_000_000, -) -> Dict[str, Tuple[float, float, int]]: - """Lightweight streaming scan for per-dim min/max/nonzero_count. - - Only used in legacy mode (when ``target_dims is None``) to detect which - merged dimensions are empty/constant and should be overwritten. - """ - dim_stats: Dict[str, Tuple[float, float, int]] = {} - with laspy.open(str(merged_laz), laz_backend=laspy.LazBackend.LazrsParallel) as reader: - for chunk in reader.chunk_iterator(chunk_size): - for dim_name in merged_dim_names - skip_core: - arr = getattr(chunk, dim_name, None) - if arr is None: - continue - a = np.asarray(arr) - c_min = float(np.min(a)) - c_max = float(np.max(a)) - c_nz = int(np.count_nonzero(a)) - if dim_name in dim_stats: - p_min, p_max, p_nz = dim_stats[dim_name] - dim_stats[dim_name] = ( - min(p_min, c_min), - max(p_max, c_max), - p_nz + c_nz, - ) - else: - dim_stats[dim_name] = (c_min, c_max, c_nz) - return dim_stats - - -def _add_original_dimensions_to_merged_impl( - merged_laz: Path, - original_input_dir: Path, - output_path: Path, - original_files: List[Path], - tolerance: float, - retile_buffer: float, - distance_threshold: Optional[float], - target_dims: Optional[Set[str]], - source_extra_dims_to_keep: Optional[Set[str]], - tmp_dir: Path, - num_threads: int = 4, - merge_chunk_size: int = 2_000_000, - spatial_slices: int = 10, - spatial_chunk_length: Optional[float] = None, - spatial_target_points: Optional[int] = None, -) -> None: - """Merged-centric streaming enrichment (v3). - - Streams the merged file chunk-by-chunk. For each chunk, determines which - original files overlap spatially, loads them into an LRU cache (KDTree + - dimension arrays), queries each tree, keeps the closest match per merged - point, and writes the enriched chunk immediately. - - Key properties: - - No merged coordinates held in RAM (streamed one chunk at a time). - - No memmaps or temp files. - - Original files cached via LRU (typically 2-3 in cache at once). - - Optional per-slice parallel subset prep across overlapping original files - when ``num_threads > 1``. - - KDTree queries still use ``workers=-1`` to consume all available CPUs. - - Peak RAM is bounded by the number of overlapping original subsets loaded - for the current slice. - """ - - skip_core = {"X", "Y", "Z"} - CHUNK_SIZE = merge_chunk_size - import time as _time - - phase_start = _time.monotonic() - - # ── Inspect merged input header ────────────────────────────────────── - # NOTE: Use single-threaded Lazrs throughout enrichment to avoid - # deadlocks with OpenMP threads persisting from earlier phases. - print(" Inspecting merged input header...", flush=True) - with laspy.open(str(merged_laz), laz_backend=laspy.LazBackend.Lazrs) as reader: - n_merged = reader.header.point_count - merged_header = reader.header - merged_point_format = merged_header.point_format - merged_dim_names = set(merged_point_format.dimension_names) - merged_extra_dims = list(merged_point_format.extra_dimensions) - if source_extra_dims_to_keep is not None: - merged_extra_dims = [ - dim for dim in merged_extra_dims - if dim.name in source_extra_dims_to_keep - ] - print( - f" Merged input: {n_merged:,} points, {len(merged_dim_names)} dimensions " - f"({_time.monotonic() - phase_start:.1f}s)", - flush=True, - ) - - if n_merged == 0: - print(" Merged file is empty; nothing to enrich.", flush=True) - return - - # ── Legacy dim-stats pre-scan (only when target_dims is None) ──────── - dim_stats: Dict[str, Tuple[float, float, int]] = {} - if target_dims is None: - phase_start = _time.monotonic() - print(" Pre-scanning merged dims for legacy overwrite detection...", flush=True) - dim_stats = _scan_merged_dim_stats(merged_laz, merged_dim_names, skip_core, chunk_size=CHUNK_SIZE) - print( - f" Pre-scan done: {len(dim_stats)} dims analysed " - f"({_time.monotonic() - phase_start:.1f}s)", - flush=True, - ) - - # ── Phase 2: Detect original-file dimensions ──────────────────────── - phase_start = _time.monotonic() - print(f" Scanning original-file dimensions from {len(original_files)} files...", flush=True) - orig_dims: Dict[str, np.dtype] = {} - orig_extra_dim_info: Dict[str, object] = {} - for i, orig_path in enumerate(original_files): - file_start = _time.monotonic() - print(f" [{i+1}/{len(original_files)}] {orig_path.name}... ", end="", flush=True) - with laspy.open(str(orig_path), laz_backend=laspy.LazBackend.Lazrs) as f: - pf = f.header.point_format - pt_dtype = None - try: - one = f.read_points(1) - if one is not None and one.size > 0: - arr_raw = getattr(one, "array", one) - dt = getattr(arr_raw, "dtype", None) - if dt is not None and getattr(dt, "names", None) is not None: - pt_dtype = dt - if one is not None and one.size > 0: - for dim_name in pf.dimension_names: - if dim_name in skip_core or dim_name in orig_dims: - continue - dim_view = getattr(one, dim_name, None) - if dim_view is not None and hasattr(dim_view, "dtype"): - orig_dims[dim_name] = np.dtype(dim_view.dtype) - elif pt_dtype is not None and dim_name in pt_dtype.names: - orig_dims[dim_name] = pt_dtype.fields[dim_name][0] - else: - one = None - except Exception: - one = None - if one is None or pt_dtype is None: - for dim_name in pf.dimension_names: - if dim_name in skip_core or dim_name in orig_dims: - continue - orig_dims[dim_name] = np.float64 - for dim in pf.extra_dimensions: - if dim.name in skip_core: - continue - if dim.name not in orig_dims: - orig_dims[dim.name] = dim.dtype - if dim.name not in orig_extra_dim_info: - orig_extra_dim_info[dim.name] = dim - n_new = len([d for d in pf.dimension_names if d not in skip_core]) - print(f"{n_new} dims found ({_time.monotonic() - file_start:.1f}s)", flush=True) - - print( - f" Original dimension scan complete: {len(orig_dims)} unique dimensions " - f"({_time.monotonic() - phase_start:.1f}s)", - flush=True, - ) - - # ── Phase 3: Filter by target_dims if standardization mode ────────── - requested_target_dims: Set[str] = set() - if target_dims is not None: - requested_target_dims = set(target_dims) - skip_core - skipped_dims = set(orig_dims.keys()) - requested_target_dims - if skipped_dims: - print(f" Standardization filter: skipping {len(skipped_dims)} dims not in target list: " - f"{sorted(skipped_dims)}", flush=True) - orig_dims = {k: v for k, v in orig_dims.items() if k in requested_target_dims} - orig_extra_dim_info = {k: v for k, v in orig_extra_dim_info.items() if k in requested_target_dims} - missing_target_dims = sorted(requested_target_dims - set(orig_dims.keys())) - if missing_target_dims: - print( - f" Standardization warning: {len(missing_target_dims)} requested dims were not found " - f"in the original files and cannot be transferred: {missing_target_dims}", - flush=True, - ) - - # ── Phase 4: Determine which dims to add / overwrite ──────────────── - if target_dims is not None: - # Standardization mode: add all requested original dims that exist, and - # force-overwrite any of those names already present in merged. - dims_to_add_new: Dict[str, np.dtype] = {} - dims_to_overwrite: Dict[str, np.dtype] = {} - for name in sorted(requested_target_dims): - if name not in orig_dims: - continue - dtype = orig_dims[name] - if name in merged_dim_names: - dims_to_overwrite[name] = dtype - else: - dims_to_add_new[name] = dtype - dims_to_add = {**dims_to_add_new, **dims_to_overwrite} - orig_dim_to_read = {k: k for k in dims_to_add} - orig_rename_for_merged: Dict[str, str] = {} - else: - # Legacy mode: use pre-computed dim_stats - def _get_dim_stat_proxy(dim_name): - if dim_name not in dim_stats: - return None - dmin, dmax, nz = dim_stats[dim_name] - if nz == 0: - return np.array([0, 0]) - return np.array([dmin, dmax]) - - dims_to_add_new, dims_to_overwrite = _dims_to_fill_from_source( - orig_dims, merged_dim_names, _get_dim_stat_proxy, skip=skip_core, - ) - used_names = set(merged_dim_names) | set(dims_to_add_new.keys()) | set(dims_to_overwrite.keys()) - collision = (set(orig_dims.keys()) & merged_dim_names) - set(dims_to_overwrite.keys()) - skip_core - orig_rename_for_merged = {} - for name in sorted(collision): - cand = f"{name}_original" - out_name = cand if cand not in used_names else _next_available_suffix(name, used_names) - orig_rename_for_merged[name] = out_name - used_names.add(out_name) - dims_to_add_renamed = {orig_rename_for_merged[n]: orig_dims[n] for n in collision} - dims_to_add = {**dims_to_add_new, **dims_to_overwrite, **dims_to_add_renamed} - orig_dim_to_read = {k: k for k in dims_to_add_new} | {k: k for k in dims_to_overwrite} - orig_dim_to_read.update({out_name: name for name, out_name in orig_rename_for_merged.items()}) - - if not dims_to_add: - print(" No dimensions to add or replace; copying merged file.", flush=True) - import shutil - shutil.copy2(str(merged_laz), str(output_path)) - return - - for dim_name in sorted(dims_to_add_new.keys()): - print(f" Adding dimension: {dim_name}", flush=True) - for dim_name in sorted(dims_to_overwrite.keys()): - print(f" Replacing dimension: {dim_name}", flush=True) - for orig_name, out_name in sorted(orig_rename_for_merged.items()): - print(f" Adding dimension (collision): {orig_name} -> {out_name}", flush=True) - - spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer - max_dist = distance_threshold if distance_threshold is not None else spatial_buffer - query_workers = max(1, int(num_threads)) - - promote_rgb_to_standard = ( - has_standard_rgb_dims(orig_dims.keys()) - and not point_format_has_standard_rgb(merged_header.point_format.id) - ) - - # ── Construct enriched output schema/header ───────────────────────── - phase_start = _time.monotonic() - print(" Constructing enriched output schema (header + extra dims)...", flush=True) - n_collision_dims = len(orig_rename_for_merged) - print( - f" Plan: carry over merged dims, add {len(dims_to_add_new)} new, " - f"replace {len(dims_to_overwrite)}, rename {n_collision_dims} collisions", - flush=True, - ) - preview_names = ( - sorted(dims_to_add_new.keys()) - + sorted(dims_to_overwrite.keys()) - + sorted(orig_rename_for_merged.values()) - ) - if preview_names: - preview = ", ".join(preview_names[:8]) - suffix = " ..." if len(preview_names) > 8 else "" - print(f" Dimension preview: {preview}{suffix}", flush=True) - - carried_extra_params = [] - if promote_rgb_to_standard: - print( - f" Promoting {', '.join(_RGB_STANDARD_DIMS)} to standard LAS RGB fields", - flush=True, - ) - for dim in merged_extra_dims: - if promote_rgb_to_standard and dim.name in _RGB_STANDARD_DIMS: - continue - carried_extra_params.append( - extra_bytes_params_from_dimension_info(dim, name=dim.name) - ) - - extra_params = [] - params_start = _time.monotonic() - for name, dtype in dims_to_add_new.items(): - if promote_rgb_to_standard and name in _RGB_STANDARD_DIMS: - continue - if name in orig_extra_dim_info: - extra_params.append( - extra_bytes_params_from_dimension_info(orig_extra_dim_info[name], name=name) - ) - else: - extra_params.append(laspy.ExtraBytesParams(name=name, type=dtype)) - # Collision dims (legacy mode only) - dims_to_add_renamed = {orig_rename_for_merged[n]: orig_dims.get(n, np.float64) - for n in orig_rename_for_merged} if orig_rename_for_merged else {} - orig_name_by_out = {out_name: name for name, out_name in orig_rename_for_merged.items()} - for out_name, dtype in dims_to_add_renamed.items(): - if promote_rgb_to_standard and out_name in _RGB_STANDARD_DIMS: - continue - orig_name = orig_name_by_out.get(out_name) - if orig_name is not None and orig_name in orig_extra_dim_info: - extra_params.append( - extra_bytes_params_from_dimension_info(orig_extra_dim_info[orig_name], name=out_name) - ) - else: - extra_params.append(laspy.ExtraBytesParams(name=out_name, type=dtype)) - print( - f" Prepared {len(carried_extra_params) + len(extra_params)} extra-dimension descriptors " - f"({_time.monotonic() - params_start:.1f}s)", - flush=True, - ) - - header_copy_start = _time.monotonic() - base_point_format_id = ( - point_format_with_standard_rgb(merged_header.point_format.id) - if promote_rgb_to_standard - else merged_header.point_format.id - ) - out_header = laspy.LasHeader( - point_format=base_point_format_id, - version=merged_header.version, - ) - out_header.offsets = merged_header.offsets - out_header.scales = merged_header.scales - print( - f" Base header: point_format={base_point_format_id}, " - f"version={merged_header.version}, merged extra dims={len(merged_extra_dims)}", - flush=True, - ) - print( - f" Copied merged header metadata " - f"({_time.monotonic() - header_copy_start:.1f}s)", - flush=True, - ) - - # Add carried merged extra dims, plus any newly introduced ones. - all_extra_params = carried_extra_params + extra_params - if all_extra_params: - add_dims_start = _time.monotonic() - out_header.add_extra_dims(all_extra_params) - print( - f" Added {len(all_extra_params)} extra dims to output schema " - f"({_time.monotonic() - add_dims_start:.1f}s)", - flush=True, - ) - - out_dim_names = set(out_header.point_format.dimension_names) - output_path.parent.mkdir(parents=True, exist_ok=True) - print( - f" Output schema ready: {len(out_dim_names)} dimensions total " - f"({_time.monotonic() - phase_start:.1f}s)", - flush=True, - ) - next_step = ( - " Next: read original file bounding boxes, then enrich the merged COPC slice-by-slice." - if merged_laz.name.endswith(".copc.laz") - else " Next: read original file bounding boxes, then stream the merged LAZ and enrich it chunk-by-chunk." - ) - print(next_step, flush=True) - - # ── Pre-compute original file bounding boxes ──────────────────────── - phase_start = _time.monotonic() - print(" Reading original file bounding boxes...", flush=True) - orig_bboxes: Dict[Path, Tuple[float, float, float, float]] = {} - eligible_files: List[Path] = [] - for orig_path in original_files: - try: - with laspy.open(str(orig_path), laz_backend=laspy.LazBackend.Lazrs) as f: - hdr = f.header - orig_bboxes[orig_path] = (hdr.x_min, hdr.x_max, hdr.y_min, hdr.y_max) - eligible_files.append(orig_path) - print(f" {orig_path.name}: x=[{hdr.x_min:.1f}, {hdr.x_max:.1f}] " - f"y=[{hdr.y_min:.1f}, {hdr.y_max:.1f}] ({hdr.point_count:,} pts)", flush=True) - except Exception as e: - print(f" {orig_path.name}: SKIPPED ({e})", flush=True) - continue - - skipped_bounds = len(original_files) - len(eligible_files) - if skipped_bounds > 0: - print(f" Skipped {skipped_bounds} original files (unreadable)", flush=True) - print( - f" Bounding box scan complete: {len(eligible_files)}/{len(original_files)} files usable " - f"({_time.monotonic() - phase_start:.1f}s)", - flush=True, - ) - - print( - f"\n Starting nearest-neighbor enrichment pass: {n_merged:,} merged points, " - f"{len(eligible_files)} original files", - flush=True, - ) - print(f" Spatial buffer: {spatial_buffer:.1f}m, max match distance: {max_dist:.1f}m", flush=True) - print(f" Dimensions to transfer: {len(dims_to_add)} " - f"({len(dims_to_add_new)} new, {len(dims_to_overwrite)} replace)", flush=True) - print(f" Worker usage: up to {query_workers} thread(s) for subset prep; KDTree queries use all available threads", flush=True) - merged_is_copc = merged_laz.name.endswith(".copc.laz") - if merged_is_copc: - merged_bounds = ( - float(merged_header.x_min), - float(merged_header.x_max), - float(merged_header.y_min), - float(merged_header.y_max), - ) - slice_specs = _build_spatial_slices( - merged_bounds, - spatial_slices, - spatial_chunk_length, - spatial_target_points, - n_merged, - ) - slice_specs = _snap_spatial_slices_to_header_grid(slice_specs, merged_header) - print( - f" Spatial slicing: {len(slice_specs)} slices along {slice_specs[0][1].upper()}", - flush=True, - ) - print(" This stage reads spatial COPC windows, matches points to originals, and rewrites the output.", flush=True) - else: - n_chunks = (n_merged + CHUNK_SIZE - 1) // CHUNK_SIZE - print(f" Chunk size: {CHUNK_SIZE:,} points ({n_chunks} chunks expected)", flush=True) - print(" This stage streams the merged LAZ, matches points to originals, and rewrites the output.", flush=True) - - # ── Main streaming loop ───────────────────────────────────────────── - t_start = _time.monotonic() - cache = _OriginalFileCache(max_entries=0, tmp_dir=tmp_dir, chunk_size=CHUNK_SIZE) - total_written = 0 - total_matched = 0 - unit_num = 0 - - with laspy.open( - str(output_path), mode="w", header=out_header, - do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel, - ) as writer: - def _process_block( - label: str, - cx: np.ndarray, - cy: np.ndarray, - cz: np.ndarray, - merged_dim_getter, - subset_count: Optional[int] = None, - ) -> None: - nonlocal total_written, total_matched - t_chunk = _time.monotonic() - overlap_dt = 0.0 - subset_prep_dt = 0.0 - query_dt = 0.0 - write_dt = 0.0 - n = len(cx) - if n == 0: - return - - chunk_xmin, chunk_xmax = float(cx.min()), float(cx.max()) - chunk_ymin, chunk_ymax = float(cy.min()), float(cy.max()) - - overlap_start = _time.monotonic() - overlapping: List[Path] = [] - for orig_path in eligible_files: - ox_min, ox_max, oy_min, oy_max = orig_bboxes[orig_path] - if (ox_max + spatial_buffer < chunk_xmin - or ox_min - spatial_buffer > chunk_xmax - or oy_max + spatial_buffer < chunk_ymin - or oy_min - spatial_buffer > chunk_ymax): - continue - overlapping.append(orig_path) - overlap_dt = _time.monotonic() - overlap_start - - best_dist = np.full(n, np.inf, dtype=np.float64) - best_dims: Dict[str, np.ndarray] = {} - for name, dtype in dims_to_add.items(): - best_dims[name] = np.zeros(n, dtype=dtype) - - chunk_pts = np.column_stack([cx, cy, cz]) - - def _load_subset(orig_path: Path): - ox_min, ox_max, oy_min, oy_max = orig_bboxes[orig_path] - mask = ( - (cx >= ox_min - spatial_buffer) & (cx <= ox_max + spatial_buffer) & - (cy >= oy_min - spatial_buffer) & (cy <= oy_max + spatial_buffer) - ) - if not mask.any(): - return orig_path, None - - query_idx = np.where(mask)[0] - query_bounds = ( - float(cx[query_idx].min()), - float(cx[query_idx].max()), - float(cy[query_idx].min()), - float(cy[query_idx].max()), - ) - load_start = _time.monotonic() - tree, dim_arrays = cache.load( - orig_path, - orig_dim_to_read, - dims_to_add, - query_bounds, - max_dist, - ) - load_dt = _time.monotonic() - load_start - return orig_path, (query_idx, tree, dim_arrays, load_dt) - - loaded_subsets: Dict[Path, Tuple[np.ndarray, Optional[cKDTree], Dict[str, np.ndarray], float]] = {} - if overlapping: - if query_workers > 1 and len(overlapping) > 1: - with ThreadPoolExecutor(max_workers=min(query_workers, len(overlapping))) as executor: - future_to_path = { - executor.submit(_load_subset, orig_path): orig_path - for orig_path in overlapping - } - for future in as_completed(future_to_path): - orig_path = future_to_path[future] - try: - loaded = future.result() - except Exception as e: - print(f" Warning: could not load {orig_path.name}: {e}", flush=True) - continue - if loaded[1] is not None: - loaded_subsets[loaded[0]] = loaded[1] - else: - for orig_path in overlapping: - try: - loaded = _load_subset(orig_path) - except Exception as e: - print(f" Warning: could not load {orig_path.name}: {e}", flush=True) - continue - if loaded[1] is not None: - loaded_subsets[loaded[0]] = loaded[1] - - for orig_path in overlapping: - loaded = loaded_subsets.get(orig_path) - if loaded is None: - continue - - query_idx, tree, dim_arrays, load_dt = loaded - subset_prep_dt += load_dt - if tree is None: - continue - - query_pts = chunk_pts[query_idx] - - query_start = _time.monotonic() - distances, orig_idx = tree.query(query_pts, k=1, workers=-1) - if distances.ndim == 2: - distances = distances[:, 0] - orig_idx = orig_idx[:, 0] - - accept = (distances <= max_dist) & (distances < best_dist[query_idx]) - if not accept.any(): - del query_idx, query_pts, distances, orig_idx, accept - continue - - acc_chunk_idx = query_idx[accept] - acc_orig_idx = orig_idx[accept] - best_dist[acc_chunk_idx] = distances[accept] - - for name, dim_arr in dim_arrays.items(): - if name in best_dims: - best_dims[name][acc_chunk_idx] = dim_arr[acc_orig_idx] - query_dt += _time.monotonic() - query_start - - del query_idx, query_pts, distances, orig_idx, accept - del acc_chunk_idx, acc_orig_idx - del tree - for v in dim_arrays.values(): - del v - dim_arrays.clear() - - write_start = _time.monotonic() - point_record = laspy.ScaleAwarePointRecord.zeros(n, header=out_header) - for dim_name in merged_dim_names: - try: - arr = merged_dim_getter(dim_name) - if arr is not None: - setattr(point_record, dim_name, arr) - except Exception: - pass - for name, arr in best_dims.items(): - if name in out_dim_names: - try: - target_dtype = getattr(point_record, name).dtype - if arr.dtype != target_dtype: - arr = arr.astype(target_dtype) - except (AttributeError, KeyError, TypeError): - pass - setattr(point_record, name, arr) - - writer.write_points(point_record) - write_dt = _time.monotonic() - write_start - - matched_in_chunk = int((best_dist < np.inf).sum()) - total_matched += matched_in_chunk - total_written += n - chunk_dt = _time.monotonic() - t_chunk - elapsed = _time.monotonic() - t_start - pct = total_written / n_merged * 100 - rate = total_written / elapsed if elapsed > 0 else 0 - eta = (n_merged - total_written) / rate if rate > 0 else 0 - - overlap_names = ", ".join(p.name for p in overlapping) if overlapping else "none" - if subset_count is not None: - print( - f" {label}: {n:,} core pts from {subset_count:,} in-bounds, " - f"{matched_in_chunk:,} matched ({len(overlapping)} originals: {overlap_names}), " - f"{chunk_dt:.1f}s", - flush=True, - ) - else: - print( - f" {label}: {n:,} pts, {matched_in_chunk:,} matched " - f"({len(overlapping)} originals: {overlap_names}), {chunk_dt:.1f}s", - flush=True, - ) - print( - f" Timings: overlap {overlap_dt:.1f}s, subset prep {subset_prep_dt:.1f}s, " - f"query/match {query_dt:.1f}s, write {write_dt:.1f}s", - flush=True, - ) - print( - f" Progress: {total_written:,}/{n_merged:,} ({pct:.1f}%), " - f"elapsed {elapsed:.0f}s, ETA {eta:.0f}s, rate {rate:,.0f} pts/s", - flush=True, - ) - - del best_dist, best_dims, chunk_pts, point_record - - if merged_is_copc: - for slice_idx, (slice_bounds, axis, include_upper) in enumerate(slice_specs, start=1): - unit_num += 1 - subset, subset_count = _load_copc_subset_all_dims( - merged_laz, - slice_bounds, - halo=0.0, - work_dir=tmp_dir, - ) - if subset is None or subset_count == 0: - print(f" Slice {slice_idx}/{len(slice_specs)}: empty merged subset", flush=True) - continue - - sx = np.asarray(subset.x, dtype=np.float64) - sy = np.asarray(subset.y, dtype=np.float64) - sz = np.asarray(subset.z, dtype=np.float64) - if axis == "x": - upper_mask = sx <= slice_bounds[1] if include_upper else sx < slice_bounds[1] - core_mask = ( - (sx >= slice_bounds[0]) & upper_mask - & (sy >= slice_bounds[2]) & (sy <= slice_bounds[3]) - ) - else: - upper_mask = sy <= slice_bounds[3] if include_upper else sy < slice_bounds[3] - core_mask = ( - (sx >= slice_bounds[0]) & (sx <= slice_bounds[1]) - & (sy >= slice_bounds[2]) & upper_mask - ) - if not np.any(core_mask): - print(f" Slice {slice_idx}/{len(slice_specs)}: no core points after crop", flush=True) - del subset, sx, sy, sz - continue - - def _slice_dim_getter(dim_name, _subset=subset, _mask=core_mask): - arr = getattr(_subset, dim_name, None) - return None if arr is None else np.asarray(arr)[_mask] - - _process_block( - f"Slice {slice_idx}/{len(slice_specs)}", - sx[core_mask], - sy[core_mask], - sz[core_mask], - _slice_dim_getter, - subset_count=subset_count, - ) - del subset, sx, sy, sz, core_mask - else: - with laspy.open(str(merged_laz), laz_backend=laspy.LazBackend.Lazrs) as reader: - for chunk in reader.chunk_iterator(CHUNK_SIZE): - unit_num += 1 - cx = np.asarray(chunk.x, dtype=np.float64) - cy = np.asarray(chunk.y, dtype=np.float64) - cz = np.asarray(chunk.z, dtype=np.float64) - - def _chunk_dim_getter(dim_name, _chunk=chunk): - return getattr(_chunk, dim_name, None) - - _process_block( - f"Chunk {unit_num}/{n_chunks}", - cx, - cy, - cz, - _chunk_dim_getter, - ) - del chunk, cx, cy, cz - - cache.clear() - gc.collect() - - total_elapsed = _time.monotonic() - t_start - match_pct = total_matched / total_written * 100 if total_written > 0 else 0 - print(f"\n Enrichment complete:", flush=True) - print(f" {total_written:,} points written, {total_matched:,} matched ({match_pct:.1f}%)", flush=True) - print(f" Total time: {total_elapsed:.1f}s ({total_elapsed/60:.1f} min)", flush=True) - - # Per-dimension summary - print(f" Per-dimension summary:", flush=True) - with laspy.open(str(output_path), laz_backend=laspy.LazBackend.Lazrs) as check: - sample = check.read_points(min(100_000, n_merged)) - for dim_name in sorted(dims_to_add.keys()): - arr = getattr(sample, dim_name, None) - if arr is not None: - a = np.asarray(arr) - nz = int(np.count_nonzero(a)) - print(f" {dim_name}: min={float(np.min(a)):.4f}, max={float(np.max(a)):.4f}, " - f"non-zero={nz}/{len(a)} ({nz/len(a)*100:.1f}%)", flush=True) - - print(f" Saved enriched merged: {output_path}", flush=True) - - - - -def _bounds_overlap_2d( - bounds_a: Tuple[float, float, float, float], - bounds_b: Tuple[float, float, float, float], - buffer: float = 0.0, -) -> bool: - """Return True when two XY bboxes overlap, optionally with extra buffer.""" - return not ( - bounds_a[1] < bounds_b[0] - buffer - or bounds_a[0] > bounds_b[1] + buffer - or bounds_a[3] < bounds_b[2] - buffer - or bounds_a[2] > bounds_b[3] + buffer - ) - - -def _prepare_collection_remap_metadata( - collections: List[Path], - target_dims: Optional[Set[str]] = None, -) -> List[dict]: - """ - Scan collection schemas once and cache file bounds for remap. - - When ``target_dims`` is omitted, only extra dimensions are exported from the - source collections. When it is provided, it is treated as a strict allowlist - and may include both extra and standard LAS dimensions. - """ - seen_dim_names: Dict[str, int] = {} - collection_meta: List[dict] = [] - - for coll_path in collections: - coll_files = [coll_path] if coll_path.is_file() else list_pointcloud_files(coll_path) - dim_entries = [] - scanned_names: set = set() - file_entries = [] - - for cf in coll_files: - try: - with laspy.open(str(cf), laz_backend=laspy.LazBackend.LazrsParallel) as f: - hdr = f.header - extra_names = {dim.name for dim in hdr.point_format.extra_dimensions} - file_entries.append({ - "path": cf, - "bounds": (hdr.x_min, hdr.x_max, hdr.y_min, hdr.y_max), - "point_count": int(hdr.point_count), - "is_copc": cf.name.endswith(".copc.laz"), - }) - for dim in hdr.point_format.dimensions: - if dim.name in scanned_names: - continue - is_standard_dim = dim.name not in extra_names - if target_dims is None and is_standard_dim: - continue - if target_dims is not None and dim.name not in target_dims: - continue - scanned_names.add(dim.name) - out_name = dim.name - count = seen_dim_names.get(dim.name, 0) - if count > 0: - out_name = f"{dim.name}_{count + 1}" - seen_dim_names[dim.name] = count + 1 - dim_entries.append(( - dim.name, - out_name, - np.dtype(dim.dtype), - None if is_standard_dim else extra_bytes_params_from_dimension_info(dim), - )) - except Exception: - continue - - collection_meta.append({ - "path": coll_path, - "dim_entries": dim_entries, - "files": file_entries, - "all_copc": bool(file_entries) and all(entry["is_copc"] for entry in file_entries), - }) - - return collection_meta - - -def _load_collection_subset_for_bounds( - coll_meta: dict, - query_bounds: Tuple[float, float, float, float], - spatial_buffer: float, - chunk_size: int, - work_dir: Optional[Path] = None, -) -> Tuple[Optional[cKDTree], Dict[str, np.ndarray], int, int]: - """ - Load only the subset of one collection needed for the current spatial window. - - Returns ``(tree, dim_arrays, indexed_points, candidate_file_count)``. - """ - orig_names = [entry[0] for entry in coll_meta["dim_entries"]] - if not orig_names: - return None, {}, 0, 0 - - candidate_files = [ - entry for entry in coll_meta["files"] - if _bounds_overlap_2d(entry["bounds"], query_bounds, buffer=spatial_buffer) - ] - if not candidate_files: - return None, {}, 0, 0 - - pts_x: List[np.ndarray] = [] - pts_y: List[np.ndarray] = [] - pts_z: List[np.ndarray] = [] - dim_bufs: Dict[str, List[np.ndarray]] = {name: [] for name in orig_names} - - for file_entry in candidate_files: - cf = file_entry["path"] - if file_entry["is_copc"]: - subset, subset_count = _load_copc_subset_all_dims( - cf, - query_bounds, - halo=spatial_buffer, - work_dir=work_dir, - ) - if subset is None or subset_count == 0: - continue - sx = np.asarray(subset.x) - if sx.size == 0: - del subset - continue - pts_x.append(sx) - pts_y.append(np.asarray(subset.y)) - pts_z.append(np.asarray(subset.z)) - for orig_name in orig_names: - arr = getattr(subset, orig_name, None) - if arr is not None: - dim_bufs[orig_name].append(np.asarray(arr)) - else: - dim_bufs[orig_name].append(np.zeros(len(sx), dtype=np.float32)) - del subset - continue - - with laspy.open(str(cf), laz_backend=laspy.LazBackend.LazrsParallel) as f: - for chunk in f.chunk_iterator(chunk_size): - cx = np.asarray(chunk.x) - cy = np.asarray(chunk.y) - mask = ( - (cx >= query_bounds[0] - spatial_buffer) - & (cx <= query_bounds[1] + spatial_buffer) - & (cy >= query_bounds[2] - spatial_buffer) - & (cy <= query_bounds[3] + spatial_buffer) - ) - if not mask.any(): - continue - pts_x.append(cx[mask]) - pts_y.append(cy[mask]) - pts_z.append(np.asarray(chunk.z)[mask]) - for orig_name in orig_names: - arr = getattr(chunk, orig_name, None) - if arr is None: - dim_bufs[orig_name].append(np.zeros(int(mask.sum()), dtype=np.float32)) - else: - dim_bufs[orig_name].append(np.asarray(arr)[mask]) - - if not pts_x: - return None, {}, 0, len(candidate_files) - - xyz = np.column_stack([ - np.concatenate(pts_x), - np.concatenate(pts_y), - np.concatenate(pts_z), - ]) - dim_arrs = {name: np.concatenate(dim_bufs[name]) for name in orig_names} - tree = cKDTree(xyz) - n_points = len(xyz) - del xyz - return tree, dim_arrs, n_points, len(candidate_files) - - -def remap_collections_to_original_files( - collections: List[Path], - original_input_dir: Path, - output_dir: Path, - tolerance: float = 0.1, - retile_buffer: float = 2.0, - chunk_size: int = 1_000_000, - target_dims: Optional[Set[str]] = None, - spatial_slices: int = 50, - spatial_chunk_length: Optional[float] = None, - spatial_target_points: Optional[int] = None, -) -> None: - """Remap N segmented collections onto original input files in a single pass per file. - - For each original file, spatial subsets from all collections are loaded, one KDTree - is built per collection, and all extra dims from all collections are written to the - output file in one streaming pass. No 3DT branding — dimension names are copied as-is. - If the same dimension name exists in more than one collection, later occurrences are - suffixed with ``_2``, ``_3``, etc. to avoid collisions. - - Args: - collections: List of paths, each a folder of LAZ/LAS files (or a single - merged LAZ file) representing one segmentation collection. - original_input_dir: Directory with original pre-tiling LAZ files. - output_dir: Directory to write enriched original files. - tolerance: KDTree match distance threshold in metres. - retile_buffer: Extra spatial buffer around each original file's bounds. - chunk_size: Points per streaming chunk. - """ - import time as _time - - print(f"\n{'=' * 60}", flush=True) - print(f"Remapping {len(collections)} collection(s) to original input files", flush=True) - print(f"{'=' * 60}", flush=True) - - original_files = list_pointcloud_files(original_input_dir) - if not original_files: - print(f" No original files found in {original_input_dir}", flush=True) - return - - output_dir.mkdir(parents=True, exist_ok=True) - spatial_buffer = tolerance * 2 + retile_buffer - collection_meta = _prepare_collection_remap_metadata(collections, target_dims=target_dims) - - for ci, coll_meta in enumerate(collection_meta): - dim_entries = coll_meta["dim_entries"] - if dim_entries: - mapped = ", ".join( - out_name if orig_name == out_name else f"{orig_name}->{out_name}" - for orig_name, out_name, _, _ in dim_entries - ) - else: - mapped = "(no selected dimensions discovered)" - source_mode = "COPC subsets" if coll_meta["all_copc"] else "stream scan fallback" - print( - f" Collection {ci + 1} dims from {coll_meta['path']}: {mapped} [{source_mode}]", - flush=True, - ) - - skipped = 0 - with tempfile.TemporaryDirectory(prefix="3dtrees_multicol_remap_") as tmp_dir_str: - tmp_dir = Path(tmp_dir_str) - - for fi, orig_file in enumerate(original_files): - output_name = orig_file.name.replace(".copc.laz", ".laz") - output_path = output_dir / output_name - if output_path.exists(): - skipped += 1 - continue - - t0 = _time.monotonic() - print(f" [{fi + 1}/{len(original_files)}] {orig_file.name}...", flush=True) - - try: - with laspy.open(str(orig_file), laz_backend=laspy.LazBackend.LazrsParallel) as f: - orig_header = f.header - orig_pf = orig_header.point_format - orig_bounds = ( - orig_header.x_min, orig_header.x_max, - orig_header.y_min, orig_header.y_max, - ) - n_orig = orig_header.point_count - - all_output_dim_names = set(orig_pf.dimension_names) - new_extra_params = [] - for dim in orig_pf.extra_dimensions: - new_extra_params.append(extra_bytes_params_from_dimension_info(dim)) - all_output_dim_names.add(dim.name) - - all_new_out_names = [] - skipped_conflicting_dims = [] - for coll_meta in collection_meta: - for _, out_name, dtype, ebp in coll_meta["dim_entries"]: - if out_name in all_output_dim_names: - if ebp is None and out_name in orig_pf.dimension_names: - continue - skipped_conflicting_dims.append(out_name) - continue - new_extra_params.append(laspy.ExtraBytesParams( - name=out_name, - type=dtype, - description=(ebp.description or "") if ebp is not None else "", - )) - all_new_out_names.append(out_name) - all_output_dim_names.add(out_name) - - promote_rgb = ( - has_standard_rgb_dims(all_output_dim_names) - and not point_format_has_standard_rgb(orig_pf.id) - ) - out_pf_id = point_format_with_standard_rgb(orig_pf.id) if promote_rgb else orig_pf.id - - out_header = laspy.LasHeader( - point_format=out_pf_id, - version=orig_header.version, - ) - out_header.offsets = orig_header.offsets - out_header.scales = orig_header.scales - - filtered_extra_params = [] - for ep in new_extra_params: - if promote_rgb and ep.name in _RGB_STANDARD_DIMS: - continue - filtered_extra_params.append(ep) - out_header.add_extra_dims(filtered_extra_params) - - if promote_rgb: - print( - f" Promoting RGB to standard LAS fields: " - f"point_format {orig_pf.id} -> {out_pf_id}", - flush=True, - ) - - added_dims_str = ", ".join(all_new_out_names) if all_new_out_names else "(none)" - skipped_dims_str = ( - ", ".join(sorted(set(skipped_conflicting_dims))) - if skipped_conflicting_dims else "(none)" - ) - - n_out = 0 - output_path.parent.mkdir(parents=True, exist_ok=True) - with laspy.open( - str(output_path), mode="w", header=out_header, - do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel, - ) as writer: - if orig_file.name.endswith(".copc.laz"): - slice_specs = _build_spatial_slices( - orig_bounds, - spatial_slices, - spatial_chunk_length, - spatial_target_points, - n_orig, - ) - slice_specs = _snap_spatial_slices_to_header_grid(slice_specs, orig_header) - print( - f" Spatial slicing: {len(slice_specs)} slices along {slice_specs[0][1].upper()}", - flush=True, - ) - - for slice_idx, (slice_bounds, axis, include_upper) in enumerate(slice_specs, start=1): - subset, subset_count = _load_copc_subset_all_dims( - orig_file, - slice_bounds, - halo=1.0, - work_dir=tmp_dir, - ) - if subset is None or subset_count == 0: - print(f" Slice {slice_idx}/{len(slice_specs)}: empty original subset", flush=True) - continue - - sx = np.asarray(subset.x) - sy = np.asarray(subset.y) - sz = np.asarray(subset.z) - if axis == "x": - upper_mask = sx <= slice_bounds[1] if include_upper else sx < slice_bounds[1] - core_mask = ( - (sx >= slice_bounds[0]) & upper_mask - & (sy >= slice_bounds[2]) & (sy <= slice_bounds[3]) - ) - else: - upper_mask = sy <= slice_bounds[3] if include_upper else sy < slice_bounds[3] - core_mask = ( - (sx >= slice_bounds[0]) & (sx <= slice_bounds[1]) - & (sy >= slice_bounds[2]) & upper_mask - ) - if not np.any(core_mask): - print(f" Slice {slice_idx}/{len(slice_specs)}: no core points after halo crop", flush=True) - del subset - continue - - orig_xyz = np.column_stack([sx[core_mask], sy[core_mask], sz[core_mask]]) - out_chunk = laspy.ScaleAwarePointRecord.zeros(len(orig_xyz), header=out_header) - for dim_name in orig_pf.dimension_names: - arr = getattr(subset, dim_name, None) - if arr is not None: - setattr(out_chunk, dim_name, np.asarray(arr)[core_mask]) - - coll_summaries = [] - for ci, coll_meta in enumerate(collection_meta, start=1): - tree, dim_arrs, n_indexed, candidate_count = _load_collection_subset_for_bounds( - coll_meta=coll_meta, - query_bounds=slice_bounds, - spatial_buffer=spatial_buffer, - chunk_size=chunk_size, - work_dir=tmp_dir, - ) - coll_summaries.append(f"c{ci}:{n_indexed:,}/{candidate_count}") - if tree is None: - continue - _, idxs = tree.query(orig_xyz, workers=-1) - for orig_name, out_name, dtype, _ in coll_meta["dim_entries"]: - if orig_name in dim_arrs: - setattr(out_chunk, out_name, dim_arrs[orig_name][idxs].astype(dtype)) - del tree, dim_arrs - - writer.write_points(out_chunk) - n_out += len(orig_xyz) - pct = n_out / n_orig * 100 if n_orig else 100 - elapsed = _time.monotonic() - t0 - rate = n_out / elapsed if elapsed > 0 else 0 - print( - f" Slice {slice_idx}/{len(slice_specs)}: {len(orig_xyz):,} core pts from {subset_count:,} in-bounds " - f"({pct:.0f}%, {rate:,.0f} pts/s) [{' '.join(coll_summaries)}]", - flush=True, - ) - del out_chunk, orig_xyz, subset, sx, sy, sz - gc.collect() - else: - n_chunks_expected = (n_orig + chunk_size - 1) // chunk_size - print( - f" Non-COPC original fallback: streaming {n_orig:,} pts in ~{n_chunks_expected} chunks", - flush=True, - ) - with laspy.open(str(orig_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: - chunk_i = 0 - for chunk in reader.chunk_iterator(chunk_size): - xyz_q = np.column_stack([ - np.asarray(chunk.x), - np.asarray(chunk.y), - np.asarray(chunk.z), - ]) - chunk_bounds = ( - float(np.min(xyz_q[:, 0])), - float(np.max(xyz_q[:, 0])), - float(np.min(xyz_q[:, 1])), - float(np.max(xyz_q[:, 1])), - ) - out_chunk = laspy.ScaleAwarePointRecord.zeros(len(chunk), header=out_header) - for dim_name in orig_pf.dimension_names: - try: - setattr(out_chunk, dim_name, np.asarray(getattr(chunk, dim_name))) - except Exception: - pass - - coll_summaries = [] - for ci, coll_meta in enumerate(collection_meta, start=1): - tree, dim_arrs, n_indexed, candidate_count = _load_collection_subset_for_bounds( - coll_meta=coll_meta, - query_bounds=chunk_bounds, - spatial_buffer=spatial_buffer, - chunk_size=chunk_size, - work_dir=tmp_dir, - ) - coll_summaries.append(f"c{ci}:{n_indexed:,}/{candidate_count}") - if tree is None: - continue - _, idxs = tree.query(xyz_q, workers=-1) - for orig_name, out_name, dtype, _ in coll_meta["dim_entries"]: - if orig_name in dim_arrs: - setattr(out_chunk, out_name, dim_arrs[orig_name][idxs].astype(dtype)) - del tree, dim_arrs - - writer.write_points(out_chunk) - n_out += len(chunk) - chunk_i += 1 - pct = n_out / n_orig * 100 if n_orig else 100 - elapsed = _time.monotonic() - t0 - rate = n_out / elapsed if elapsed > 0 else 0 - print( - f" Chunk {chunk_i}/{n_chunks_expected}: {len(chunk):,} pts " - f"({pct:.0f}%, {rate:,.0f} pts/s) [{' '.join(coll_summaries)}]", - flush=True, - ) - del out_chunk, xyz_q - - print( - f" {n_orig:,} pts → {n_out:,} pts written " - f"({_time.monotonic() - t0:.1f}s)", - flush=True, - ) - print(f" Added dims: {added_dims_str}", flush=True) - if skipped_conflicting_dims: - print( - f" Skipped conflicting dims already present in original schema: {skipped_dims_str}", - flush=True, - ) - - except Exception as e: - print(f" Error processing {orig_file.name}: {e}", flush=True) - import traceback - traceback.print_exc() - - if skipped: - print(f" Skipped {skipped} already-processed files.", flush=True) - print(f"\n Done: {len(original_files)} files processed to {output_dir}", flush=True) - - -def add_original_dimensions_to_merged( - merged_laz: Path, - original_input_dir: Path, - output_path: Path, - tolerance: float = 0.1, - retile_buffer: float = 2.0, - distance_threshold: Optional[float] = None, - num_threads: int = 4, - target_dims: Optional[Set[str]] = None, - merge_chunk_size: int = 2_000_000, - spatial_slices: int = 10, - spatial_chunk_length: Optional[float] = None, - spatial_target_points: Optional[int] = None, -) -> None: - """ - Add dimensions from original input files to the merged point cloud by - nearest-neighbor matching, and write an enriched merged LAZ file. - - Args: - merged_laz: Path to the merged LAZ file. - original_input_dir: Directory containing original LAZ/LAS files (pre-tiling). - output_path: Path for the output merged LAZ with original dimensions added. - tolerance: Distance tolerance for spatial buffer calculation. - retile_buffer: Additional spatial buffer in meters. - distance_threshold: Max distance for accepting a match (default: 2 * tolerance + retile_buffer). - num_threads: Number of original files to process in parallel (default 4). Use 1 for sequential. - target_dims: If provided (e.g. from standardization JSON), only these - dimensions are transferred from originals, and they **overwrite** - existing merged values. Dimensions not in this set (e.g. - PredInstance) are kept from merged. When None, the legacy - behaviour applies (add missing, overwrite empty/constant, rename - collisions to _original). - """ - if output_path.resolve() == Path(merged_laz).resolve(): - raise ValueError("output_path must differ from merged_laz to avoid overwriting input") - - original_files = list_pointcloud_files(original_input_dir) - if not original_files: - print(" No original input files found; skipping merged-with-originals output.", flush=True) - return - - print(f"\n{'=' * 60}", flush=True) - print("Adding original-file dimensions to merged point cloud", flush=True) - print(f"{'=' * 60}", flush=True) - - # v3 streaming enrichment: no memmaps, no temp files needed. - # tmp_dir kept for API compatibility but unused internally. - tmp_dir = output_path.parent / f"_enrich_tmp_{os.getpid()}" - tmp_dir.mkdir(parents=True, exist_ok=True) - try: - return _add_original_dimensions_to_merged_impl( - merged_laz, original_input_dir, output_path, original_files, - tolerance, retile_buffer, distance_threshold, target_dims, None, tmp_dir, - num_threads=num_threads, - merge_chunk_size=merge_chunk_size, - spatial_slices=spatial_slices, - spatial_chunk_length=spatial_chunk_length, - spatial_target_points=spatial_target_points, - ) - finally: - import shutil as _shutil - if tmp_dir.exists(): - _shutil.rmtree(tmp_dir, ignore_errors=True) - - -def enrich_collection_tiles_with_original_dimensions( - tile_collection: Path, - original_input_dir: Path, - output_dir: Path, - tolerance: float = 0.1, - retile_buffer: float = 2.0, - distance_threshold: Optional[float] = None, - num_threads: int = 4, - target_dims: Optional[Set[str]] = None, - source_extra_dims_to_keep: Optional[Set[str]] = None, - merge_chunk_size: int = 2_000_000, - spatial_slices: int = 10, - spatial_chunk_length: Optional[float] = None, - spatial_target_points: Optional[int] = None, -) -> None: - """ - Enrich a tile collection with original-file dimensions while keeping the - tile geometry unchanged. - - Each input tile is processed independently with the same spatially chunked - enrichment logic used for merged-file enrichment. This keeps the duplicate- - free tile layout intact so the enriched outputs can be concatenated directly - without global overlap deduplication. - """ - import time as _time - - tile_collection = Path(tile_collection) - original_input_dir = Path(original_input_dir) - output_dir = Path(output_dir) - - tile_files = [tile_collection] if tile_collection.is_file() else list_pointcloud_files(tile_collection) - if not tile_files: - print(f" No tile files found in {tile_collection}; skipping tile enrichment.", flush=True) - return - - original_files = list_pointcloud_files(original_input_dir) - if not original_files: - print(" No original input files found; skipping tile enrichment.", flush=True) - return - - output_dir.mkdir(parents=True, exist_ok=True) - - print(f"\n{'=' * 60}", flush=True) - print("Enriching tiled remap source with original-file dimensions", flush=True) - print(f"{'=' * 60}", flush=True) - print(f" Tile source: {tile_collection}", flush=True) - print(f" Originals: {original_input_dir}", flush=True) - print(f" Output tiles: {output_dir}", flush=True) - print(f" Tile count: {len(tile_files)}", flush=True) - - def _fmt_elapsed(seconds: float) -> str: - if seconds < 60: - return f"{seconds:.1f}s" - return f"{seconds / 60:.1f} min" - - skipped = 0 - total_start = _time.monotonic() - for idx, tile_path in enumerate(tile_files, start=1): - output_name = tile_path.name.replace(".copc.laz", ".laz") - output_path = output_dir / output_name - if output_path.resolve() == tile_path.resolve(): - raise ValueError( - f"Tile enrichment output would overwrite input tile: {tile_path}" - ) - if output_path.exists(): - skipped += 1 - print( - f" [{idx}/{len(tile_files)}] {tile_path.name}: output already exists, skipping", - flush=True, - ) - continue - - tile_start = _time.monotonic() - print( - f"\n [{idx}/{len(tile_files)}] Enriching tile {tile_path.name} " - f"→ {output_name}", - flush=True, - ) - tmp_dir = output_dir / f"_enrich_tmp_{os.getpid()}_{idx:04d}" - tmp_dir.mkdir(parents=True, exist_ok=True) - try: - _add_original_dimensions_to_merged_impl( - merged_laz=tile_path, - original_input_dir=original_input_dir, - output_path=output_path, - original_files=original_files, - tolerance=tolerance, - retile_buffer=retile_buffer, - distance_threshold=distance_threshold, - target_dims=target_dims, - source_extra_dims_to_keep=source_extra_dims_to_keep, - tmp_dir=tmp_dir, - num_threads=num_threads, - merge_chunk_size=merge_chunk_size, - spatial_slices=spatial_slices, - spatial_chunk_length=spatial_chunk_length, - spatial_target_points=spatial_target_points, - ) - finally: - shutil.rmtree(tmp_dir, ignore_errors=True) - - print( - f" [{idx}/{len(tile_files)}] Tile enrichment finished in " - f"{_fmt_elapsed(_time.monotonic() - tile_start)}", - flush=True, - ) - - if skipped: - print(f" Skipped {skipped} already-enriched tile(s).", flush=True) - total_elapsed = _time.monotonic() - total_start - print( - f" Tile enrichment complete in {_fmt_elapsed(total_elapsed)}", - flush=True, - ) +def _merge_input_files(input_dir: Path) -> List[Path]: + """Return merge input files, including mixed LAS/LAZ and preferring COPC twins.""" + return point_cloud_files(input_dir) # ============================================================================= @@ -4364,7 +106,6 @@ def merge_tiles( transfer_original_dims_to_merged: bool = True, threedtrees_dims: Optional[List[str]] = None, threedtrees_suffix: str = "SAT", - standardization_json: Optional[Path] = None, ): """ Main merge function implementing the tile merging pipeline. @@ -4407,18 +148,39 @@ def merge_tiles( print(f"\n{'=' * 60}") print(f"Merged file already exists: {output_merged}") print(f"{'=' * 60}") - merged_for_downstream = output_merged + validate_merged_output_contract(output_merged, instance_dimension) + print(" Loading merged file and proceeding to retiling stage...") + + merged_points, all_merged_dims, merged_extra_dim_params = load_merged_file(output_merged) + # For retile we need (instances, extra_dims) split; for remap we pass all_merged_dims as-is + if instance_dimension in all_merged_dims: + merged_instances = all_merged_dims[instance_dimension] + merged_extra_dims = {k: v for k, v in all_merged_dims.items() if k != instance_dimension} + elif "treeID" in all_merged_dims: + merged_instances = all_merged_dims["treeID"] + merged_extra_dims = {k: v for k, v in all_merged_dims.items() if k != "treeID"} + else: + merged_instances = None + merged_extra_dims = {} + for k, v in all_merged_dims.items(): + if merged_instances is None and np.issubdtype(v.dtype, np.integer): + merged_instances = v + else: + merged_extra_dims[k] = v + if merged_instances is None: + merged_instances = np.zeros(len(merged_points), dtype=np.int32) - retile_to_original_files_streaming( - merged_file=merged_for_downstream, - original_tiles_dir=original_tiles_dir, - output_dir=output_tiles_dir, + retile_to_original_files( + merged_points, + merged_instances, + merged_extra_dims, + merged_extra_dim_params, + original_tiles_dir, + output_tiles_dir, tolerance=0.1, + num_threads=num_threads, retile_buffer=retile_buffer, - chunk_size=chunk_size, instance_dimension=instance_dimension, - threedtrees_dims=threedtrees_dims, - threedtrees_suffix=threedtrees_suffix, ) print(f" ✓ Stage 6 completed: Retiled to original files") @@ -4426,18 +188,20 @@ def merge_tiles( print(f"\n{'=' * 60}") print("Stage 7: Remapping to Original Input Files") print(f"{'=' * 60}") - + original_output_dir = output_tiles_dir.parent / "original_with_predictions" - remap_to_original_input_files_streaming( - merged_file=merged_for_downstream, - original_input_dir=original_input_dir, - output_dir=original_output_dir, + remap_to_original_input_files( + merged_points, + all_merged_dims, + merged_extra_dim_params, + original_input_dir, + original_output_dir, tolerance=retile_max_radius, + num_threads=num_threads, retile_buffer=retile_buffer, - chunk_size=chunk_size, threedtrees_dims=threedtrees_dims, threedtrees_suffix=threedtrees_suffix, - target_dims=_target_dims, + prefer_copc_sources=False, ) print(f" ✓ Stage 7 completed: Remapped to original input files") else: @@ -4448,10 +212,8 @@ def merge_tiles( print(f"{'=' * 60}") return - # Find all input LAZ files - laz_files = sorted(input_dir.glob("*.laz")) - if not laz_files: - laz_files = sorted(input_dir.glob("*.las")) + # Find all input point clouds, preferring COPC twins for the same source. + laz_files = _merge_input_files(input_dir) if len(laz_files) == 0: print(f"No LAZ/LAS files found in {input_dir}") @@ -4463,49 +225,32 @@ def merge_tiles( print(" Extracting tile bounds from headers...") tile_boundaries: Dict[str, Tuple[float, float, float, float]] = {} tile_names: List[str] = [] - + source_file_by_tile_name: Dict[str, Path] = {} + for f in laz_files: - name = normalize_tile_id(f.stem) + name = merge_tile_name(f) tile_names.append(name) - + source_file_by_tile_name[name] = f + bounds = get_tile_bounds_from_header(f) if bounds: tile_boundaries[name] = bounds else: print(f" Warning: Could not extract bounds from {f.name}") - + if len(tile_boundaries) == 0: raise ValueError("Could not extract bounds from any tile files") - + print(f" Extracted bounds from {len(tile_boundaries)} tiles") # Build neighbor graph from tile_bounds_tindex.json and match to loaded tiles print(" Loading neighbor graph from tile_bounds_tindex.json...") - with tile_bounds_json.open() as f: - json_data = json.load(f) - json_labels = [ - f"c{int(tile['col']):02d}_r{int(tile['row']):02d}" - if "col" in tile and "row" in tile else None - for tile in json_data.get("tiles", []) - ] - json_bounds, centers, neighbors_idx = build_neighbor_graph_from_bounds_json( - tile_bounds_json, - bounds_field="planned_bounds", - ) + json_bounds, centers, neighbors_idx = build_neighbor_graph_from_bounds_json(tile_bounds_json) print(f" JSON tiles in bounds file: {len(json_bounds)}") - tile_to_json, json_to_tile = _match_tiles_to_json_bounds( - tile_boundaries, - json_bounds, - centers, - json_labels=json_labels, - ) + tile_to_json, json_to_tile = match_tiles_to_json_bounds(tile_boundaries, json_bounds, centers) print(" Matched tiles to JSON bounds successfully") - planned_tile_boundaries: Dict[str, Tuple[float, float, float, float]] = {} - for tile_name, json_idx in tile_to_json.items(): - planned_tile_boundaries[tile_name] = json_bounds[json_idx] - # Build neighbors per tile name using the JSON neighbor graph. # JSON can contain tiles that were never created (no points in bounds); json_to_tile only # has entries for JSON indices that matched an actual LAZ file. So neighbor_name = @@ -4539,7 +284,7 @@ def merge_tiles( # Load tiles in parallel using ProcessPoolExecutor for true CPU parallelism # Prepare arguments for multiprocessing (must be pickleable) - load_args = [(f, planned_tile_boundaries, buffer, neighbors_by_tile, instance_dimension) for f in laz_files] + load_args = [(f, tile_boundaries, buffer, neighbors_by_tile, instance_dimension) for f in laz_files] with ProcessPoolExecutor(max_workers=num_threads) as executor: results = list(executor.map(_load_tile_wrapper, load_args)) @@ -4561,20 +306,20 @@ def merge_tiles( if len(tiles) == 0: print("No tiles loaded successfully") return - + # Log extra dims found across tiles all_extra_dim_names = set() for tile in tiles: all_extra_dim_names.update(tile.extra_dims.keys()) if all_extra_dim_names: print(f" Extra dimensions (passenger data): {', '.join(sorted(all_extra_dim_names))}") - + total_points = sum(len(tile.points) for tile in tiles) total_kept = sum(len(kept) for kept in kept_instances_per_tile.values()) total_filtered = sum(len(filtered) for filtered in filtered_instances_per_tile.values()) print(f" ✓ Stage 1 completed: {len(tiles)} tiles loaded, {total_points:,} total points") print(f" Kept {total_kept} instances, filtered {total_filtered} buffer zone instances") - + # Save filtered tiles (with filtered instances removed) filtered_tiles_dir = output_tiles_dir / "filtered_tiles" filtered_tiles_dir.mkdir(parents=True, exist_ok=True) @@ -4582,822 +327,138 @@ def merge_tiles( for tile in tiles: kept_instances = kept_instances_per_tile[tile.name] keep_mask = np.isin(tile.instances, list(kept_instances) + [0]) - + filtered_points = tile.points[keep_mask] filtered_instances = tile.instances[keep_mask] filtered_extra_dims = {name: arr[keep_mask] for name, arr in tile.extra_dims.items()} - + if len(filtered_points) == 0: print(f" Warning: {tile.name} has no points after filtering, skipping") continue - + filtered_output_path = filtered_tiles_dir / f"{tile.name}.laz" header = laspy.LasHeader(point_format=6, version="1.4") header.offsets = [filtered_points[:, 0].min(), filtered_points[:, 1].min(), filtered_points[:, 2].min()] - header.scales = [0.01, 0.01, 0.01] - + header.scales = MERGED_OUTPUT_SCALES + output_las = laspy.LasData(header) output_las.x = filtered_points[:, 0] output_las.y = filtered_points[:, 1] output_las.z = filtered_points[:, 2] - - extra_dims_params = [laspy.ExtraBytesParams(name=instance_dimension, type=np.int32)] + + extra_dims_params = [instance_extra_bytes_params(instance_dimension, filtered_instances)] for dim_name, dim_arr in filtered_extra_dims.items(): extra_dims_params.append(laspy.ExtraBytesParams(name=dim_name, type=dim_arr.dtype)) output_las.add_extra_dims(extra_dims_params) - - setattr(output_las, instance_dimension, filtered_instances) + + setattr(output_las, instance_dimension, cast_instances_for_output(filtered_instances, instance_dimension)) for dim_name, dim_arr in filtered_extra_dims.items(): setattr(output_las, dim_name, dim_arr) - + output_las.write(str(filtered_output_path)) print(f" ✓ Saved {len(tiles)} filtered tiles as .laz files (filtered instances removed)") # ========================================================================= - # Stage 2: Assign Global Instance IDs - # ========================================================================= - print(f"\n{'=' * 60}") - print("Stage 2: Assigning global instance IDs") - print(f"{'=' * 60}") - - TILE_OFFSET = 100000 # Unique global ID: tile_idx * OFFSET + local_id - - def global_id(tile_idx: int, local_id: int) -> int: - return tile_idx * TILE_OFFSET + local_id - - def local_id(gid: int) -> Tuple[int, int]: - return gid // TILE_OFFSET, gid % TILE_OFFSET - - # Initialize Union-Find and track instance sizes - uf = UnionFind() - instance_sizes = {} # global_id -> point count - - for tile_idx, tile in enumerate(tiles): - print(f" Processing tile {tile_idx + 1}/{len(tiles)}: {tile.name} ({len(tile.points):,} points)...") - kept_instances = kept_instances_per_tile[tile.name] - - unique_inst, inst_counts = np.unique(tile.instances, return_counts=True) - - for i, local_inst in enumerate(unique_inst): - if local_inst <= 0 or local_inst not in kept_instances: - continue - - gid = global_id(tile_idx, local_inst) - size = int(inst_counts[i]) - uf.make_set(gid, size) - instance_sizes[gid] = size - - print(f" Total global instances: {len(instance_sizes)}") - print(f" ✓ Stage 2 completed: Assigned global IDs to {len(instance_sizes)} instances") - - # Helper functions for border matching - def get_opposite_direction(direction: str) -> str: - """Get opposite direction.""" - opposites = {"east": "west", "west": "east", "north": "south", "south": "north"} - return opposites.get(direction, direction) - - def log_instance_pair_analysis( - inst_id_a: int, - inst_id_b: int, - tile_a_name: str, - tile_b_name: str, - direction: str, - bbox_a: Tuple[float, float, float, float], - bbox_b: Tuple[float, float, float, float], - overlap_ratio: float, - overlap_threshold: float, - bbox_overlaps: bool, - centroid_a: np.ndarray, - centroid_b: np.ndarray, - size_a: int, - size_b: int, - matched: bool, - ): - """Log detailed analysis of an instance pair for debugging.""" - print(f"\n{'='*60}") - print(f"DEBUG: Instance Pair Analysis") - print(f"{'='*60}") - print(f"Instance {inst_id_a} ({tile_a_name}) <-> Instance {inst_id_b} ({tile_b_name})") - print(f"Direction: {tile_a_name} ({direction}) <-> {tile_b_name} ({get_opposite_direction(direction)})") - print(f"\nInstance {inst_id_a}:") - print(f" Tile: {tile_a_name}") - print(f" Point count: {size_a:,}") - print(f" Centroid: ({centroid_a[0]:.2f}, {centroid_a[1]:.2f}, {centroid_a[2]:.2f})") - print(f" BBox: ({bbox_a[0]:.2f}, {bbox_a[1]:.2f}) x ({bbox_a[2]:.2f}, {bbox_a[3]:.2f})") - print(f"\nInstance {inst_id_b}:") - print(f" Tile: {tile_b_name}") - print(f" Point count: {size_b:,}") - print(f" Centroid: ({centroid_b[0]:.2f}, {centroid_b[1]:.2f}, {centroid_b[2]:.2f})") - print(f" BBox: ({bbox_b[0]:.2f}, {bbox_b[1]:.2f}) x ({bbox_b[2]:.2f}, {bbox_b[3]:.2f})") - centroid_dist = np.linalg.norm(centroid_a - centroid_b) - print(f"\nCentroid distance: {centroid_dist:.2f}m") - print(f"BBox overlaps (10cm tolerance): {'YES' if bbox_overlaps else 'NO'}") - print(f"FF3D overlap ratio: {overlap_ratio:.4f}") - print(f"Overlap threshold: {overlap_threshold:.4f}") - print(f"Match result: {'MATCHED' if matched else 'NOT MATCHED'}") - if not matched: - reasons = [] - if not bbox_overlaps: - reasons.append("BBox doesn't overlap (within 10cm)") - if overlap_ratio < overlap_threshold: - reasons.append(f"Overlap ratio {overlap_ratio:.4f} < threshold {overlap_threshold:.4f}") - if reasons: - print(f" Reasons: {', '.join(reasons)}") - print(f"{'='*60}\n") - - def bboxes_overlap(bbox_a: Tuple[float, float, float, float], bbox_b: Tuple[float, float, float, float], tolerance: float = 0.1) -> bool: - """ - Check if two bounding boxes overlap or are within tolerance distance. - - Args: - bbox_a: (minx, maxx, miny, maxy) of first bounding box - bbox_b: (minx, maxx, miny, maxy) of second bounding box - tolerance: Maximum distance between boxes to still consider them (default: 0.1m = 10cm) - - Returns: - True if boxes overlap or are within tolerance distance - """ - minx_a, maxx_a, miny_a, maxy_a = bbox_a - minx_b, maxx_b, miny_b, maxy_b = bbox_b - - # Check if boxes overlap (original check) - if not (maxx_a < minx_b or minx_a > maxx_b or maxy_a < miny_b or miny_a > maxy_b): - return True - - # Check if boxes are within tolerance distance (almost touching) - # Compute gaps in X and Y dimensions - # If boxes don't overlap, find the minimum separation - x_gap = 0.0 - if maxx_a < minx_b: - x_gap = minx_b - maxx_a # A is to the left of B - elif maxx_b < minx_a: - x_gap = minx_a - maxx_b # B is to the left of A - # else: they overlap in X, x_gap = 0 - - y_gap = 0.0 - if maxy_a < miny_b: - y_gap = miny_b - maxy_a # A is below B - elif maxy_b < miny_a: - y_gap = miny_a - maxy_b # B is below A - # else: they overlap in Y, y_gap = 0 - - # Minimum separation is the diagonal distance between closest corners - # For non-overlapping boxes: √(x_gap² + y_gap²) - # But if boxes overlap in one dimension, we use the gap in the other dimension - separation = np.sqrt(x_gap * x_gap + y_gap * y_gap) - - return separation <= tolerance + match_result = assign_and_match_instances( + tiles=tiles, + tile_boundaries=tile_boundaries, + neighbors_by_tile=neighbors_by_tile, + kept_instances_per_tile=kept_instances_per_tile, + filtered_instances_per_tile=filtered_instances_per_tile, + buffer_direction_per_tile=buffer_direction_per_tile, + buffer=buffer, + border_zone_width=border_zone_width, + overlap_threshold=overlap_threshold, + correspondence_tolerance=correspondence_tolerance, + num_threads=num_threads, + debug_instance_ids=debug_instance_ids, + match_all_instances=match_all_instances, + verbose=verbose, + ) + global_to_merged = match_result.global_to_merged + merged_instance_sources = match_result.merged_instance_sources + tile_idx_to_name = match_result.tile_idx_to_name + if skip_merged_file: + print(f"\n{'=' * 60}") + print("Writing filtered output tiles without global merged cloud") + print(f"{'=' * 60}") + print(" --skip_merged_file is enabled; writing per-tile outputs directly") + output_tiles_dir.mkdir(parents=True, exist_ok=True) - # ========================================================================= - # Stage 3: Border Region Instance Matching (or All Instance Matching) - # ========================================================================= - # Note: Cross-tile matching is optimized - each tile pair is checked exactly once - # using `for j in range(i + 1, len(tiles))`, avoiding duplicate A->B and B->A checks. - stage_name = "All Instance Matching" if match_all_instances else "Border Region Instance Matching" - print(f"\n{'=' * 60}") - print(f"Stage 3: {stage_name}") - print(f"{'=' * 60}") - - # Instance tracking for debugging - instance_tracking = {} # (tile_name, local_inst_id) -> tracking info - if debug_instance_ids: - print(f" Debug mode enabled for instances: {sorted(debug_instance_ids)}") - # Initialize tracking for all instances in all tiles + written_tiles = 0 + total_output_points = 0 for tile_idx, tile in enumerate(tiles): - unique_instances = np.unique(tile.instances[tile.instances > 0]) - for local_inst in unique_instances: - gid = global_id(tile_idx, local_inst) - if local_inst in debug_instance_ids: - instance_tracking[(tile.name, local_inst)] = { - "tile_name": tile.name, - "local_id": local_inst, - "global_id": gid, - "filtered_in_stage1": local_inst not in kept_instances_per_tile[tile.name], - "in_border_region": False, - "border_direction": None, - "compared_with": [], - "matched_with": None - } - - if match_all_instances: - print(f" Finding all instances (matching all instances, not just border region)...") - else: - print(f" Finding border region instances (centroids in buffer to buffer+{border_zone_width}m zone)...") - - # Find instances to match (border region or all instances) - border_instances = {} # tile_name -> {instance_id: {'centroid': [...], 'points': [...], 'boundary': [...]}} - - # Build tile name to index mapping - tile_name_to_idx = {tile.name: idx for idx, tile in enumerate(tiles)} - - for tile_idx, tile in enumerate(tiles): - print(f" Processing tile {tile_idx + 1}/{len(tiles)}: {tile.name} ({len(tile.points):,} points)...") - tile_name = tile.name - # Use neighbors from JSON graph when available; fall back to spatial neighbors only - # if this tile was somehow not present in the JSON mapping (should not happen). - if tile_name in neighbors_by_tile: - neighbors = neighbors_by_tile[tile_name] - else: - neighbors = find_spatial_neighbors(tile.boundary, tile_name, tile_boundaries, tolerance=buffer) - kept_instances = kept_instances_per_tile[tile_name] - - neighbor_names = [n for n in neighbors.values() if n is not None] - print(f" Neighbors: {', '.join(neighbor_names) if neighbor_names else 'none'}") - if verbose: - for direction, neighbor_name in neighbors.items(): - if neighbor_name is not None: - neighbor_boundary = tile_boundaries.get(neighbor_name) - if neighbor_boundary: - overlap = find_overlap_region(tile.boundary, neighbor_boundary) - if overlap: - ov_minx, ov_maxx, ov_miny, ov_maxy = overlap - ov_width = ov_maxx - ov_minx - ov_height = ov_maxy - ov_miny - print(f" {direction.upper()} {neighbor_name}: overlap {ov_width:.1f}m x {ov_height:.1f}m") - - min_x, max_x, min_y, max_y = tile.boundary - border_zone_end = buffer + border_zone_width # border_zone_width beyond buffer - - # Define border region boundaries (buffer to buffer+border_zone_width from edges with neighbors) - # Inner edge of border region (end of buffer zone) - border_inner_min_x = min_x + (buffer if neighbors["west"] is not None else 0) - border_inner_max_x = max_x - (buffer if neighbors["east"] is not None else 0) - border_inner_min_y = min_y + (buffer if neighbors["south"] is not None else 0) - border_inner_max_y = max_y - (buffer if neighbors["north"] is not None else 0) - - # Outer edge of border region (buffer+border_zone_width from tile edge) - border_outer_min_x = min_x + (border_zone_end if neighbors["west"] is not None else 0) - border_outer_max_x = max_x - (border_zone_end if neighbors["east"] is not None else 0) - border_outer_min_y = min_y + (border_zone_end if neighbors["south"] is not None else 0) - border_outer_max_y = max_y - (border_zone_end if neighbors["north"] is not None else 0) - - border_instances[tile_name] = {} - - if match_all_instances: - # Collect ALL kept instances (not just border region) - all_unique_insts = kept_instances - {0} # All kept instances except ground - - if len(all_unique_insts) == 0: - print(f" No instances to match in {tile.name}") - continue - - # Compute centroids for all instances - print(f" Computing centroids for {len(all_unique_insts)} instances (all instances)...") - all_centroids = compute_centroids_vectorized(tile.points, tile.instances) - instance_centroids = { - inst_id: all_centroids[inst_id] - for inst_id in all_unique_insts - if inst_id in all_centroids - } - - instance_count = 0 - - # For each instance, extract full points (no direction filtering) - for inst_id in all_unique_insts: - if inst_id not in instance_centroids: + kept_instances = kept_instances_per_tile[tile.name] + max_local_inst = int(tile.instances.max()) + 1 + inst_to_merged = np.full(max_local_inst, -1, dtype=np.int32) + if max_local_inst > 0: + inst_to_merged[0] = 0 + + for local_inst in kept_instances: + if local_inst <= 0: continue - - centroid = instance_centroids[inst_id] - - # Extract full instance points - inst_mask = tile.instances == inst_id - inst_points = tile.points[inst_mask] - - # Compute instance bounding box - inst_minx = inst_points[:, 0].min() - inst_maxx = inst_points[:, 0].max() - inst_miny = inst_points[:, 1].min() - inst_maxy = inst_points[:, 1].max() - - # Use "all" as direction to indicate this is not border-specific - border_instances[tile_name][inst_id] = { - 'centroid': centroid, - 'points': inst_points, - 'boundary': (inst_minx, inst_maxx, inst_miny, inst_maxy), - 'direction': 'all', # Special direction for all-instance matching - 'tile_idx': tile_idx - } - instance_count += 1 - - # Update tracking for debug instances - if debug_instance_ids and inst_id in debug_instance_ids: - key = (tile_name, inst_id) - if key in instance_tracking: - instance_tracking[key]["in_border_region"] = True - instance_tracking[key]["border_direction"] = 'all' - print(f" DEBUG: Instance {inst_id} included in all-instance matching") - - print(f" Found {instance_count} instances in {tile.name} (all instances)") - else: - pass + gid = global_id(tile_idx, local_inst) + inst_to_merged[local_inst] = global_to_merged.get(gid, -1) - border_mask = get_border_region_mask( - tile.points, tile.boundary, buffer, border_zone_end, neighbors - ) - border_points = tile.points[border_mask] - border_inst_ids = tile.instances[border_mask] - - # Get unique instances in border region (much smaller set than all instances) - border_unique_insts = set(np.unique(border_inst_ids)) - {0} - border_unique_insts &= kept_instances # Only kept instances - - if len(border_unique_insts) == 0: - continue - - if verbose: - print(f" Computing centroids for {len(border_unique_insts)} border instances...") - all_centroids = compute_centroids_vectorized(tile.points, tile.instances) - border_centroids = { - inst_id: all_centroids[inst_id] - for inst_id in border_unique_insts - if inst_id in all_centroids - } - - border_count = 0 - - # For each border instance, determine direction and extract full points - for inst_id in border_unique_insts: - if inst_id not in border_centroids: - continue - - centroid = border_centroids[inst_id] - cx, cy = centroid[0], centroid[1] - - # Determine border direction based on centroid position - border_direction = None - if neighbors["west"] is not None and cx < min_x + border_zone_end: - border_direction = "west" - elif neighbors["east"] is not None and cx > max_x - border_zone_end: - border_direction = "east" - elif neighbors["south"] is not None and cy < min_y + border_zone_end: - border_direction = "south" - elif neighbors["north"] is not None and cy > max_y - border_zone_end: - border_direction = "north" - - if border_direction is None: - continue - - # Extract full instance points (from original tile, not just border region) - inst_mask = tile.instances == inst_id - inst_points = tile.points[inst_mask] - - # Compute instance bounding box - inst_minx = inst_points[:, 0].min() - inst_maxx = inst_points[:, 0].max() - inst_miny = inst_points[:, 1].min() - inst_maxy = inst_points[:, 1].max() - - border_instances[tile_name][inst_id] = { - 'centroid': centroid, - 'points': inst_points, - 'boundary': (inst_minx, inst_maxx, inst_miny, inst_maxy), - 'direction': border_direction, - 'tile_idx': tile_idx - } - border_count += 1 - - # Update tracking for debug instances - if debug_instance_ids and inst_id in debug_instance_ids: - key = (tile_name, inst_id) - if key in instance_tracking: - instance_tracking[key]["in_border_region"] = True - instance_tracking[key]["border_direction"] = border_direction - print(f" DEBUG: Instance {inst_id} in border region ({border_direction})") - - if border_count > 0: - print(f" {tile.name}: {border_count} border instances") - - # Match instances between neighbor tiles - total_border_insts = sum(len(insts) for insts in border_instances.values()) - tiles_with_border = len([t for t in border_instances if border_instances[t]]) - if match_all_instances: - print(f" Found {total_border_insts} instances across {tiles_with_border} tiles (all instances)") - else: - print(f" Found {total_border_insts} border region instances across {tiles_with_border} tiles") - print(f" Processing tile pairs...") - - # Track which global IDs have already been matched to avoid duplicate checks - matched_gids = set() - - border_matches = 0 - total_bbox_checks = 0 - total_ff3d_computations = 0 - tiles_processed = 0 - - for i in range(len(tiles)): - tile_a = tiles[i] - # Use JSON-based neighbors when available - if tile_a.name in neighbors_by_tile: - neighbors_a = neighbors_by_tile[tile_a.name] - else: - neighbors_a = find_spatial_neighbors(tile_a.boundary, tile_a.name, tile_boundaries) - - for direction, neighbor_name in neighbors_a.items(): - if neighbor_name is None: - continue - - # Find neighbor tile index - tile_b_idx = tile_name_to_idx.get(neighbor_name) - if tile_b_idx is None: - continue - - tile_b = tiles[tile_b_idx] - - # Get instances from both tiles - if match_all_instances: - # Match ALL instances between neighbor tiles (no direction filtering) - border_insts_a = border_instances.get(tile_a.name, {}) - border_insts_b = border_instances.get(tile_b.name, {}) - else: - # Original logic: only match border instances in specific directions - border_insts_a = { - inst_id: data for inst_id, data in border_instances.get(tile_a.name, {}).items() - if data['direction'] == direction - } - border_insts_b = { - inst_id: data for inst_id, data in border_instances.get(tile_b.name, {}).items() - if data['direction'] == get_opposite_direction(direction) - } - - if not border_insts_a or not border_insts_b: + safe_instances = np.clip(tile.instances, 0, max_local_inst - 1) + remapped_instances = inst_to_merged[safe_instances] + valid_mask = remapped_instances != -1 + if not np.any(valid_mask): + print(f" Warning: {tile.name} has no valid points after filtering, skipping") continue - - # Progress: Show which tile pair is being processed - matches_before = border_matches - if match_all_instances: - print(f" Checking {tile_a.name} <-> {tile_b.name} ({direction} neighbors): " - f"{len(border_insts_a)} vs {len(border_insts_b)} instances", end=" ... ") - else: - print(f" Checking {tile_a.name} ({direction}) <-> {tile_b.name} ({get_opposite_direction(direction)}): " - f"{len(border_insts_a)} vs {len(border_insts_b)} border instances", end=" ... ") - - # Build list of candidate instances from tile B (not already matched) - candidates_b = [] - for inst_id_b, data_b in border_insts_b.items(): - gid_b = global_id(tile_b_idx, inst_id_b) - if gid_b not in matched_gids: - candidates_b.append((inst_id_b, gid_b, data_b)) - - if not candidates_b: - continue - - # For each instance in tile A, check overlap with all candidate instances in tile B - for inst_id_a, data_a in border_insts_a.items(): - gid_a = global_id(i, inst_id_a) - - # Skip if already matched - if gid_a in matched_gids: - continue - - bbox_a = data_a['boundary'] - - # Check each candidate in tile B - for inst_id_b, gid_b, data_b in candidates_b: - # Skip if already matched - if gid_b in matched_gids: - continue - - bbox_b = data_b['boundary'] - - # Quick bounding box overlap/nearby check (within 10cm tolerance) - total_bbox_checks += 1 - bbox_overlaps = bboxes_overlap(bbox_a, bbox_b, tolerance=0.1) - - # Check if we should debug this pair - should_debug = ( - debug_instance_ids is not None and - (inst_id_a in debug_instance_ids or inst_id_b in debug_instance_ids) - ) - - if should_debug: - print(f"\n DEBUG: Checking pair {inst_id_a} <-> {inst_id_b}") - print(f" BBox overlap check: {'PASS' if bbox_overlaps else 'FAIL'}") - - if not bbox_overlaps: - if should_debug: - print(f" Skipping: BBox doesn't overlap (within 10cm tolerance)") - continue - - # Now compute expensive FF3D overlap ratio - total_ff3d_computations += 1 - points_a = data_a['points'] - points_b = data_b['points'] - instances_a = np.full(len(points_a), inst_id_a, dtype=np.int32) - instances_b = np.full(len(points_b), inst_id_b, dtype=np.int32) - - overlap_ratios_dict, size_a, size_b = compute_ff3d_overlap_ratios( - instances_a, instances_b, points_a, points_b, correspondence_tolerance - ) - - overlap_ratio = overlap_ratios_dict.get((inst_id_a, inst_id_b), 0.0) - - # Debug logging for instance pairs - if should_debug: - centroid_a = data_a['centroid'] - centroid_b = data_b['centroid'] - size_a_val = size_a.get(inst_id_a, 0) - size_b_val = size_b.get(inst_id_b, 0) - - # Update tracking - key_a = (tile_a.name, inst_id_a) - key_b = (tile_b.name, inst_id_b) - if key_a in instance_tracking: - instance_tracking[key_a]["compared_with"].append({ - "tile": tile_b.name, - "instance": inst_id_b, - "overlap_ratio": overlap_ratio, - "matched": overlap_ratio >= overlap_threshold - }) - if key_b in instance_tracking: - instance_tracking[key_b]["compared_with"].append({ - "tile": tile_a.name, - "instance": inst_id_a, - "overlap_ratio": overlap_ratio, - "matched": overlap_ratio >= overlap_threshold - }) - - log_instance_pair_analysis( - inst_id_a, inst_id_b, - tile_a.name, tile_b.name, direction, - bbox_a, bbox_b, - overlap_ratio, overlap_threshold, - bbox_overlaps, - centroid_a, centroid_b, - size_a_val, size_b_val, - overlap_ratio >= overlap_threshold - ) - - if overlap_ratio >= overlap_threshold: - # Merge via Union-Find - root = uf.union(gid_a, gid_b) - matched_gids.add(gid_a) - matched_gids.add(gid_b) - border_matches += 1 - - # Update tracking for matched instances - if debug_instance_ids: - key_a = (tile_a.name, inst_id_a) - key_b = (tile_b.name, inst_id_b) - if key_a in instance_tracking: - instance_tracking[key_a]["matched_with"] = (tile_b.name, inst_id_b) - if key_b in instance_tracking: - instance_tracking[key_b]["matched_with"] = (tile_a.name, inst_id_a) - - if verbose: - print(f" ✓ Match: {tile_a.name}:{inst_id_a} <-> {tile_b.name}:{inst_id_b} (overlap: {overlap_ratio:.3f})") - - # Progress: Show results for this tile pair - matches_this_pair = border_matches - matches_before - if matches_this_pair > 0: - print(f"{matches_this_pair} match(es) found") - else: - print("no matches") - tiles_processed += 1 - - # Periodic progress update every 10 tile pairs - if tiles_processed % 10 == 0: - print(f" Progress: {tiles_processed} tile pairs processed, {border_matches} total matches so far...") - - if match_all_instances: - print(f" Matched {border_matches} instance pairs (all instances)") - else: - print(f" Matched {border_matches} border region instance pairs") - print(f" Performance: {total_bbox_checks} bbox checks, {total_ff3d_computations} FF3D computations") - print(f" ✓ Stage 3 completed: {stage_name} done") - - # Print instance tracking summary for debug instances - if debug_instance_ids and instance_tracking: - print(f"\n{'='*60}") - print("Instance Tracking Summary") - print(f"{'='*60}") - for (tile_name, local_id), info in sorted(instance_tracking.items()): - print(f"\nInstance {local_id} (Tile: {tile_name}):") - print(f" Global ID: {info['global_id']}") - print(f" Filtered in Stage 1: {'YES' if info['filtered_in_stage1'] else 'NO'}") - print(f" In border region: {'YES' if info['in_border_region'] else 'NO'}") - if info['in_border_region']: - print(f" Border direction: {info['border_direction']}") - print(f" Compared with {len(info['compared_with'])} instance(s):") - for comp in info['compared_with']: - print(f" - {comp['tile']}:{comp['instance']} (overlap: {comp['overlap_ratio']:.4f}, matched: {comp['matched']})") - if info['matched_with']: - print(f" Matched with: {info['matched_with'][0]}:{info['matched_with'][1]}") - else: - print(f" Matched with: NONE") - print(f"{'='*60}\n") - - # Get connected components - components = uf.get_components() - print(f" Connected components: {len(components)}") - print(f" ✓ Instance matching completed: {len(components)} merged instance groups") - - # Create mapping from global ID to final merged ID - global_to_merged = {} - merged_instance_sources = {} # merged_id -> list of source global IDs (for CSV tracking) - - for merged_id, (root, members) in enumerate(components.items(), start=1): - merged_instance_sources[merged_id] = list(members) - - if len(members) > 1 and verbose: - print(f" Merged ID {merged_id} created from {len(members)} global IDs: {sorted(members)}") - - for gid in members: - global_to_merged[gid] = merged_id - - # ========================================================================= - # Orphan Recovery: Recover filtered instances that would otherwise be lost - # ========================================================================= - # Problem: A tree can be filtered from BOTH tiles if segmented slightly - # differently (centroids in buffer zones of both tiles). - # - # Solution: Recover filtered instances from ANY buffer direction if - # the neighboring tile doesn't have the tree covered (checked via bbox overlap). - # This ensures no instances are lost, even if filtered from both tiles. - print("\n Checking for orphaned filtered instances...") - - # Build tile name to index map - tile_name_to_idx = {tile.name: idx for idx, tile in enumerate(tiles)} - - # Store tile index to name mapping for diagnostic logging later (used in renumbering) - tile_idx_to_name = {idx: tile.name for idx, tile in enumerate(tiles)} - - # Bounding boxes only for instances we need: orphans (filtered) + kept instances in border region. - # Use centroids to decide which kept instances are in border, then compute bbox only for those. - border_zone_end = buffer + border_zone_width - instance_bboxes = {} # tile_name -> {inst_id -> (min_xyz, max_xyz)} - for tile in tiles: - centroids = compute_centroids_vectorized(tile.points, tile.instances) - kept_instances = kept_instances_per_tile[tile.name] - neighbors = neighbors_by_tile.get(tile.name) or {"east": None, "west": None, "north": None, "south": None} - min_x, max_x, min_y, max_y = tile.boundary - bi_min_x = min_x + (buffer if neighbors.get("west") is not None else 0) - bi_max_x = max_x - (buffer if neighbors.get("east") is not None else 0) - bi_min_y = min_y + (buffer if neighbors.get("south") is not None else 0) - bi_max_y = max_y - (buffer if neighbors.get("north") is not None else 0) - bo_min_x = min_x + (border_zone_end if neighbors.get("west") is not None else 0) - bo_max_x = max_x - (border_zone_end if neighbors.get("east") is not None else 0) - bo_min_y = min_y + (border_zone_end if neighbors.get("south") is not None else 0) - bo_max_y = max_y - (border_zone_end if neighbors.get("north") is not None else 0) - - def in_border(cx: float, cy: float) -> bool: - return ( - (neighbors.get("west") is not None and bi_min_x <= cx <= bo_min_x) - or (neighbors.get("east") is not None and bo_max_x <= cx <= bi_max_x) - or (neighbors.get("south") is not None and bi_min_y <= cy <= bo_min_y) - or (neighbors.get("north") is not None and bo_max_y <= cy <= bi_max_y) + output_dims = { + instance_dimension: cast_instances_for_output( + remapped_instances[valid_mask], + instance_dimension, + ) + } + output_dims.update({ + dim_name: values[valid_mask] + for dim_name, values in tile.extra_dims.items() + }) + + source_file = source_file_by_tile_name[tile.name] + output_file = output_tiles_dir / f"{tile.name}.laz" + write_loaded_point_cloud( + source_file, + output_file, + tile.points[valid_mask], + output_dims, + source_indices=np.flatnonzero(valid_mask), ) - - kept_in_border = { - inst_id for inst_id in kept_instances - if inst_id in centroids and in_border(centroids[inst_id][0], centroids[inst_id][1]) + written_tiles += 1 + total_output_points += int(np.count_nonzero(valid_mask)) + print(f" Wrote {output_file.name}: {np.count_nonzero(valid_mask):,} points") + + import csv + + csv_output_path = output_merged.parent / f"{output_merged.stem}_instance_metadata.csv" + csv_output_path.parent.mkdir(parents=True, exist_ok=True) + instances_with_clusters = { + merged_id + for merged_id, sources in merged_instance_sources.items() + if len(sources) > 1 } - need_bbox = filtered_instances_per_tile[tile.name] | kept_in_border + final_instance_ids = sorted(set(global_to_merged.values()) - {0, -1}) - sort_idx = np.argsort(tile.instances) - sorted_inst = tile.instances[sort_idx] - sorted_points = tile.points[sort_idx] - unique_inst, first_idx, counts = np.unique( - sorted_inst, return_index=True, return_counts=True - ) - bboxes = {} - for i, inst_id in enumerate(unique_inst): - if inst_id <= 0 or inst_id not in need_bbox: - continue - start = first_idx[i] - end = start + counts[i] - pts = sorted_points[start:end] - bboxes[inst_id] = (pts.min(axis=0), pts.max(axis=0)) - instance_bboxes[tile.name] = bboxes + with open(csv_output_path, "w", newline="") as csvfile: + writer = csv.writer(csvfile) + writer.writerow([instance_dimension, "has_added_clusters"]) + for final_id in final_instance_ids: + writer.writerow([final_id, 1 if final_id in instances_with_clusters else 0]) - # Build spatial index of kept instance centers in border region only - # (tile_bboxes only contains border kept + filtered, so every kept in tile_bboxes is border) - print(" Building spatial index of kept instances (border region only)...") - kept_instance_data = [] # List of (center_x, center_y, tile_idx, inst_id, bbox_min, bbox_max) + csv_copy_path = output_tiles_dir / csv_output_path.name + if csv_copy_path != csv_output_path: + import shutil + shutil.copy2(str(csv_output_path), str(csv_copy_path)) - for tile_idx, tile in enumerate(tiles): - tile_bboxes = instance_bboxes[tile.name] - kept_instances = kept_instances_per_tile[tile.name] - for inst_id in kept_instances: - if inst_id not in tile_bboxes: - continue - bbox_min, bbox_max = tile_bboxes[inst_id] - center_x = (bbox_min[0] + bbox_max[0]) / 2.0 - center_y = (bbox_min[1] + bbox_max[1]) / 2.0 - kept_instance_data.append((center_x, center_y, tile_idx, inst_id, bbox_min, bbox_max)) - - # Build cKDTree for spatial queries (5m search radius) - search_radius = 5.0 # 5m radius - if kept_instance_data: - centers = np.array([(x, y) for x, y, _, _, _, _ in kept_instance_data]) - kept_tree = cKDTree(centers) - else: - kept_tree = None - print(" Warning: No kept instances found for spatial indexing") - - # Pre-build KDTrees for all kept instances in border (used by parallel orphan check) - neighbor_trees: Dict[Tuple[int, int], cKDTree] = {} - for _, _, check_tile_idx, neighbor_inst, _, _ in kept_instance_data: - cache_key = (check_tile_idx, neighbor_inst) - if cache_key in neighbor_trees: - continue - check_tile = tiles[check_tile_idx] - neighbor_mask = check_tile.instances == neighbor_inst - neighbor_points = check_tile.points[neighbor_mask] - if len(neighbor_points) > 0: - neighbor_trees[cache_key] = cKDTree(neighbor_points[:, :2]) - - overlap_tolerance = 1.0 # 1m tolerance for tree instances - - def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: - """Returns (tile_idx, local_inst, covered). True if a neighbor covers this orphan.""" - tile_idx, local_inst = item - tile = tiles[tile_idx] - tile_name = tile.name - tile_bboxes = instance_bboxes[tile_name] - if local_inst <= 0 or local_inst not in tile_bboxes: - return (tile_idx, local_inst, True) - if buffer_direction_per_tile[tile_name].get(local_inst) is None: - return (tile_idx, local_inst, True) - fmin, fmax = tile_bboxes[local_inst] - orphan_center = ((fmin[0] + fmax[0]) / 2.0, (fmin[1] + fmax[1]) / 2.0) - inst_mask = tile.instances == local_inst - filtered_points = tile.points[inst_mask] - if len(filtered_points) == 0: - return (tile_idx, local_inst, True) - neighbor_has_tree = False - if kept_tree is not None: - nearby_indices = kept_tree.query_ball_point(orphan_center, r=search_radius) - for idx in nearby_indices: - _, _, check_tile_idx, neighbor_inst, nmin, nmax = kept_instance_data[idx] - if check_tile_idx == tile_idx: - continue - if (fmax[0] < nmin[0] - overlap_tolerance or fmin[0] > nmax[0] + overlap_tolerance or - fmax[1] < nmin[1] - overlap_tolerance or fmin[1] > nmax[1] + overlap_tolerance): - continue - cache_key = (check_tile_idx, neighbor_inst) - neighbor_tree = neighbor_trees.get(cache_key) - if neighbor_tree is None: - continue - distances, _ = neighbor_tree.query(filtered_points[:, :2], k=1) - n_within = np.sum(distances <= overlap_tolerance) - fraction_within = n_within / len(filtered_points) - if fraction_within > 0.50: - neighbor_gid = global_id(check_tile_idx, neighbor_inst) - if neighbor_gid in global_to_merged: - neighbor_has_tree = True - break - else: - neighbor_has_tree = True - break - return (tile_idx, local_inst, neighbor_has_tree) - - # Collect orphan candidates - orphan_candidates: List[Tuple[int, int]] = [] - for tile_idx, tile in enumerate(tiles): - filtered_instances = filtered_instances_per_tile[tile.name] - buffer_directions = buffer_direction_per_tile[tile.name] - tile_bboxes = instance_bboxes[tile.name] - for local_inst in filtered_instances: - if local_inst <= 0 or local_inst not in tile_bboxes: - continue - if buffer_directions.get(local_inst) is None: - continue - inst_mask = tile.instances == local_inst - if np.sum(inst_mask) == 0: - continue - orphan_candidates.append((tile_idx, local_inst)) - - next_merged_id = max(global_to_merged.values()) + 1 if global_to_merged else 1 - recovered_count = 0 - skipped_covered = 0 - orphan_parallel_workers = 10 - - if orphan_candidates: - print(f" Checking {len(orphan_candidates)} orphan candidates with {orphan_parallel_workers} workers...", flush=True) - with ThreadPoolExecutor(max_workers=orphan_parallel_workers) as executor: - orphan_results = list(executor.map(_check_one_orphan_covered, orphan_candidates)) - for (tile_idx, local_inst, covered) in orphan_results: - if covered: - skipped_covered += 1 - continue - tile = tiles[tile_idx] - gid = global_id(tile_idx, local_inst) - global_to_merged[gid] = next_merged_id - merged_instance_sources[next_merged_id] = [gid] - if verbose: - print(f" Recovered orphan - global_id={gid} (tile={tile.name}, local={local_inst}) -> merged_id={next_merged_id}") - kept_instances_per_tile[tile.name].add(local_inst) - next_merged_id += 1 - recovered_count += 1 - - del instance_bboxes - - if recovered_count > 0 or skipped_covered > 0: - print(f" Recovered {recovered_count} orphaned instances") - print(f" Skipped {skipped_covered} instances (neighbor has overlapping tree)") - else: - print(f" No orphaned instances found") + print(f" ✓ Wrote {written_tiles} filtered output tiles, {total_output_points:,} points") + print(f" Instance metadata CSV: {csv_output_path}") + print(f"\n{'=' * 60}") + print("Merge complete!") + print(f"{'=' * 60}") + return # ========================================================================= # Stage 4: Merge and Deduplicate @@ -5416,7 +477,7 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: max_local_inst = tile.instances.max() + 1 inst_to_merged = np.full(max_local_inst, -1, dtype=np.int32) - + if max_local_inst > 0: inst_to_merged[0] = 0 @@ -5432,7 +493,7 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: all_points.append(tile.points) all_instances.append(remapped_instances) - + # Collect extra dims (passenger data) - use zeros for dims missing from this tile for dim_name in all_extra_dim_names: if dim_name in tile.extra_dims: @@ -5484,8 +545,6 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: print(f" Removed {n_removed:,} duplicate points ({100*n_removed/total_before:.1f}%)") print(f" ✓ Stage 4 completed: {len(merged_points):,} points, {n_tree_instances} tree instances") - del uf - del instance_sizes del global_to_merged gc.collect() @@ -5501,19 +560,19 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: nonzero_count = pos_mask.sum() zero_count = len(merged_instances) - nonzero_count print(f" Instance points: {nonzero_count:,}, ground points: {zero_count:,}") - + pos_points = merged_points[pos_mask] pos_instances = merged_instances[pos_mask] - + sort_idx = np.argsort(pos_instances) sorted_points = pos_points[sort_idx] sorted_instances = pos_instances[sort_idx] - + unique_inst, first_idx, inst_counts = np.unique( sorted_instances, return_index=True, return_counts=True ) print(f" Processing {len(unique_inst):,} unique instances...", flush=True) - + merged_instances, _ = merge_small_volume_instances( merged_points, merged_instances, @@ -5547,7 +606,7 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: pos_mask = merged_instances > 0 pos_points = merged_points[pos_mask] pos_instances = merged_instances[pos_mask] - + # Sort and compute unique instances sort_idx = np.argsort(pos_instances) sorted_points = pos_points[sort_idx] @@ -5577,12 +636,12 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: print(f" Instances renumbered from north to south") max_old_id = int(merged_instances.max()) + 1 - + instance_lookup = np.zeros(max_old_id, dtype=np.int32) for old_id, new_id in old_to_new.items(): if old_id < max_old_id: instance_lookup[old_id] = new_id - + merged_instances = instance_lookup[merged_instances] print(f" Final instance count: {len(sorted_by_location)}") @@ -5590,8 +649,6 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: # ========================================================================= # Save merged output (optional - can be skipped with skip_merged_file=True) # ========================================================================= - merged_file_for_downstream = output_merged - if skip_merged_file: print(f"\n{'=' * 60}") print("Saving merged output (SKIPPED)") @@ -5599,12 +656,6 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: print(f" Skipped merged LAZ file creation (--skip_merged_file)") print(f" Total points: {len(merged_points):,}") print(f" Total instances: {len(sorted_by_location)}") - merged_file_for_downstream = output_merged.parent / (output_merged.stem + "_temp_for_downstream.laz") - _write_points_with_dimensions_to_laz( - merged_file_for_downstream, - merged_points, - {instance_dimension: merged_instances, **merged_extra_dims}, - ) else: print(f"\n{'=' * 60}") print("Saving merged output") @@ -5615,12 +666,32 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: # Write initial merged to a temp path so we can either enrich it or use as final merged_init = output_merged.parent / (output_merged.stem + "_init.laz") - _write_points_with_dimensions_to_laz( - merged_init, + header = merged_product_header( merged_points, - {instance_dimension: merged_instances, **merged_extra_dims}, + original_input_dir, + original_tiles_dir, + ) + + output_las = laspy.LasData(header) + output_las.x = merged_points[:, 0] + output_las.y = merged_points[:, 1] + output_las.z = merged_points[:, 2] + + # Add instance dimension and passenger extra dimensions from the merge. + extra_dims_params = [instance_extra_bytes_params(instance_dimension, merged_instances)] + for dim_name, dim_arr in merged_extra_dims.items(): + extra_dims_params.append(laspy.ExtraBytesParams(name=dim_name, type=dim_arr.dtype)) + output_las.add_extra_dims(extra_dims_params) + + setattr(output_las, instance_dimension, cast_instances_for_output(merged_instances, instance_dimension)) + for dim_name, dim_arr in merged_extra_dims.items(): + setattr(output_las, dim_name, dim_arr) + + output_las.write( + str(merged_init), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel ) - merged_file_for_downstream = merged_init + + del output_las gc.collect() print(f" Saved merged (initial): {merged_init}") @@ -5630,12 +701,11 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: import shutil # Add original-file dimensions and write directly to final output (so merged file is always enriched when requested) if original_input_dir is not None and transfer_original_dims_to_merged: - enriched_output = output_merged try: add_original_dimensions_to_merged( - merged_file_for_downstream, + merged_init, original_input_dir, - enriched_output, + output_merged, tolerance=0.1, retile_buffer=retile_buffer, num_threads=num_threads, @@ -5643,17 +713,14 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: print(f" Enriched merged file with original-file dimensions: {output_merged}") except Exception as e: print(f" Warning: Could not add original dimensions to merged file: {e}") - try: - if enriched_output.exists(): - enriched_output.unlink() - except OSError: - pass shutil.copy2(str(merged_init), str(output_merged)) print(f" Wrote un-enriched merged to {output_merged}") else: shutil.copy2(str(merged_init), str(output_merged)) print(f" Saved merged output: {output_merged}") + validate_merged_output_contract(output_merged, instance_dimension) + try: merged_init.unlink() except OSError: @@ -5669,36 +736,36 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: # Create CSV with instance metadata # ========================================================================= import csv - + csv_output_path = output_merged.parent / f"{output_merged.stem}_instance_metadata.csv" - + instances_with_clusters = set() for old_merged_id, sources in merged_instance_sources.items(): if len(sources) > 1: final_id = old_to_new.get(old_merged_id, None) if final_id is not None and final_id > 0: instances_with_clusters.add(final_id) - + print(f"\n Writing instance metadata CSV: {csv_output_path}") print(f" Found {len(instances_with_clusters)} final instances with added clusters from cross-tile merging") - + # Collect all final instance IDs final_instance_ids = sorted(set(old_to_new.values()) - {0}) - + with open(csv_output_path, "w", newline="") as csvfile: writer = csv.writer(csvfile) writer.writerow([instance_dimension, "has_added_clusters"]) - + for final_id in final_instance_ids: has_clusters = final_id in instances_with_clusters writer.writerow([final_id, 1 if has_clusters else 0]) - + csv_copy_path = output_tiles_dir / csv_output_path.name if csv_copy_path != csv_output_path: import shutil shutil.copy2(str(csv_output_path), str(csv_copy_path)) print(f" Copied CSV to output tiles folder: {csv_copy_path}") - + del merged_instance_sources gc.collect() @@ -5708,17 +775,18 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: print(f"\n{'=' * 60}") print("Stage 6: Retiling to Original Files") print(f"{'=' * 60}") - - retile_to_original_files_streaming( - merged_file=merged_file_for_downstream, - original_tiles_dir=original_tiles_dir, - output_dir=output_tiles_dir, + + retile_to_original_files( + merged_points, + merged_instances, + merged_extra_dims, + None, + original_tiles_dir, + output_tiles_dir, tolerance=0.1, + num_threads=num_threads, retile_buffer=retile_buffer, - chunk_size=chunk_size, instance_dimension=instance_dimension, - threedtrees_dims=threedtrees_dims, - threedtrees_suffix=threedtrees_suffix, ) print(f" ✓ Stage 6 completed: Retiled to original files") @@ -5729,29 +797,26 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: print(f"\n{'=' * 60}") print("Stage 7: Remapping to Original Input Files") print(f"{'=' * 60}") - + original_output_dir = output_tiles_dir.parent / "original_with_predictions" - remap_to_original_input_files_streaming( - merged_file=merged_file_for_downstream, - original_input_dir=original_input_dir, - output_dir=original_output_dir, + all_merged_dims = {instance_dimension: merged_instances, **merged_extra_dims} + remap_to_original_input_files( + merged_points, + all_merged_dims, + None, + original_input_dir, + original_output_dir, tolerance=0.1, + num_threads=num_threads, retile_buffer=retile_buffer, - chunk_size=chunk_size, threedtrees_dims=threedtrees_dims, threedtrees_suffix=threedtrees_suffix, - target_dims=_target_dims, + prefer_copc_sources=False, ) print(f" ✓ Stage 7 completed: Remapped to original input files") else: print(f"\n Note: --original-input-dir not provided, skipping Stage 7 (remap to original input files)") - if skip_merged_file and merged_file_for_downstream.exists(): - try: - merged_file_for_downstream.unlink() - except OSError: - pass - print(f"\n{'=' * 60}") print("Merge complete!") print(f"{'=' * 60}") @@ -5763,209 +828,10 @@ def _check_one_orphan_covered(item: Tuple[int, int]) -> Tuple[int, int, bool]: def main(): - parser = argparse.ArgumentParser( - description="Tile Merger - Merge segmented point cloud tiles with species ID preservation", - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - - parser.add_argument( - "--input-dir", - "-i", - type=Path, - required=True, - help="Directory containing segmented LAZ tiles", - ) - - parser.add_argument( - "--original-tiles-dir", - type=Path, - default=None, - help="Directory containing original tile files for retiling", - ) - - parser.add_argument( - "--output-merged", - "-o", - type=Path, - required=True, - help="Output path for merged LAZ file", - ) - - parser.add_argument( - "--output-tiles-dir", - type=Path, - default=None, - help="Output directory for retiled files", - ) - - parser.add_argument( - "--original-input-dir", - type=Path, - default=None, - help="Directory with original input LAZ files for final remap (optional, enables Stage 7)", - ) - - parser.add_argument( - "--buffer", - type=float, - default=10.0, - help="Buffer zone distance in meters (default: 10.0)", - ) - - parser.add_argument( - "--overlap-threshold", - "--ff3d-threshold", - type=float, - default=0.3, - dest="overlap_threshold", - help="Overlap ratio threshold for instance matching (default: 0.3 = 30%%)", - ) - - parser.add_argument( - "--correspondence-tolerance", - type=float, - default=0.1, - help="Max distance for point correspondence in meters (default: 0.1). " - "Should be small (~10cm) to only match actual duplicate points from overlapping tiles.", - ) + from merge_tiles_cli import main as cli_main - parser.add_argument( - "--max-volume-for-merge", - type=float, - default=4.0, - help="Max convex hull volume (m³) for small instance merging (default: 4.0)", - ) - - parser.add_argument( - "--border-zone-width", - type=float, - default=10.0, - help="Width of border zone beyond buffer for instance matching (default: 10.0m)", - ) - - parser.add_argument( - "--workers", - type=int, - default=4, - dest="num_threads", - help="Number of workers for parallel processing (default: 4)", - ) - - parser.add_argument( - "--disable-matching", - "--disable-ff3d", - action="store_true", - dest="disable_matching", - help="Disable cross-tile instance matching", - ) - - parser.add_argument( - "--disable-volume-merge", - action="store_true", - help="Disable small volume instance merging", - ) - - parser.add_argument( - "--skip-merged-file", - action="store_true", - help="Skip creating merged LAZ file (only create retiled outputs)", - ) - - parser.add_argument( - "--verbose", "-v", action="store_true", help="Print detailed merge decisions" - ) - - parser.add_argument( - "--debug-instances", - type=str, - default=None, - help="Comma-separated list of instance IDs to debug (e.g., '485,73'). Enables detailed logging for these instances in Stage 3.", - ) - - parser.add_argument( - "--match-all-instances", - action="store_true", - dest="match_all_instances", - help="Match all instances between neighbor tiles, not just border region instances. " - "When enabled, Stage 3 will check all instances in overlapping tiles for matching, " - "not just those in border regions. Default: False (only border region instances are matched).", - ) - - parser.add_argument( - "--retile-buffer", - type=float, - default=2.0, - help="Spatial buffer expansion in meters for filtering merged points during retiling (fixed: 2.0m)", - ) - - parser.add_argument( - "--retile-max-radius", - type=float, - default=0.2, - help="Maximum distance threshold in meters for cKDTree nearest neighbor matching during retiling (default: 2.0m)", - ) - - parser.add_argument( - "--instance-dimension", - type=str, - default="PredInstance", - help="Name of the instance ID dimension in input files (default: PredInstance, fallback: treeID)", - ) - - parser.add_argument( - "--tile-bounds-json", - type=Path, - required=True, - help="Path to tile_bounds_tindex.json (required; used for neighbor graph)", - ) - - parser.add_argument( - "--standardization-json", - type=Path, - default=None, - help="Optional collection_summary.json; when provided, remap/original-dimension transfer only populates listed dimensions.", - ) - - args = parser.parse_args() - - # Parse debug instance IDs - debug_instance_ids = None - if args.debug_instances: - try: - debug_instance_ids = set(int(x.strip()) for x in args.debug_instances.split(',')) - except ValueError: - print(f"ERROR: Invalid --debug-instances format: {args.debug_instances}") - print("Expected format: comma-separated integers (e.g., '485,73')") - sys.exit(1) - - merge_tiles( - input_dir=args.input_dir, - original_tiles_dir=args.original_tiles_dir, - output_merged=args.output_merged, - output_tiles_dir=args.output_tiles_dir, - tile_bounds_json=args.tile_bounds_json, - original_input_dir=args.original_input_dir, - buffer=args.buffer, - overlap_threshold=args.overlap_threshold, - correspondence_tolerance=args.correspondence_tolerance, - max_volume_for_merge=args.max_volume_for_merge, - border_zone_width=args.border_zone_width, - num_threads=args.num_threads, - enable_matching=not args.disable_matching, - enable_volume_merge=not args.disable_volume_merge, - skip_merged_file=args.skip_merged_file, - verbose=args.verbose, - retile_buffer=args.retile_buffer, - retile_max_radius=args.retile_max_radius, - debug_instance_ids=debug_instance_ids, - match_all_instances=args.match_all_instances, - instance_dimension=args.instance_dimension, - standardization_json=args.standardization_json, - ) + cli_main(merge_tiles) if __name__ == "__main__": main() - _target_dims = None - if standardization_json is not None: - _target_dims = load_standardization_dims(standardization_json) diff --git a/src/merge_tiles_cli.py b/src/merge_tiles_cli.py new file mode 100644 index 0000000..ebd7bbb --- /dev/null +++ b/src/merge_tiles_cli.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""Command-line parser for the SmartTile merge task.""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + + +def main(merge_tiles_func): + parser = argparse.ArgumentParser( + description="Tile Merger - Merge segmented point cloud tiles with species ID preservation", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + + parser.add_argument( + "--input-dir", + "-i", + type=Path, + required=True, + help="Directory containing segmented LAZ tiles", + ) + + parser.add_argument( + "--original-tiles-dir", + type=Path, + default=None, + help="Directory containing original tile files for retiling", + ) + + parser.add_argument( + "--output-merged", + "-o", + type=Path, + required=True, + help="Output path for merged LAZ file", + ) + + parser.add_argument( + "--output-tiles-dir", + type=Path, + default=None, + help="Output directory for retiled files", + ) + + parser.add_argument( + "--original-input-dir", + type=Path, + default=None, + help="Directory with original input LAZ files for final remap (optional, enables Stage 7)", + ) + + parser.add_argument( + "--buffer", + type=float, + default=10.0, + help="Buffer zone distance in meters (default: 10.0)", + ) + + parser.add_argument( + "--overlap-threshold", + "--ff3d-threshold", + type=float, + default=0.3, + dest="overlap_threshold", + help="Overlap ratio threshold for instance matching (default: 0.3 = 30%%)", + ) + + parser.add_argument( + "--correspondence-tolerance", + type=float, + default=0.1, + help="Max distance for point correspondence in meters (default: 0.1). " + "Should be small (~10cm) to only match actual duplicate points from overlapping tiles.", + ) + + parser.add_argument( + "--max-volume-for-merge", + type=float, + default=4.0, + help="Max convex hull volume (m³) for small instance merging (default: 4.0)", + ) + + parser.add_argument( + "--border-zone-width", + type=float, + default=10.0, + help="Width of border zone beyond buffer for instance matching (default: 10.0m)", + ) + + parser.add_argument( + "--workers", + type=int, + default=4, + dest="num_threads", + help="Number of workers for parallel processing (default: 4)", + ) + + parser.add_argument( + "--disable-matching", + "--disable-ff3d", + action="store_true", + dest="disable_matching", + help="Disable cross-tile instance matching", + ) + + parser.add_argument( + "--disable-volume-merge", + action="store_true", + help="Disable small volume instance merging", + ) + + parser.add_argument( + "--skip-merged-file", + action="store_true", + help="Skip creating merged LAZ file (only create retiled outputs)", + ) + + parser.add_argument( + "--verbose", "-v", action="store_true", help="Print detailed merge decisions" + ) + + parser.add_argument( + "--debug-instances", + type=str, + default=None, + help="Comma-separated list of instance IDs to debug (e.g., '485,73'). Enables detailed logging for these instances in Stage 3.", + ) + + parser.add_argument( + "--match-all-instances", + action="store_true", + dest="match_all_instances", + help="Match all instances between neighbor tiles, not just border region instances. " + "When enabled, Stage 3 will check all instances in overlapping tiles for matching, " + "not just those in border regions. Default: False (only border region instances are matched).", + ) + + parser.add_argument( + "--retile-buffer", + type=float, + default=2.0, + help="Spatial buffer expansion in meters for filtering merged points during retiling (fixed: 2.0m)", + ) + + parser.add_argument( + "--retile-max-radius", + type=float, + default=0.2, + help="Maximum distance threshold in meters for cKDTree nearest neighbor matching during retiling (default: 2.0m)", + ) + + parser.add_argument( + "--instance-dimension", + type=str, + default="PredInstance", + help="Name of the instance ID dimension in input files (default: PredInstance, fallback: treeID)", + ) + + parser.add_argument( + "--tile-bounds-json", + type=Path, + required=True, + help="Path to tile_bounds_tindex.json (required; used for neighbor graph)", + ) + + args = parser.parse_args() + + # Parse debug instance IDs + debug_instance_ids = None + if args.debug_instances: + try: + debug_instance_ids = set(int(x.strip()) for x in args.debug_instances.split(',')) + except ValueError: + print(f"ERROR: Invalid --debug-instances format: {args.debug_instances}") + print("Expected format: comma-separated integers (e.g., '485,73')") + sys.exit(1) + + merge_tiles_func( + input_dir=args.input_dir, + original_tiles_dir=args.original_tiles_dir, + output_merged=args.output_merged, + output_tiles_dir=args.output_tiles_dir, + tile_bounds_json=args.tile_bounds_json, + original_input_dir=args.original_input_dir, + buffer=args.buffer, + overlap_threshold=args.overlap_threshold, + correspondence_tolerance=args.correspondence_tolerance, + max_volume_for_merge=args.max_volume_for_merge, + border_zone_width=args.border_zone_width, + num_threads=args.num_threads, + enable_matching=not args.disable_matching, + enable_volume_merge=not args.disable_volume_merge, + skip_merged_file=args.skip_merged_file, + verbose=args.verbose, + retile_buffer=args.retile_buffer, + retile_max_radius=args.retile_max_radius, + debug_instance_ids=debug_instance_ids, + match_all_instances=args.match_all_instances, + instance_dimension=args.instance_dimension, + ) + + diff --git a/src/output_remap.py b/src/output_remap.py new file mode 100644 index 0000000..6e2a965 --- /dev/null +++ b/src/output_remap.py @@ -0,0 +1,1301 @@ +#!/usr/bin/env python3 +"""Retile/remap merged SmartTile predictions back onto original point clouds.""" + +from __future__ import annotations + +import gc +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Dict, List, Optional + +import laspy +import numpy as np +from scipy.spatial import cKDTree + +from dimension_transfer import next_available_suffix, suffixes_for_collision +from instance_labels import cast_instances_for_output, instance_extra_bytes_params +from point_cloud_metadata import ( + copy_single_source_header, + extra_bytes_params_from_dimension_info, + extra_bytes_params_from_params, + point_cloud_files, + raw_point_cloud_files, +) + + +MIN_ORIGINAL_REMAP_MATCH_FRACTION = 0.99 + + +def _match_fraction_is_acceptable(matched: int, total: int, min_fraction: float) -> bool: + if total <= 0: + return True + return (matched / total) >= min_fraction + + +def _match_quality_message( + matched: int, + total: int, + tolerance: float, + min_fraction: float, + context: str = "points", +) -> str: + min_pct = min_fraction * 100.0 + return ( + f"Merged points matched {matched:,}/{total:,} {context} " + f"within tolerance {tolerance} m; requires at least {min_pct:.1f}%" + ) + + +def _process_single_tile(args): + """Process one original tile when retiling merged predictions.""" + ( + orig_file, + output_file, + merged_points, + merged_instances, + merged_extra_dims, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + instance_dimension, + ) = args + + try: + with laspy.open(str(orig_file), laz_backend=laspy.LazBackend.LazrsParallel) as f: + bounds = (f.header.x_min, f.header.x_max, f.header.y_min, f.header.y_max) + n_orig_points = f.header.point_count + + mask = ( + (merged_points[:, 0] >= bounds[0] - spatial_buffer) + & (merged_points[:, 0] <= bounds[1] + spatial_buffer) + & (merged_points[:, 1] >= bounds[2] - spatial_buffer) + & (merged_points[:, 1] <= bounds[3] + spatial_buffer) + ) + + local_merged_points = merged_points[mask] + local_merged_instances = merged_instances[mask] + local_merged_extras = {name: arr[mask] for name, arr in merged_extra_dims.items()} + + if len(local_merged_points) == 0: + return (orig_file.name, 0, n_orig_points, 0, False, "No merged points in tile region") + + local_tree = cKDTree(local_merged_points) + orig_las = laspy.read(str(orig_file), laz_backend=laspy.LazBackend.LazrsParallel) + orig_points = np.empty((n_orig_points, 3), dtype=np.float64) + orig_points[:, 0] = orig_las.x + orig_points[:, 1] = orig_las.y + orig_points[:, 2] = orig_las.z + + distances, indices = local_tree.query(orig_points, workers=kdtree_workers) + matched_mask = distances <= tolerance + matched = int(np.count_nonzero(matched_mask)) + if matched != n_orig_points: + return ( + orig_file.name, + matched, + n_orig_points, + 0, + False, + f"Merged points matched {matched:,}/{n_orig_points:,} " + f"points within tolerance {tolerance} m", + ) + new_instances = local_merged_instances[indices] + new_extras = {name: arr[indices] for name, arr in local_merged_extras.items()} + + new_header = copy_single_source_header( + orig_las.header, + preserve_extra_dimensions=False, + ) + output_las = laspy.LasData(new_header) + + orig_standard_names = set(orig_las.point_format.dimension_names) + orig_extra_dim_names = {dim.name for dim in orig_las.point_format.extra_dimensions} + orig_dim_names = orig_standard_names | orig_extra_dim_names + merged_dim_names = {instance_dimension} | set(new_extras.keys()) + collision = orig_dim_names & merged_dim_names + + used_names = set(orig_dim_names) + orig_rename = {} + merged_rename = {} + for name in sorted(collision): + original_name, merged_name = suffixes_for_collision(name, used_names) + orig_rename[name] = original_name + merged_rename[name] = merged_name + + output_extra_dim_names = {dim.name for dim in output_las.point_format.extra_dimensions} + extra_dims_to_add = [] + for dim in orig_las.point_format.extra_dimensions: + out_name = orig_rename.get(dim.name, dim.name) + if out_name not in output_extra_dim_names: + extra_dims_to_add.append(extra_bytes_params_from_dimension_info(dim, name=out_name)) + output_extra_dim_names.add(out_name) + + inst_out_name = merged_rename.get(instance_dimension, instance_dimension) + if inst_out_name not in output_extra_dim_names: + extra_dims_to_add.append(instance_extra_bytes_params(inst_out_name, new_instances)) + output_extra_dim_names.add(inst_out_name) + + for dim_name, values in new_extras.items(): + out_name = merged_rename.get(dim_name, dim_name) + if out_name not in output_extra_dim_names: + if merged_extra_dim_params and dim_name in merged_extra_dim_params: + params = merged_extra_dim_params[dim_name] + extra_dims_to_add.append(extra_bytes_params_from_params(params, name=out_name)) + else: + extra_dims_to_add.append(laspy.ExtraBytesParams(name=out_name, type=values.dtype)) + output_extra_dim_names.add(out_name) + + if extra_dims_to_add: + output_las.add_extra_dims(extra_dims_to_add) + + for dim_name in orig_las.point_format.dimension_names: + try: + if hasattr(orig_las, dim_name): + setattr(output_las, dim_name, getattr(orig_las, dim_name)) + except Exception: + pass + for dim in orig_las.point_format.extra_dimensions: + name = dim.name + out_name = orig_rename.get(name, name) + if hasattr(orig_las, name): + try: + setattr(output_las, out_name, getattr(orig_las, name)) + except Exception: + pass + + setattr( + output_las, + merged_rename.get(instance_dimension, instance_dimension), + cast_instances_for_output(new_instances, instance_dimension), + ) + for dim_name, values in new_extras.items(): + setattr(output_las, merged_rename.get(dim_name, dim_name), values) + + output_las.write( + str(output_file), + do_compress=True, + laz_backend=laspy.LazBackend.LazrsParallel, + ) + + del orig_las + del output_las + + unique_inst = len(np.unique(new_instances[new_instances > 0])) + return (orig_file.name, matched, n_orig_points, unique_inst, True, "OK") + + except Exception as e: + return (orig_file.name, 0, 0, 0, False, str(e)) + + +def retile_to_original_files( + merged_points: np.ndarray, + merged_instances: np.ndarray, + merged_extra_dims: Dict[str, np.ndarray], + merged_extra_dim_params: Optional[Dict[str, laspy.ExtraBytesParams]], + original_tiles_dir: Path, + output_dir: Path, + tolerance: float = 0.1, + num_threads: int = 8, + chunk_size: int = 1_000_000, + parallel_tiles: int = 1, + retile_buffer: float = 2.0, + instance_dimension: str = "PredInstance", +): + """Map merged instance IDs back to original tile point clouds.""" + print(f"\n{'=' * 60}", flush=True) + print("Retiling merged results to original tile files", flush=True) + print(f"{'=' * 60}", flush=True) + + original_files = point_cloud_files(original_tiles_dir) + + if len(original_files) == 0: + print(f" No LAZ/LAS files found in {original_tiles_dir}", flush=True) + return + + print(f" Found {len(original_files)} original tile files", flush=True) + output_dir.mkdir(parents=True, exist_ok=True) + + spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer + tiles_to_process = [] + skipped = 0 + for orig_file in original_files: + output_name = orig_file.name.replace(".copc.laz", ".laz") + output_file = output_dir / output_name + if output_file.exists(): + skipped += 1 + else: + tiles_to_process.append((orig_file, output_file)) + + if skipped > 0: + print(f" Skipping {skipped} already processed tiles", flush=True) + + if len(tiles_to_process) == 0: + print(" All tiles already processed!", flush=True) + return + + print(f" Processing {len(tiles_to_process)} tiles...", flush=True) + + kdtree_workers = -1 + process_args = [ + ( + orig_file, + output_file, + merged_points, + merged_instances, + merged_extra_dims, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + instance_dimension, + ) + for orig_file, output_file in tiles_to_process + ] + + failures = [] + if parallel_tiles > 1: + completed = 0 + with ThreadPoolExecutor(max_workers=parallel_tiles) as executor: + for result in executor.map(_process_single_tile, process_args): + filename, matched, total, unique_inst, success, message = result + completed += 1 + match_pct = (matched / total * 100) if total > 0 else 0 + if success: + print( + f" [{completed}/{len(tiles_to_process)}] {filename}: " + f"{matched:,}/{total:,} matched ({match_pct:.1f}%), " + f"{unique_inst} instances", + flush=True, + ) + else: + print( + f" [{completed}/{len(tiles_to_process)}] {filename}: FAILED - {message}", + flush=True, + ) + failures.append(f"{filename}: {message}") + else: + for i, args in enumerate(process_args): + filename, matched, total, unique_inst, success, message = _process_single_tile(args) + if success: + match_pct = (matched / total * 100) if total > 0 else 0 + print( + f" [{i+1}/{len(tiles_to_process)}] {matched:,}/{total:,} " + f"matched ({match_pct:.1f}%), {unique_inst} instances -> {filename}", + flush=True, + ) + else: + print(f" [{i+1}/{len(tiles_to_process)}] FAILED: {message} -> {filename}", flush=True) + failures.append(f"{filename}: {message}") + gc.collect() + + if failures: + raise RuntimeError("Retile to original files failed:\n " + "\n ".join(failures)) + + print(f"\n ✓ Retiling complete: {len(tiles_to_process)} tiles processed", flush=True) + gc.collect() + + +def _copy_record_dimensions(source_points, out_record) -> None: + """Copy dimensions from a source point record into an output record.""" + output_names = set(out_record.point_format.dimension_names) + output_names.update(dim.name for dim in out_record.point_format.extra_dimensions) + for dim_name in source_points.point_format.dimension_names: + if dim_name in output_names: + out_record[dim_name] = source_points[dim_name] + for dim in source_points.point_format.extra_dimensions: + if dim.name in output_names: + out_record[dim.name] = source_points[dim.name] + + +def _original_remap_output_header(source_header, merged_extra_dims, merged_extra_dim_params, threedtrees_dims, threedtrees_suffix): + """Build output header for original-with-predictions remap products.""" + new_header = copy_single_source_header( + source_header, + preserve_extra_dimensions=False, + ) + output_las = laspy.LasData(new_header) + output_standard_dim_names = set(output_las.point_format.dimension_names) + output_extra_dim_names = {dim.name for dim in output_las.point_format.extra_dimensions} + extra_dims_to_add = [] + added_extra_names = output_standard_dim_names | output_extra_dim_names + + for dim in source_header.point_format.extra_dimensions: + if dim.name not in added_extra_names: + extra_dims_to_add.append(extra_bytes_params_from_dimension_info(dim)) + added_extra_names.add(dim.name) + + selected_names = [ + name for name in merged_extra_dims + if not threedtrees_dims or name in threedtrees_dims + ] + branded_names = {} + for dim_name in selected_names: + desired_name = f"{dim_name}_{threedtrees_suffix}" if threedtrees_suffix else dim_name + out_name = ( + next_available_suffix(desired_name, added_extra_names) + if desired_name in added_extra_names + else desired_name + ) + branded_names[dim_name] = out_name + if out_name not in added_extra_names: + values = merged_extra_dims[dim_name] + if merged_extra_dim_params and dim_name in merged_extra_dim_params: + params = merged_extra_dim_params[dim_name] + extra_dims_to_add.append(extra_bytes_params_from_params(params, name=out_name)) + else: + extra_dims_to_add.append(laspy.ExtraBytesParams(name=out_name, type=values.dtype)) + added_extra_names.add(out_name) + + if extra_dims_to_add: + output_las.add_extra_dims(extra_dims_to_add) + return output_las.header, branded_names + + +def _original_remap_output_header_from_params( + source_header, + merged_extra_dim_params: Dict[str, laspy.ExtraBytesParams], + threedtrees_dims, + threedtrees_suffix, +): + """Build an original-with-predictions header from merged COPC header metadata.""" + new_header = copy_single_source_header( + source_header, + preserve_extra_dimensions=False, + ) + output_las = laspy.LasData(new_header) + output_standard_dim_names = set(output_las.point_format.dimension_names) + output_extra_dim_names = {dim.name for dim in output_las.point_format.extra_dimensions} + extra_dims_to_add = [] + added_extra_names = output_standard_dim_names | output_extra_dim_names + + for dim in source_header.point_format.extra_dimensions: + if dim.name not in added_extra_names: + extra_dims_to_add.append(extra_bytes_params_from_dimension_info(dim)) + added_extra_names.add(dim.name) + + selected_names = [ + name for name in merged_extra_dim_params + if not threedtrees_dims or name in threedtrees_dims + ] + branded_names = {} + for dim_name in selected_names: + desired_name = f"{dim_name}_{threedtrees_suffix}" if threedtrees_suffix else dim_name + out_name = ( + next_available_suffix(desired_name, added_extra_names) + if desired_name in added_extra_names + else desired_name + ) + branded_names[dim_name] = out_name + if out_name not in added_extra_names: + params = merged_extra_dim_params[dim_name] + extra_dims_to_add.append(extra_bytes_params_from_params(params, name=out_name)) + added_extra_names.add(out_name) + + if extra_dims_to_add: + output_las.add_extra_dims(extra_dims_to_add) + return output_las.header, branded_names + + +def _extra_dim_params_from_header(header) -> Dict[str, laspy.ExtraBytesParams]: + return { + dim.name: extra_bytes_params_from_dimension_info(dim) + for dim in header.point_format.extra_dimensions + } + + +def _chunk_xy_bounds(points: np.ndarray): + return ( + float(np.min(points[:, 0])), + float(np.max(points[:, 0])), + float(np.min(points[:, 1])), + float(np.max(points[:, 1])), + ) + + +def _existing_output_is_reusable( + input_file: Path, + output_file: Path, + expected_prediction_dims, +) -> bool: + """Return True when an existing enriched original matches this remap request.""" + if not output_file.exists() or output_file.stat().st_size == 0: + return False + try: + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as input_reader: + expected_count = int(input_reader.header.point_count) + with laspy.open(str(output_file), laz_backend=laspy.LazBackend.LazrsParallel) as output_reader: + if int(output_reader.header.point_count) != expected_count: + return False + output_dims = set(output_reader.header.point_format.dimension_names) + output_dims.update(dim.name for dim in output_reader.header.point_format.extra_dimensions) + return set(expected_prediction_dims).issubset(output_dims) + except Exception as exc: + print(f" Existing output is not reusable ({output_file.name}): {exc}", flush=True) + return False + + +def _queue_original_outputs( + original_files, + output_dir: Path, + expected_prediction_dims, +): + """Build remap work items, keeping only outputs valid for this request.""" + files_to_process = [] + skipped = 0 + stale = 0 + for input_file in original_files: + output_name = input_file.name.replace(".copc.laz", ".laz") + output_file = output_dir / output_name + if output_file.exists(): + expected_dims = ( + expected_prediction_dims(input_file) + if callable(expected_prediction_dims) + else expected_prediction_dims + ) + if _existing_output_is_reusable(input_file, output_file, expected_dims): + skipped += 1 + continue + stale += 1 + try: + output_file.unlink() + except OSError as exc: + raise RuntimeError(f"Could not replace stale output {output_file}: {exc}") from exc + files_to_process.append((input_file, output_file)) + return files_to_process, skipped, stale + + +def _expected_output_dims_from_loaded_merge( + input_file: Path, + merged_extra_dims, + merged_extra_dim_params, + threedtrees_dims, + threedtrees_suffix, +): + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as input_reader: + _header, branded_names = _original_remap_output_header( + input_reader.header, + merged_extra_dims, + merged_extra_dim_params, + threedtrees_dims, + threedtrees_suffix, + ) + return list(branded_names.values()) + + +def _expected_output_dims_from_merged_copc( + input_file: Path, + merged_extra_dim_params, + threedtrees_dims, + threedtrees_suffix, +): + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as input_reader: + _header, branded_names = _original_remap_output_header_from_params( + input_reader.header, + merged_extra_dim_params, + threedtrees_dims, + threedtrees_suffix, + ) + return list(branded_names.values()) + + +def _merged_copc_points_for_chunk(copc_reader, merged_header, xy_bounds, spatial_buffer, selected_dims): + query_bounds = laspy.copc.Bounds( + mins=np.array( + [ + xy_bounds[0] - spatial_buffer, + xy_bounds[2] - spatial_buffer, + float(merged_header.z_min), + ], + dtype=np.float64, + ), + maxs=np.array( + [ + xy_bounds[1] + spatial_buffer, + xy_bounds[3] + spatial_buffer, + float(merged_header.z_max), + ], + dtype=np.float64, + ), + ) + merged_record = copc_reader.spatial_query(query_bounds) + if len(merged_record) == 0: + return None, {} + + merged_points = np.column_stack([merged_record.x, merged_record.y, merged_record.z]) + merged_dims = { + dim_name: np.asarray(merged_record[dim_name]) + for dim_name in selected_dims + } + return merged_points, merged_dims + + +def _process_single_original_input_file_from_merged_copc(args): + """Stream one uploaded LAZ/LAS original and query merged COPC windows per chunk.""" + ( + input_file, + output_file, + merged_copc_file, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + threedtrees_dims, + threedtrees_suffix, + chunk_size, + min_match_fraction, + ) = args + + try: + selected_dims = [ + name for name in merged_extra_dim_params + if not threedtrees_dims or name in threedtrees_dims + ] + if not selected_dims: + return (input_file.name, 0, 0, 0, False, "No selected prediction dimensions in merged COPC") + + output_file.parent.mkdir(parents=True, exist_ok=True) + matched_count = 0 + total_points = 0 + unique_instances = 0 + + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as input_reader: + source_header = input_reader.header + n_input_points = int(source_header.point_count) + with laspy.CopcReader.open(str(merged_copc_file)) as copc_reader: + merged_header = copc_reader.header + output_header, branded_names = _original_remap_output_header_from_params( + source_header, + merged_extra_dim_params, + threedtrees_dims, + threedtrees_suffix, + ) + with laspy.open( + str(output_file), + mode="w", + header=output_header, + laz_backend=laspy.LazBackend.LazrsParallel, + ) as writer: + for input_chunk in input_reader.chunk_iterator(chunk_size): + input_points = np.column_stack([input_chunk.x, input_chunk.y, input_chunk.z]) + xy_bounds = _chunk_xy_bounds(input_points) + merged_points, merged_dims = _merged_copc_points_for_chunk( + copc_reader, + merged_header, + xy_bounds, + spatial_buffer, + selected_dims, + ) + if merged_points is None or len(merged_points) == 0: + raise ValueError( + f"No merged COPC points near chunk bounds {xy_bounds} " + f"from {input_file.name}" + ) + + local_tree = cKDTree(merged_points) + distances, indices = local_tree.query(input_points, workers=kdtree_workers) + matched = distances <= tolerance + window_matched = int(np.count_nonzero(matched)) + + out_chunk = laspy.ScaleAwarePointRecord.zeros(len(input_chunk), header=output_header) + _copy_record_dimensions(input_chunk, out_chunk) + for dim_name, values in merged_dims.items(): + remapped = values[indices] + out_chunk[branded_names[dim_name]] = remapped + if np.issubdtype(remapped.dtype, np.integer) and len(remapped) > 0: + unique_instances = max( + unique_instances, + len(np.unique(remapped[remapped > 0])), + ) + + writer.write_points(out_chunk) + matched_count += window_matched + total_points += len(input_chunk) + del input_points, merged_points, merged_dims, local_tree, distances, indices, out_chunk + + if total_points != n_input_points: + return ( + input_file.name, + matched_count, + n_input_points, + unique_instances, + False, + f"Wrote {total_points:,}/{n_input_points:,} original points", + ) + if not _match_fraction_is_acceptable(matched_count, n_input_points, min_match_fraction): + output_file.unlink(missing_ok=True) + return ( + input_file.name, + matched_count, + n_input_points, + unique_instances, + False, + _match_quality_message(matched_count, n_input_points, tolerance, min_match_fraction), + ) + return (input_file.name, matched_count, n_input_points, unique_instances, True, "Success") + except Exception as e: + try: + output_file.unlink(missing_ok=True) + except Exception: + pass + return (input_file.name, 0, 0, 0, False, str(e)) + + +def _copc_windows_from_header(header, num_windows: int): + min_x = float(header.x_min) + max_x = float(header.x_max) + if max_x <= min_x: + return [(min_x, max_x, True)] + count = max(1, int(num_windows or 1)) + step = (max_x - min_x) / count + return [ + ( + min_x + idx * step, + max_x if idx == count - 1 else min_x + (idx + 1) * step, + idx == count - 1, + ) + for idx in range(count) + ] + + +def _process_single_original_copc_file( + input_file: Path, + output_file: Path, + merged_points: np.ndarray, + merged_extra_dims: Dict[str, np.ndarray], + merged_extra_dim_params: Optional[Dict[str, laspy.ExtraBytesParams]], + tolerance: float, + spatial_buffer: float, + kdtree_workers: int, + threedtrees_dims, + threedtrees_suffix, + num_spatial_chunks: int = 1, + min_match_fraction: float = MIN_ORIGINAL_REMAP_MATCH_FRACTION, +): + """Process one COPC original with native spatial queries.""" + with laspy.CopcReader.open(str(input_file)) as copc_reader: + source_header = copc_reader.header + n_input_points = int(source_header.point_count) + output_header, branded_names = _original_remap_output_header( + source_header, + merged_extra_dims, + merged_extra_dim_params, + threedtrees_dims, + threedtrees_suffix, + ) + windows = _copc_windows_from_header(source_header, num_spatial_chunks) + + output_file.parent.mkdir(parents=True, exist_ok=True) + matched_count = 0 + total_points = 0 + unique_instances = 0 + with laspy.open( + str(output_file), + mode="w", + header=output_header, + laz_backend=laspy.LazBackend.LazrsParallel, + ) as writer: + for window_idx, (min_x, max_x, include_upper) in enumerate(windows, start=1): + query_bounds = laspy.copc.Bounds( + mins=np.array([min_x, float(source_header.y_min), float(source_header.z_min)], dtype=np.float64), + maxs=np.array([max_x, float(source_header.y_max), float(source_header.z_max)], dtype=np.float64), + ) + source_points = copc_reader.spatial_query(query_bounds) + if len(source_points) == 0: + continue + xs = np.asarray(source_points.x) + point_mask = (xs >= min_x) & ((xs <= max_x) if include_upper else (xs < max_x)) + if not np.any(point_mask): + continue + if not np.all(point_mask): + source_points = source_points[point_mask] + + input_points = np.column_stack([source_points.x, source_points.y, source_points.z]) + bounds = ( + float(np.min(input_points[:, 0])), + float(np.max(input_points[:, 0])), + float(np.min(input_points[:, 1])), + float(np.max(input_points[:, 1])), + ) + merge_mask = ( + (merged_points[:, 0] >= bounds[0] - spatial_buffer) + & (merged_points[:, 0] <= bounds[1] + spatial_buffer) + & (merged_points[:, 1] >= bounds[2] - spatial_buffer) + & (merged_points[:, 1] <= bounds[3] + spatial_buffer) + ) + local_merged_points = merged_points[merge_mask] + if len(local_merged_points) == 0: + raise ValueError(f"No merged points in COPC window {window_idx}/{len(windows)}") + local_tree = cKDTree(local_merged_points) + distances, indices = local_tree.query(input_points, workers=kdtree_workers) + matched = distances <= tolerance + window_matched = int(np.count_nonzero(matched)) + out_chunk = laspy.ScaleAwarePointRecord.zeros(len(source_points), header=output_header) + _copy_record_dimensions(source_points, out_chunk) + + for dim_name, values in merged_extra_dims.items(): + if threedtrees_dims and dim_name not in threedtrees_dims: + continue + remapped = values[merge_mask][indices] + out_chunk[branded_names[dim_name]] = remapped + if np.issubdtype(remapped.dtype, np.integer) and len(remapped) > 0: + unique_instances = max(unique_instances, len(np.unique(remapped[remapped > 0]))) + + writer.write_points(out_chunk) + matched_count += window_matched + total_points += len(source_points) + print( + f" COPC original remap {input_file.name}: " + f"window {window_idx}/{len(windows)}, {matched_count:,} points", + flush=True, + ) + + if total_points != n_input_points: + output_file.unlink(missing_ok=True) + return ( + input_file.name, + matched_count, + n_input_points, + unique_instances, + False, + f"Wrote {total_points:,}/{n_input_points:,} original points", + ) + if not _match_fraction_is_acceptable(matched_count, n_input_points, min_match_fraction): + output_file.unlink(missing_ok=True) + return ( + input_file.name, + matched_count, + n_input_points, + unique_instances, + False, + _match_quality_message(matched_count, n_input_points, tolerance, min_match_fraction), + ) + return (input_file.name, matched_count, n_input_points, unique_instances, True, "Success") + + +def _process_single_original_input_file(args): + """Process one original input file for final prediction remapping.""" + ( + input_file, + output_file, + merged_points, + merged_extra_dims, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + threedtrees_dims, + threedtrees_suffix, + num_spatial_chunks, + min_match_fraction, + ) = args + + try: + if input_file.name.lower().endswith(".copc.laz"): + return _process_single_original_copc_file( + input_file, + output_file, + merged_points, + merged_extra_dims, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + threedtrees_dims, + threedtrees_suffix, + num_spatial_chunks=num_spatial_chunks, + min_match_fraction=min_match_fraction, + ) + + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as f: + bounds = (f.header.x_min, f.header.x_max, f.header.y_min, f.header.y_max) + n_input_points = f.header.point_count + + mask = ( + (merged_points[:, 0] >= bounds[0] - spatial_buffer) + & (merged_points[:, 0] <= bounds[1] + spatial_buffer) + & (merged_points[:, 1] >= bounds[2] - spatial_buffer) + & (merged_points[:, 1] <= bounds[3] + spatial_buffer) + ) + + local_merged_points = merged_points[mask] + local_merged_extras = {name: arr[mask] for name, arr in merged_extra_dims.items()} + + if len(local_merged_points) == 0: + return (input_file.name, 0, n_input_points, 0, False, "No merged points in file region") + + local_tree = cKDTree(local_merged_points) + input_las = laspy.read(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) + input_points = np.empty((n_input_points, 3), dtype=np.float64) + input_points[:, 0] = input_las.x + input_points[:, 1] = input_las.y + input_points[:, 2] = input_las.z + + distances, indices = local_tree.query(input_points, workers=kdtree_workers) + matched = distances <= tolerance + matched_count = int(np.count_nonzero(matched)) + if not _match_fraction_is_acceptable(matched_count, n_input_points, min_match_fraction): + return ( + input_file.name, + matched_count, + n_input_points, + 0, + False, + _match_quality_message(matched_count, n_input_points, tolerance, min_match_fraction), + ) + + if threedtrees_dims: + filtered_extras = { + name: arr[indices] + for name, arr in local_merged_extras.items() + if name in threedtrees_dims + } + else: + filtered_extras = {name: arr[indices] for name, arr in local_merged_extras.items()} + + unique_instances = 0 + for arr in filtered_extras.values(): + if np.issubdtype(arr.dtype, np.integer) and len(arr) > 0: + unique_instances = max(unique_instances, len(np.unique(arr[arr > 0]))) + break + + branded_names = {} + for dim_name in filtered_extras: + if threedtrees_suffix: + branded_names[dim_name] = f"{dim_name}_{threedtrees_suffix}" + else: + branded_names[dim_name] = dim_name + + new_header = copy_single_source_header( + input_las.header, + preserve_extra_dimensions=False, + ) + output_las = laspy.LasData(new_header) + output_standard_dim_names = set(output_las.point_format.dimension_names) + output_extra_dim_names = {dim.name for dim in output_las.point_format.extra_dimensions} + + extra_dims_to_add = [] + added_extra_names = output_standard_dim_names | output_extra_dim_names + for dim in input_las.point_format.extra_dimensions: + if dim.name not in added_extra_names: + extra_dims_to_add.append(extra_bytes_params_from_dimension_info(dim)) + added_extra_names.add(dim.name) + + for dim_name in input_las.point_format.dimension_names: + if dim_name not in added_extra_names: + arr = getattr(input_las, dim_name, None) + dtype = arr.dtype if arr is not None else np.int32 + extra_dims_to_add.append(laspy.ExtraBytesParams(name=dim_name, type=dtype)) + added_extra_names.add(dim_name) + + for dim_name, values in filtered_extras.items(): + desired_name = branded_names[dim_name] + out_name = ( + next_available_suffix(desired_name, added_extra_names) + if desired_name in added_extra_names + else desired_name + ) + branded_names[dim_name] = out_name + if out_name not in added_extra_names: + if merged_extra_dim_params and dim_name in merged_extra_dim_params: + params = merged_extra_dim_params[dim_name] + extra_dims_to_add.append(extra_bytes_params_from_params(params, name=out_name)) + else: + extra_dims_to_add.append(laspy.ExtraBytesParams(name=out_name, type=values.dtype)) + added_extra_names.add(out_name) + + if extra_dims_to_add: + output_las.add_extra_dims(extra_dims_to_add) + + for dim_name in output_las.point_format.dimension_names: + try: + if hasattr(input_las, dim_name): + setattr(output_las, dim_name, getattr(input_las, dim_name)) + except Exception: + pass + for dim in input_las.point_format.extra_dimensions: + if hasattr(input_las, dim.name): + try: + setattr(output_las, dim.name, getattr(input_las, dim.name)) + except Exception: + pass + + for dim_name, values in filtered_extras.items(): + setattr(output_las, branded_names[dim_name], values) + + output_file.parent.mkdir(parents=True, exist_ok=True) + output_las.write( + str(output_file), + do_compress=True, + laz_backend=laspy.LazBackend.LazrsParallel, + ) + + del input_las + del output_las + return (input_file.name, matched_count, n_input_points, unique_instances, True, "Success") + + except Exception as e: + try: + output_file.unlink(missing_ok=True) + except Exception: + pass + return (input_file.name, 0, 0, 0, False, str(e)) + + +def validate_common_dimensions_minmax(original_path: Path, output_path: Path, rel_tol: float = 1e-5) -> None: + """Warn when common non-coordinate dimensions changed range after remap.""" + try: + orig = laspy.read(str(original_path), laz_backend=laspy.LazBackend.LazrsParallel) + out = laspy.read(str(output_path), laz_backend=laspy.LazBackend.LazrsParallel) + except Exception as e: + print(f" Validation skip: could not read files ({e})", flush=True) + return + try: + orig_names = set(orig.point_format.dimension_names) | {d.name for d in orig.point_format.extra_dimensions} + out_names = set(out.point_format.dimension_names) | {d.name for d in out.point_format.extra_dimensions} + common = orig_names & out_names - {"X", "Y", "Z"} + if not common: + return + diffs = [] + for name in sorted(common): + oa = getattr(orig, name, None) + oo = getattr(out, name, None) + if oa is None or oo is None or len(oa) != len(oo): + continue + oa, oo = np.asarray(oa), np.asarray(oo) + omin, omax = float(np.min(oa)), float(np.max(oa)) + wmin, wmax = float(np.min(oo)), float(np.max(oo)) + if np.issubdtype(oa.dtype, np.integer) and np.issubdtype(oo.dtype, np.integer): + if (omin != wmin or omax != wmax) and (int(omin) != int(wmin) or int(omax) != int(wmax)): + diffs.append((name, omin, omax, wmin, wmax)) + else: + span = max(omax - omin, 1e-12) + if abs(omin - wmin) > rel_tol * span or abs(omax - wmax) > rel_tol * span: + diffs.append((name, omin, omax, wmin, wmax)) + if diffs: + print(" Warning: common dimensions min/max differ (output may have rounding/loss):", flush=True) + for name, omin, omax, wmin, wmax in diffs: + print(f" {name}: original [{omin}, {omax}] vs output [{wmin}, {wmax}]", flush=True) + print(" Tip: dimensions above were not overwritten by merged where range would be lost.", flush=True) + del orig + del out + except Exception as e: + print(f" Validation skip: {e}", flush=True) + + +def remap_to_original_input_files( + merged_points: np.ndarray, + merged_extra_dims: Dict[str, np.ndarray], + merged_extra_dim_params: Optional[Dict[str, laspy.ExtraBytesParams]], + original_input_dir: Path, + output_dir: Path, + tolerance: float = 0.1, + num_threads: int = 8, + retile_buffer: float = 2.0, + threedtrees_dims: Optional[List[str]] = None, + threedtrees_suffix: str = "SAT", + num_spatial_chunks: int = 1, + prefer_copc_sources: bool = True, + min_match_fraction: float = MIN_ORIGINAL_REMAP_MATCH_FRACTION, +): + """Transfer selected merged prediction dimensions back to original input files.""" + print(f"\n{'=' * 60}", flush=True) + print("Remapping to original input files", flush=True) + print(f"{'=' * 60}", flush=True) + + original_files = ( + point_cloud_files(original_input_dir) + if prefer_copc_sources + else raw_point_cloud_files(original_input_dir) + ) + if len(original_files) == 0: + print(f" No LAZ/LAS files found in {original_input_dir}", flush=True) + return + if not prefer_copc_sources: + print(" Raw-original mode: ignoring COPC twins in original input dir", flush=True) + + print(f" Found {len(original_files)} original input files", flush=True) + print(f" Output: {output_dir}", flush=True) + + output_dir.mkdir(parents=True, exist_ok=True) + spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer + + if threedtrees_dims is None: + threedtrees_dims = ["PredInstance", "PredSemantic"] + threedtrees_dims_set = set(threedtrees_dims) + + available_3dt = sorted(threedtrees_dims_set & set(merged_extra_dims.keys())) + expected_output_dims = [ + f"{d}_{threedtrees_suffix}" if threedtrees_suffix else d + for d in available_3dt + ] + if available_3dt: + print(f" 3DTrees dimensions to transfer: {', '.join(available_3dt)} -> {', '.join(expected_output_dims)}", flush=True) + else: + print( + " Warning: No 3DTrees dimensions found in merged file " + f"(looked for: {', '.join(sorted(threedtrees_dims_set))})", + flush=True, + ) + + files_to_process, skipped, stale = _queue_original_outputs( + original_files, + output_dir, + lambda input_file: _expected_output_dims_from_loaded_merge( + input_file, + merged_extra_dims, + merged_extra_dim_params, + threedtrees_dims_set, + threedtrees_suffix, + ), + ) + if skipped > 0: + print(f" Skipping {skipped} already processed files", flush=True) + if stale > 0: + print(f" Reprocessing {stale} stale existing output file(s)", flush=True) + + if len(files_to_process) == 0: + print(" All files already processed!", flush=True) + return + + print(f" Processing {len(files_to_process)} files...", flush=True) + + kdtree_workers = -1 + process_args = [ + ( + input_file, + output_file, + merged_points, + merged_extra_dims, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + threedtrees_dims_set, + threedtrees_suffix, + num_spatial_chunks, + min_match_fraction, + ) + for input_file, output_file in files_to_process + ] + + total_matched = 0 + total_points = 0 + failures = [] + parallel_workers = min(num_threads, len(files_to_process)) if num_threads > 1 else 1 + if parallel_workers > 1: + print(f" Processing with {parallel_workers} parallel workers...", flush=True) + with ThreadPoolExecutor(max_workers=parallel_workers) as executor: + results = executor.map(_process_single_original_input_file, process_args) + for i, result in enumerate(results): + filename, matched, total, unique_inst, success, message = result + if success: + match_pct = (matched / total * 100) if total > 0 else 0 + print( + f" [{i+1}/{len(files_to_process)}] {matched:,}/{total:,} " + f"matched ({match_pct:.1f}%), {unique_inst} instances -> {filename}", + flush=True, + ) + total_matched += matched + total_points += total + else: + print(f" [{i+1}/{len(files_to_process)}] FAILED: {message} -> {filename}", flush=True) + failures.append(f"{filename}: {message}") + else: + for i, args in enumerate(process_args): + filename, matched, total, unique_inst, success, message = _process_single_original_input_file(args) + if success: + match_pct = (matched / total * 100) if total > 0 else 0 + print( + f" [{i+1}/{len(files_to_process)}] {matched:,}/{total:,} " + f"matched ({match_pct:.1f}%), {unique_inst} instances -> {filename}", + flush=True, + ) + total_matched += matched + total_points += total + else: + print(f" [{i+1}/{len(files_to_process)}] FAILED: {message} -> {filename}", flush=True) + failures.append(f"{filename}: {message}") + gc.collect() + + if failures: + raise RuntimeError("Original remap failed:\n " + "\n ".join(failures)) + + overall_match_pct = (total_matched / total_points * 100) if total_points > 0 else 0 + print( + f"\n ✓ Remap complete: {len(files_to_process)} files, " + f"{total_matched:,}/{total_points:,} matched ({overall_match_pct:.1f}%)", + flush=True, + ) + + if files_to_process and total_matched > 0: + first_input, first_output = files_to_process[0] + if first_output.exists(): + validate_common_dimensions_minmax(first_input, first_output) + + gc.collect() + + +def remap_merged_file_to_original_input_files( + merged_file: Path, + original_input_dir: Path, + output_dir: Path, + tolerance: float = 0.1, + num_threads: int = 8, + retile_buffer: float = 2.0, + threedtrees_dims: Optional[List[str]] = None, + threedtrees_suffix: str = "SAT", + num_spatial_chunks: int = 1, + chunk_size: int = 5_000_000, + prefer_copc_sources: bool = True, + min_match_fraction: float = MIN_ORIGINAL_REMAP_MATCH_FRACTION, +): + """Transfer merged prediction dimensions back to originals. + + COPC merged sources use a streaming raw-original path: each uploaded + original LAZ/LAS chunk queries the merged COPC by chunk XY bounds and builds + a small local KDTree. Non-COPC merged sources fall back to the legacy loaded + merged-cloud path. + """ + merged_file = Path(merged_file) + if not merged_file.name.lower().endswith(".copc.laz"): + from merge_loaded_cloud import load_merged_file + + merged_points, merged_extra_dims, merged_extra_dim_params = load_merged_file(merged_file) + return remap_to_original_input_files( + merged_points, + merged_extra_dims, + merged_extra_dim_params, + original_input_dir, + output_dir, + tolerance=tolerance, + num_threads=num_threads, + retile_buffer=retile_buffer, + threedtrees_dims=threedtrees_dims, + threedtrees_suffix=threedtrees_suffix, + num_spatial_chunks=num_spatial_chunks, + prefer_copc_sources=prefer_copc_sources, + min_match_fraction=min_match_fraction, + ) + + print(f"\n{'=' * 60}", flush=True) + print("Remapping merged COPC to original input files", flush=True) + print(f"{'=' * 60}", flush=True) + original_files = ( + point_cloud_files(original_input_dir) + if prefer_copc_sources + else raw_point_cloud_files(original_input_dir) + ) + if len(original_files) == 0: + print(f" No LAZ/LAS files found in {original_input_dir}", flush=True) + return + if not prefer_copc_sources: + print(" Raw-original mode: ignoring COPC twins in original input dir", flush=True) + + with laspy.open(str(merged_file), laz_backend=laspy.LazBackend.LazrsParallel) as merged_reader: + merged_extra_dim_params = _extra_dim_params_from_header(merged_reader.header) + + if threedtrees_dims is None: + threedtrees_dims = ["PredInstance", "PredSemantic"] + threedtrees_dims_set = set(threedtrees_dims) + available_3dt = sorted(threedtrees_dims_set & set(merged_extra_dim_params.keys())) + if available_3dt: + branded = [f"{d}_{threedtrees_suffix}" if threedtrees_suffix else d for d in available_3dt] + print(f" 3DTrees dimensions to transfer: {', '.join(available_3dt)} -> {', '.join(branded)}", flush=True) + else: + print( + " Warning: No 3DTrees dimensions found in merged COPC " + f"(looked for: {', '.join(sorted(threedtrees_dims_set))})", + flush=True, + ) + + print(f" Merged COPC source: {merged_file}", flush=True) + print(f" Original input files: {len(original_files)}", flush=True) + print(f" Output: {output_dir}", flush=True) + print(f" Original chunk size: {chunk_size:,} points", flush=True) + print(" Per-chunk strategy: original LAZ chunk -> merged COPC spatial query -> local KDTree", flush=True) + + output_dir.mkdir(parents=True, exist_ok=True) + spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer + expected_output_dims = [ + f"{d}_{threedtrees_suffix}" if threedtrees_suffix else d + for d in available_3dt + ] + files_to_process, skipped, stale = _queue_original_outputs( + original_files, + output_dir, + lambda input_file: _expected_output_dims_from_merged_copc( + input_file, + merged_extra_dim_params, + threedtrees_dims_set, + threedtrees_suffix, + ), + ) + + if skipped > 0: + print(f" Skipping {skipped} already processed files", flush=True) + if stale > 0: + print(f" Reprocessing {stale} stale existing output file(s)", flush=True) + if not files_to_process: + print(" All files already processed!", flush=True) + return + + parallel_workers = min(max(1, num_threads), len(files_to_process)) + kdtree_workers = -1 if parallel_workers == 1 else max(1, num_threads // parallel_workers) + process_args = [ + ( + input_file, + output_file, + merged_file, + merged_extra_dim_params, + tolerance, + spatial_buffer, + kdtree_workers, + threedtrees_dims_set, + threedtrees_suffix, + chunk_size, + min_match_fraction, + ) + for input_file, output_file in files_to_process + ] + + total_matched = 0 + total_points = 0 + failures = [] + if parallel_workers > 1: + print(f" Processing with {parallel_workers} parallel original-file workers...", flush=True) + with ThreadPoolExecutor(max_workers=parallel_workers) as executor: + results = list(executor.map(_process_single_original_input_file_from_merged_copc, process_args)) + else: + results = [_process_single_original_input_file_from_merged_copc(args) for args in process_args] + + for i, (filename, matched, total, unique_inst, success, message) in enumerate(results): + if success: + match_pct = (matched / total * 100) if total > 0 else 0 + print( + f" [{i+1}/{len(files_to_process)}] {matched:,}/{total:,} " + f"matched ({match_pct:.1f}%), {unique_inst} instances -> {filename}", + flush=True, + ) + total_matched += matched + total_points += total + else: + print(f" [{i+1}/{len(files_to_process)}] FAILED: {message} -> {filename}", flush=True) + failures.append(f"{filename}: {message}") + + if failures: + raise RuntimeError("Merged COPC remap failed:\n " + "\n ".join(failures)) + + overall_match_pct = (total_matched / total_points * 100) if total_points > 0 else 0 + print( + f"\n ✓ Merged COPC remap complete: {len(files_to_process)} files, " + f"{total_matched:,}/{total_points:,} matched ({overall_match_pct:.1f}%)", + flush=True, + ) + + if files_to_process and total_matched > 0: + first_input, first_output = files_to_process[0] + if first_output.exists(): + validate_common_dimensions_minmax(first_input, first_output) + + gc.collect() diff --git a/src/parameters.py b/src/parameters.py index 4f44e97..7aa96f0 100644 --- a/src/parameters.py +++ b/src/parameters.py @@ -1,218 +1,610 @@ """ -Centralized parameter configuration for the 3DTrees smart tile pipeline. +Centralized parameter configuration for the 3DTrees smart tiling pipeline. -The public CLI exposes the active task entrypoints: -- ``tile``: COPC-first tiling plus two-stage subsampling -- ``filter``: buffer-zone filtering of segmented tiles, optionally followed by a remap tail -- ``remap``: remap segmented collection dimensions back to original files +Uses Pydantic BaseSettings for CLI argument parsing, environment variable support, +and parameter validation. + +Usage: + python run.py --task tile --input-dir /path/to/input --output-dir /path/to/output + python run.py --task merge --subsampled-10cm-folder /path/to/10cm --original-input-dir /path/to/input """ +from pydantic_settings import BaseSettings, SettingsConfigDict +from pydantic import Field, AliasChoices, field_validator from pathlib import Path -from typing import Literal, Optional +from collections.abc import Iterable +from typing import Optional -from pydantic import AliasChoices, Field, field_validator -from pydantic_settings import BaseSettings, SettingsConfigDict +class Parameters(BaseSettings): + """ + Pipeline parameters with CLI and environment variable support. -TaskName = Literal["tile", "filter", "remap"] -FilterAnchor = Literal["centroid", "highest_point", "lowest_point"] + All parameters can be passed via: + - CLI arguments: --param-name value + - Environment variables: PARAM_NAME=value + """ + # ========================================================================== + # Common parameters + # ========================================================================== -class Parameters(BaseSettings): - """Pipeline parameters with CLI and environment variable support.""" + task: str = Field( + "tile", + description=( + "Task to perform: 'tile' (tiling + subsampling), 'merge' (remap + merge), " + "'filter' (remove duplicate buffer-zone instances), 'remap' " + "(prediction source -> original files), or 'create_merged_file' " + "(prod-merged from original_with_predictions)" + ), + ) - # Shared - task: Optional[TaskName] = Field( + input_dir: Optional[Path] = Field( default=None, - description="Task to perform: 'tile', 'filter', or 'remap'.", + description="Input directory with LAZ/LAS files (required for 'tile' task)", + validation_alias=AliasChoices("input-dir", "input_dir"), ) + output_dir: Optional[Path] = Field( default=None, - description="Output directory for the selected task.", + description="Output directory (required for 'tile' task)", validation_alias=AliasChoices("output-dir", "output_dir"), ) + workers: int = Field( - default=4, - description="Number of parallel workers.", + 4, + description="Number of parallel workers for processing", validation_alias=AliasChoices("workers", "number-of-threads", "number_of_threads"), ) - chunk_size: int = Field( - default=20_000_000, - description="Points per streaming chunk for filtering, remap, tile COPC preparation, and tile subsampling fallback.", - validation_alias=AliasChoices("chunk-size", "chunk_size"), - ) - # Tile - input_dir: Optional[Path] = Field( - default=None, - description="Input directory with LAZ/LAS files for the tile task.", - validation_alias=AliasChoices("input-dir", "input_dir"), - ) - tile_length: int = Field( - default=100, - description="Tile size in meters for the tile task.", + # ========================================================================== + # Tile task parameters + # ========================================================================== + + tile_length: Optional[int] = Field( + 100, + description="Tile size in meters (only for 'tile' task)", validation_alias=AliasChoices("tile-length", "tile_length"), ) - tile_buffer: int = Field( - default=20, - description="Tile overlap buffer in meters for the tile task.", + + tile_buffer: Optional[int] = Field( + 20, + description="Buffer overlap in meters (only for 'tile' task)", validation_alias=AliasChoices("tile-buffer", "tile_buffer"), ) - threads: int = Field( - default=10, - description="Threads per tile/subsampling worker for the tile task.", - validation_alias=AliasChoices("threads"), + + threads: Optional[int] = Field( + 10, + description="Threads per COPC writer (only for 'tile' task)", ) - resolution_1: float = Field( - default=0.01, - description="First subsampling resolution in meters for the tile task.", + + resolution_1: Optional[float] = Field( + 0.01, + description="First subsampling resolution in meters (1cm) (only for 'tile' task)", validation_alias=AliasChoices("resolution-1", "resolution_1"), ) - resolution_2: float = Field( - default=0.1, - description="Second subsampling resolution in meters for the tile task.", + + resolution_2: Optional[float] = Field( + 0.1, + description="Second subsampling resolution in meters (10cm) (only for 'tile' task)", validation_alias=AliasChoices("resolution-2", "resolution_2"), ) + output_copc_res1: bool = Field( - default=True, - description="Write res1 subsampling outputs as COPC ('.copc.laz') instead of standard LAZ ('.laz').", + True, + description="Write first-resolution subsampled outputs as COPC LAZ (default: True for 1cm products)", validation_alias=AliasChoices("output-copc-res1", "output_copc_res1"), ) + output_copc_res2: bool = Field( - default=False, - description="Write res2 subsampling outputs as COPC ('.copc.laz') instead of standard LAZ ('.laz').", + False, + description="Write second-resolution subsampled outputs as COPC LAZ (default: False, regular LAZ for 10cm products)", validation_alias=AliasChoices("output-copc-res2", "output_copc_res2"), ) - dimension_reduction: bool = Field( - default=True, - description="Reduce subsampled outputs to minimal LAS dimensions for smaller files.", - validation_alias=AliasChoices("dimension-reduction", "dimension_reduction"), + + + skip_dimension_reduction: bool = Field( + False, + description=( + "Keep extra point dimensions in LAZ intermediates. Intermediate COPC " + "conversion still strips extra dimensions by default; prod-merged " + "creation preserves enriched dimensions." + ), + validation_alias=AliasChoices("skip-dimension-reduction", "skip_dimension_reduction"), + ) + + instance_dimension: str = Field( + "PredInstance", + description="Name of the instance ID dimension in input files (default: PredInstance, fallback: treeID)", + validation_alias=AliasChoices("instance-dimension", "instance_dimension"), ) - skip_dimension_reduction: Optional[bool] = Field( + + filter_suffix: str = Field( + "_filtered", + description="Suffix added to output filenames for the filter task", + validation_alias=AliasChoices("filter-suffix", "filter_suffix", "suffix"), + ) + + filter_output_extension: Optional[str] = Field( default=None, - description="Deprecated inverse alias for dimension reduction. When set, overrides dimension_reduction as not skip_dimension_reduction.", - validation_alias=AliasChoices("skip-dimension-reduction", "skip_dimension_reduction"), + description="Optional output extension override for the filter task, e.g. .laz", + validation_alias=AliasChoices("filter-output-extension", "filter_output_extension", "output-extension", "output_extension"), ) + num_spatial_chunks: Optional[int] = Field( default=None, - description="Number of spatial chunks per tile during subsampling. Defaults to the worker count when omitted.", + description="Subsampling parallelism per file: COPC COM window workers or stripe chunks (default: equals workers)", validation_alias=AliasChoices("num-spatial-chunks", "num_spatial_chunks"), ) + + subsampling_method: str = Field( + default="center-of-mass", + description="Subsampling method: center-of-mass (default) or nearest-to-centroid", + validation_alias=AliasChoices("subsampling-method", "subsampling_method"), + ) + tiling_threshold: Optional[float] = Field( default=None, - description="If the tile input folder contains a single file below this size in MB, skip tile generation after COPC normalization.", + description="File size threshold in MB. If input folder has single file below this size, skip tiling (only for 'tile' task)", validation_alias=AliasChoices("tiling-threshold", "tiling_threshold"), ) - chunkwise_copc_source_creation: bool = Field( - default=False, - description="Build source COPC files through temporary chunked LAS parts to reduce peak RAM during tile normalization.", - validation_alias=AliasChoices("chunkwise-copc-source-creation", "chunkwise_copc_source_creation"), + + chunk_size: Optional[int] = Field( + default=20_000_000, + description="Points per chunk when reading LAZ/LAS in tiling Phase 1 or multi-collection remap (smaller = less peak RAM, more overhead)", + validation_alias=AliasChoices("chunk-size", "chunk_size"), ) - # Filter - segmented_folders: str = Field( - default="", - description="Comma-separated list of segmented tile files and/or folders.", - validation_alias=AliasChoices("segmented-folders", "segmented_folders"), + # ========================================================================== + # Merge task parameters + # ========================================================================== + + subsampled_10cm_folder: Optional[Path] = Field( + default=None, + description="Path to subsampled 10cm folder with segmented results (for 'merge' task)", + validation_alias=AliasChoices("subsampled-10cm-folder", "subsampled_10cm_folder", "subsampled-segmented-folder"), ) - tile_bounds_json: Optional[Path] = Field( + + subsampled_target_folder: Optional[Path] = Field( default=None, - description="Path to tile_bounds_tindex.json used for filter neighbor/border logic.", - validation_alias=AliasChoices("tile-bounds-json", "tile_bounds_json"), + description="Path to target resolution subsampled folder (auto-derived if not specified)", + validation_alias=AliasChoices("subsampled-target-folder", "subsampled_target_folder"), ) - instance_dimension: str = Field( - default="PredInstance", - description="Name of the instance ID dimension in segmented input files.", - validation_alias=AliasChoices("instance-dimension", "instance_dimension"), + + segmented_remapped_folder: Optional[Path] = Field( + default=None, + description="Path to segmented remapped folder (for 'merge' task, skip remap step)", + validation_alias=AliasChoices("segmented-remapped-folder", "segmented_remapped_folder"), ) - border_zone_width: Optional[float] = Field( + + original_tiles_dir: Optional[Path] = Field( default=None, - description="Optional explicit border-zone width in meters; derived from tile_bounds_json when omitted.", - validation_alias=AliasChoices("border-zone-width", "border_zone_width"), + description="Directory with original tile files for retiling (for 'merge' task)", + validation_alias=AliasChoices("original-tiles-dir", "original_tiles_dir"), ) - filter_anchor: FilterAnchor = Field( - default="centroid", - description="Representative point used to classify an instance as border-adjacent.", - validation_alias=AliasChoices("filter-anchor", "filter_anchor"), + + tile_bounds_json: Optional[Path] = Field( + default=None, + description="Path to tile_bounds_tindex.json for neighbor graph and remap matching (merge task). If set, used instead of auto-derived paths.", + validation_alias=AliasChoices("tile-bounds-json", "tile_bounds_json"), ) - min_cluster_size: int = Field( - default=300, - description="Minimum cluster size in points for reassignment.", - validation_alias=AliasChoices("min-cluster-size", "min_cluster_size"), + + original_input_dir: Optional[Path] = Field( + default=None, + description=( + "Legacy directory with original input LAZ/LAS files for final remap. " + "Prefer --original-laz-input-dir for the explicit production workflow. " + "COPC-only original enrichment is no longer supported." + ), + validation_alias=AliasChoices("original-input-dir", "original_input_dir"), ) - max_volume_for_merge: float = Field( - default=5.0, - description="Maximum 3D convex hull volume for small-instance reassignment in m^3.", - validation_alias=AliasChoices("max-volume-for-merge", "max_volume_for_merge"), + + original_copc_input_dir: Optional[Path] = Field( + default=None, + description=( + "Optional directory with original COPC LAZ files matching the uploaded " + "LAZ/LAS originals. Used for source-pair validation and source-matching " + "workflows, but remap enriches the uploaded LAZ/LAS files directly." + ), + validation_alias=AliasChoices("original-copc-input-dir", "original_copc_input_dir"), ) - enable_volume_merge: bool = Field( - default=True, - description="Enable reassignment of very small kept instances.", - validation_alias=AliasChoices("enable-volume-merge", "enable_volume_merge"), + + original_raw_input_dir: Optional[Path] = Field( + default=None, + description=( + "Directory with uploaded LAZ/LAS originals to enrich. This is the " + "production Original-with-predictions writer/metadata lane and ignores " + "COPC twins so raw source metadata/VLRs remain the metadata source." + ), + validation_alias=AliasChoices( + "original-laz-input-dir", + "original_laz_input_dir", + "original-raw-input-dir", + "original_raw_input_dir", + "original-download-input-dir", + "original_download_input_dir", + ), ) - remap_merge: bool = Field( - default=False, - description="After filtering, run the remap task on the filtered collection.", - validation_alias=AliasChoices("remap-merge", "remap_merge"), + + original_raw_output_dir: Optional[Path] = Field( + default=None, + description=( + "Output directory for enriched uploaded LAZ/LAS originals. Defaults to " + "--output-dir when set, otherwise original_with_predictions next to the " + "raw input directory." + ), + validation_alias=AliasChoices( + "original-laz-output-dir", + "original_laz_output_dir", + "original-raw-output-dir", + "original_raw_output_dir", + "original-download-output-dir", + "original_download_output_dir", + ), ) - # Remap / filter-remap shared - original_input_dir: Optional[Path] = Field( + output_merged_laz: Optional[Path] = Field( default=None, - description="Directory with original input LAZ/LAS files for remap outputs.", - validation_alias=AliasChoices("original-input-dir", "original_input_dir"), + description="Output path for merged LAZ file (auto-derived if not specified)", + validation_alias=AliasChoices("output-merged-laz", "output_merged_laz"), ) - subsampled_target_folder: Optional[Path] = Field( + + output_tiles_folder: Optional[Path] = Field( default=None, - description="Optional target-resolution collection to receive remapped dimensions.", - validation_alias=AliasChoices("subsampled-target-folder", "subsampled_target_folder"), + description="Output folder for per-tile results (auto-derived if not specified)", + validation_alias=AliasChoices("output-tiles-folder", "output_tiles_folder"), ) - produce_merged_file: bool = Field( - default=True, - description="Also write a single merged output file.", - validation_alias=AliasChoices("produce-merged-file", "produce_merged_file"), + + output_folder: Optional[Path] = Field( + default=None, + description="Output folder for remapped files (auto-derived if not specified)", + validation_alias=AliasChoices("output-folder", "output_folder"), ) - transfer_original_dims_to_merged: bool = Field( - default=True, - description="When producing a merged file, enrich it with original-file attributes.", - validation_alias=AliasChoices("transfer-original-dims-to-merged", "transfer_original_dims_to_merged"), + + original_with_predictions_dir: Optional[Path] = Field( + default=None, + description="Directory with Original-with-predictions files for create_merged_file task", + validation_alias=AliasChoices("original-with-predictions-dir", "original_with_predictions_dir"), ) - output_merged_with_originals: Optional[Path] = Field( + + staged_copc_dir: Optional[Path] = Field( default=None, - description="Optional explicit path for merged output enriched with original attributes.", - validation_alias=AliasChoices("output-merged-with-originals", "output_merged_with_originals"), + description=( + "Optional directory with already converted Original-with-predictions COPC files " + "for create_merged_file/prod-merged outputs. Matching COPCs are reused after " + "a readable-header check." + ), + validation_alias=AliasChoices("staged-copc-dir", "staged_copc_dir"), ) + standardization_json: Optional[Path] = Field( default=None, - description="Optional collection_summary.json to restrict which original attributes are transferred into merged outputs. Source prediction/remap dimensions in the merged file remain controlled independently by --remap-dims.", + description=( + "Optional tool_standard collection_summary.json. When provided, SmartTile " + "validates that Original-with-predictions COPCs and LAS/COPC prod-merged " + "outputs still expose the expected standardized source dimensions." + ), validation_alias=AliasChoices("standardization-json", "standardization_json"), ) + + merged_resolutions: str = Field( + "res1,res2", + description=( + "Comma-separated prod-merged output resolutions for create_merged_file. " + "Use res1/res2, numeric meters, or centimeter labels such as 1cm,10cm." + ), + validation_alias=AliasChoices("merged-resolutions", "merged_resolutions"), + ) + + merged_output_formats: str = Field( + "copc.laz", + description="Comma-separated prod-merged output formats: laz, copc.laz, or ply.", + validation_alias=AliasChoices("merged-output-formats", "merged_output_formats"), + ) + + # ========================================================================== + # Remap-to-originals task parameters + # ========================================================================== + + merged_laz: Optional[Path] = Field( + default=None, + description="Path to merged LAZ/COPC LAZ file (for 'remap' task). Selected dimensions from this file are added to original files.", + validation_alias=AliasChoices("merged-laz", "merged_laz"), + ) + + segmented_folders: Optional[str] = Field( + default=None, + description=( + "Comma-separated list of finalized prediction collection folders/files " + "for multi-collection 'remap'. Extra dimensions are copied as-is and " + "duplicate prediction dimension names fail." + ), + validation_alias=AliasChoices("segmented-folders", "segmented_folders"), + ) + remap_dims: Optional[str] = Field( default=None, - description="Optional strict allowlist of source dimensions to transfer during remap. When omitted, all extra dimensions are transferred. Listed names may be standard or extra dimensions. COPC-safe aliases may appear in outputs, for example 3DT_* source dimensions may be written as TDT_*.", + description=( + "Optional comma-separated allowlist of extra dimension names to transfer " + "from prediction collections during multi-collection remap." + ), validation_alias=AliasChoices("remap-dims", "remap_dims"), ) - @field_validator("workers", "threads", "tile_length", "tile_buffer", "chunk_size", "min_cluster_size") + output_merged_with_originals: Optional[Path] = Field( + default=None, + description="Legacy path for the old merged-with-originals remap output. Prod-merged outputs now use --merged-resolutions.", + validation_alias=AliasChoices("output-merged-with-originals", "output_merged_with_originals"), + ) + + transfer_original_dims_to_merged: bool = Field( + True, + description="Create prod-merged files from Original-with-predictions after merge/remap. Uses the create_merged_file implementation.", + validation_alias=AliasChoices("transfer-original-dims-to-merged", "transfer_original_dims_to_merged"), + ) + + threedtrees_dims: str = Field( + "PredInstance,PredSemantic", + description="Comma-separated list of dimension names produced by 3DTrees to transfer to original files. These are renamed to {name}_{suffix} in the output (e.g. PredInstance_SAT).", + validation_alias=AliasChoices("threedtrees-dims", "threedtrees_dims"), + ) + + threedtrees_suffix: str = Field( + "SAT", + description="Suffix for 3DTrees dimension names (e.g. SAT -> PredInstance_SAT).", + validation_alias=AliasChoices("threedtrees-suffix", "threedtrees_suffix"), + ) + + pre_remap_reassign_instances: bool = Field( + False, + description="Before remapping to originals, reassign small instances in the segmented/merged point cloud.", + validation_alias=AliasChoices("pre-remap-reassign-instances", "pre_remap_reassign_instances"), + ) + + pre_remap_reassign_instance_dimension: Optional[str] = Field( + default=None, + description="Instance dimension to update during pre-remap reassignment. Defaults to the first transferred dimension containing 'Instance'.", + validation_alias=AliasChoices("pre-remap-reassign-instance-dimension", "pre_remap_reassign_instance_dimension"), + ) + + pre_remap_reassign_min_cluster_size: int = Field( + 250, + description="Pre-remap reassignment: instances below this point count are reassigned to the nearest larger instance.", + validation_alias=AliasChoices("pre-remap-reassign-min-cluster-size", "pre_remap_reassign_min_cluster_size"), + ) + + pre_remap_reassign_hull_point_threshold: int = Field( + 5000, + description="Pre-remap reassignment: compute convex hulls for instances below this point count.", + validation_alias=AliasChoices("pre-remap-reassign-hull-point-threshold", "pre_remap_reassign_hull_point_threshold"), + ) + + pre_remap_reassign_max_volume: float = Field( + 5.0, + description="Pre-remap reassignment: instances below the hull point threshold and this hull volume in m3 are reassigned.", + validation_alias=AliasChoices("pre-remap-reassign-max-volume", "pre_remap_reassign_max_volume"), + ) + + pre_remap_reassigned_laz: Optional[Path] = Field( + default=None, + description="Optional path to save the segmented/merged point cloud after pre-remap reassignment.", + validation_alias=AliasChoices("pre-remap-reassigned-laz", "pre_remap_reassigned_laz"), + ) + + # ========================================================================== + # Remap task parameters + # ========================================================================== + + source_folder: Optional[Path] = Field( + default=None, + description="Path to source LAZ files (e.g., segmented files) for 'remap' task", + validation_alias=AliasChoices("source-folder", "source_folder"), + ) + + target_folder: Optional[Path] = Field( + default=None, + description="Path to target LAZ files (e.g., subsampled files) for 'remap' task", + validation_alias=AliasChoices("target-folder", "target_folder"), + ) + + # Merge algorithm parameters + buffer: Optional[float] = Field( + 10.0, + description="Buffer distance for filtering in meters (for 'merge' task)", + ) + + overlap_threshold: Optional[float] = Field( + 0.3, + description="Overlap ratio threshold for instance matching (0.3 = 30%)", + validation_alias=AliasChoices("overlap-threshold", "overlap_threshold"), + ) + + max_centroid_distance: Optional[float] = Field( + 3.0, + description="Max centroid distance to merge instances in meters", + validation_alias=AliasChoices("max-centroid-distance", "max_centroid_distance"), + ) + + max_volume_for_merge: Optional[float] = Field( + 4.0, + description="Max convex hull volume for small instance merging in m³", + validation_alias=AliasChoices("max-volume-for-merge", "max_volume_for_merge"), + ) + + border_zone_width: Optional[float] = Field( + 10.0, + description="Width of border zone beyond buffer for instance matching (meters)", + validation_alias=AliasChoices("border-zone-width", "border_zone_width"), + ) + + min_cluster_size: Optional[int] = Field( + 300, + description="Minimum cluster size in points for reassignment", + validation_alias=AliasChoices("min-cluster-size", "min_cluster_size"), + ) + + disable_matching: bool = Field( + False, + description="Disable cross-tile instance matching", + validation_alias=AliasChoices("disable-matching", "disable_matching"), + ) + + disable_volume_merge: bool = Field( + False, + description="Disable small volume instance merging", + validation_alias=AliasChoices("disable-volume-merge", "disable_volume_merge"), + ) + + skip_merged_file: bool = Field( + False, + description="Skip creating merged LAZ file (only create retiled outputs)", + validation_alias=AliasChoices("skip-merged-file", "skip_merged_file"), + ) + + verbose: bool = Field( + False, + description="Print detailed merge decisions", + ) + + # ========================================================================== + # Validators + # ========================================================================== + + @field_validator( + "input_dir", + "output_dir", + ) + @classmethod + def validate_tile_required_params(cls, v, info): + """Validate that tile task required parameters are provided.""" + # Note: Actual validation happens in run.py after instantiation + # since we need to check the task value + return v + + @field_validator( + "tile_length", + "resolution_1", + "resolution_2", + "threads", + "chunk_size", + ) @classmethod - def validate_positive_int(cls, value, info): - if value <= 0: + def validate_tile_params(cls, v, info): + """Validate tile parameters are positive when provided.""" + if v is not None and v <= 0: raise ValueError(f"{info.field_name} must be positive") - return value + return v + + @field_validator("tile_buffer") + @classmethod + def validate_tile_buffer(cls, v, info): + """Validate tile buffer is non-negative so zero-overlap tiling is allowed.""" + if v is not None and v < 0: + raise ValueError(f"{info.field_name} must be non-negative") + return v - @field_validator("resolution_1", "resolution_2", "border_zone_width", "max_volume_for_merge", "tiling_threshold") + @field_validator( + "buffer", + "overlap_threshold", + "max_centroid_distance", + "max_volume_for_merge", + "pre_remap_reassign_max_volume", + ) @classmethod - def validate_non_negative_float(cls, value, info): - if value is not None and value < 0: + def validate_merge_params(cls, v, info): + """Validate merge parameters are positive when provided.""" + if v is not None and v < 0: raise ValueError(f"{info.field_name} must be non-negative") - return value + return v + + @field_validator("overlap_threshold") + @classmethod + def validate_overlap_threshold(cls, v): + """Validate overlap threshold is between 0 and 1.""" + if v is not None and (v < 0 or v > 1): + raise ValueError("overlap_threshold must be between 0 and 1") + return v + + @field_validator("workers", "num_spatial_chunks", "min_cluster_size", "pre_remap_reassign_min_cluster_size", "pre_remap_reassign_hull_point_threshold") + @classmethod + def validate_positive_int(cls, v, info): + """Validate integer parameters are positive.""" + if v is not None and v <= 0: + raise ValueError(f"{info.field_name} must be positive") + return v + + @field_validator("subsampling_method") + @classmethod + def validate_subsampling_method(cls, v): + """Validate and normalize the subsampling method.""" + normalized = (v or "center-of-mass").strip().lower() + aliases = { + "com": "center-of-mass", + "center_of_mass": "center-of-mass", + "centroid": "nearest-to-centroid", + "voxelcentroidnearestneighbor": "nearest-to-centroid", + "voxel-centroid-nearest-neighbor": "nearest-to-centroid", + } + normalized = aliases.get(normalized, normalized) + if normalized not in {"center-of-mass", "nearest-to-centroid"}: + raise ValueError("subsampling_method must be 'center-of-mass' or 'nearest-to-centroid'") + return normalized + + @field_validator("merged_output_formats", mode="before") + @classmethod + def validate_merged_output_formats(cls, v): + """Validate and normalize prod-merged output formats.""" + aliases = { + "las": "laz", + "laz": "laz", + ".laz": "laz", + "copc": "copc.laz", + "copc_laz": "copc.laz", + "copc-laz": "copc.laz", + "copc.laz": "copc.laz", + ".copc.laz": "copc.laz", + "ply": "ply", + ".ply": "ply", + } + parsed = [] + seen = set() + + def tokens(value): + if isinstance(value, Iterable) and not isinstance(value, (str, bytes)): + for item in value: + yield from tokens(item) + return + text = str(value or "") + if text.startswith("[") and text.endswith("]"): + text = text[1:-1] + for token in text.split(","): + yield token.strip().strip("'\"") + + for raw_token in tokens(v or "copc.laz"): + token = raw_token.strip().lower() + if not token: + continue + output_format = aliases.get(token) + if output_format is None: + raise ValueError("merged_output_formats must contain only 'laz', 'copc.laz', or 'ply'") + if output_format in seen: + continue + seen.add(output_format) + parsed.append(output_format) + if not parsed: + raise ValueError("merged_output_formats must contain at least one format") + return ",".join(parsed) + + # ========================================================================== + # Model configuration + # ========================================================================== model_config = SettingsConfigDict( case_sensitive=False, cli_parse_args=True, cli_ignore_unknown_args=True, - env_prefix="", - extra="ignore", + env_prefix="", # No prefix for env vars + extra="ignore", # Ignore unknown fields ) @@ -221,55 +613,136 @@ def print_params(params: Parameters): print("=" * 60) print("Current Parameters") print("=" * 60) - print(f"task: {params.task}") - print(f"output_dir: {params.output_dir}") - print(f"workers: {params.workers}") - print(f"chunk_size: {params.chunk_size}") - print(f"input_dir: {params.input_dir}") - print(f"tile_length: {params.tile_length}") - print(f"tile_buffer: {params.tile_buffer}") - print(f"output_copc_res1: {params.output_copc_res1}") - print(f"output_copc_res2: {params.output_copc_res2}") - print(f"threads: {params.threads}") - print(f"resolution_1: {params.resolution_1}") - print(f"resolution_2: {params.resolution_2}") - print(f"dimension_reduction: {params.dimension_reduction}") - print(f"skip_dimension_reduction: {params.skip_dimension_reduction}") - print(f"num_spatial_chunks: {params.num_spatial_chunks}") - print(f"tiling_threshold: {params.tiling_threshold}") - print(f"chunkwise_copc_source_creation: {params.chunkwise_copc_source_creation}") - print(f"segmented_folders: {params.segmented_folders}") - print(f"tile_bounds_json: {params.tile_bounds_json}") - print(f"instance_dimension: {params.instance_dimension}") - print(f"border_zone_width: {params.border_zone_width}") - print(f"filter_anchor: {params.filter_anchor}") - print(f"min_cluster_size: {params.min_cluster_size}") - print(f"max_volume_for_merge: {params.max_volume_for_merge}") - print(f"enable_volume_merge: {params.enable_volume_merge}") - print(f"remap_merge: {params.remap_merge}") - print(f"original_input_dir: {params.original_input_dir}") - print(f"subsampled_target_folder: {params.subsampled_target_folder}") - print(f"produce_merged_file: {params.produce_merged_file}") - print(f"transfer_original_dims_to_merged: {params.transfer_original_dims_to_merged}") - print(f"output_merged_with_originals: {params.output_merged_with_originals}") - print(f"standardization_json: {params.standardization_json}") - print(f"remap_dims: {params.remap_dims}") + + print("\nCommon:") + print(f" task: {params.task}") + print(f" input_dir: {params.input_dir}") + print(f" output_dir: {params.output_dir}") + print(f" workers: {params.workers}") + print(f" num_spatial_chunks: {params.num_spatial_chunks}") + print(f" instance_dimension: {params.instance_dimension}") + print(f" filter_suffix: {params.filter_suffix}") + print(f" filter_output_extension: {params.filter_output_extension}") + + print("\nTile Task:") + print(f" tile_length: {params.tile_length}") + print(f" tile_buffer: {params.tile_buffer}") + print(f" threads: {params.threads}") + print(f" chunk_size: {params.chunk_size}") + print(f" resolution_1: {params.resolution_1}") + print(f" resolution_2: {params.resolution_2}") + print(f" output_copc_res1: {params.output_copc_res1}") + print(f" output_copc_res2: {params.output_copc_res2}") + print(f" subsampling_method: {params.subsampling_method}") + print(f" skip_dimension_reduction: {params.skip_dimension_reduction}") + + print("\nMerge Task:") + print(f" subsampled_10cm_folder: {params.subsampled_10cm_folder}") + print(f" original_input_dir: {params.original_input_dir}") + print(f" original_copc_input_dir: {params.original_copc_input_dir}") + print(f" original_raw_input_dir: {params.original_raw_input_dir}") + print(f" original_raw_output_dir: {params.original_raw_output_dir}") + print(f" buffer: {params.buffer}") + print(f" overlap_threshold: {params.overlap_threshold}") + print(f" max_centroid_distance: {params.max_centroid_distance}") + print(f" max_volume_for_merge: {params.max_volume_for_merge}") + print(f" min_cluster_size: {params.min_cluster_size}") + print(f" disable_matching: {params.disable_matching}") + print(f" verbose: {params.verbose}") + + print("\nCreate Merged File Task:") + print(f" original_with_predictions_dir: {params.original_with_predictions_dir}") + print(f" staged_copc_dir: {params.staged_copc_dir}") + print(f" standardization_json: {params.standardization_json}") + print(f" merged_resolutions: {params.merged_resolutions}") + print(f" merged_output_formats: {params.merged_output_formats}") + + print("\nRemap Task:") + print(f" merged_laz: {params.merged_laz}") + print(f" segmented_folders: {params.segmented_folders}") + print(f" remap_dims: {params.remap_dims}") + print(f" original_copc_input_dir: {params.original_copc_input_dir}") + print(f" original_raw_input_dir: {params.original_raw_input_dir}") + print(f" original_raw_output_dir: {params.original_raw_output_dir}") + print(f" threedtrees_dims: {params.threedtrees_dims}") + print(f" threedtrees_suffix: {params.threedtrees_suffix}") + print("=" * 60) -# Internal compatibility constants retained for helper modules that still import them. +# Legacy compatibility: provide dict-like access for modules that need it +def get_tile_params(params: Parameters) -> dict: + """Get tile parameters as a dictionary for legacy compatibility.""" + return { + 'tile_length': params.tile_length, + 'tile_buffer': params.tile_buffer, + 'threads': params.threads, + 'workers': params.workers, + 'resolution_1': params.resolution_1, + 'resolution_2': params.resolution_2, + 'output_copc_res1': params.output_copc_res1, + 'output_copc_res2': params.output_copc_res2, + 'subsampling_method': params.subsampling_method, + 'skip_dimension_reduction': params.skip_dimension_reduction, + 'chunk_size': params.chunk_size, + } + + +def get_merge_params(params: Parameters) -> dict: + """Get merge parameters as a dictionary for legacy compatibility.""" + return { + 'buffer': params.buffer, + 'overlap_threshold': params.overlap_threshold, + 'max_centroid_distance': params.max_centroid_distance, + 'max_volume_for_merge': params.max_volume_for_merge, + 'min_cluster_size': params.min_cluster_size, + 'workers': params.workers, + 'verbose': params.verbose, + 'instance_dimension': params.instance_dimension, + } + + +def get_remap_params(params: Parameters) -> dict: + """Get remap parameters as a dictionary for legacy compatibility.""" + return { + 'workers': params.workers, + 'instance_dimension': params.instance_dimension, + } + + +# Legacy dict exports for backwards compatibility with modules that import them directly TILE_PARAMS = { - "tile_length": 100, - "tile_buffer": 20, - "threads": 10, - "workers": 4, - "resolution_1": 0.02, - "resolution_2": 0.1, - "skip_dimension_reduction": False, - "chunk_size": 20_000_000, - "chunkwise_copc_source_creation": False, + 'tile_length': 100, + 'tile_buffer': 20, + 'threads': 10, + 'workers': 4, + 'resolution_1': 0.01, + 'resolution_2': 0.1, + 'output_copc_res1': True, + 'output_copc_res2': False, + 'subsampling_method': 'center-of-mass', + 'skip_dimension_reduction': False, + 'chunk_size': 20_000_000, +} + +REMAP_PARAMS = { + 'target_resolution_cm': 2, + 'workers': 4, +} + +MERGE_PARAMS = { + 'buffer': 10.0, + 'overlap_threshold': 0.3, + 'max_centroid_distance': 3.0, + 'max_volume_for_merge': 4.0, + 'min_cluster_size': 300, + 'workers': 4, + 'verbose': True, + 'retile_buffer': 2.0, # Fixed to 2.0m } if __name__ == "__main__": - print_params(Parameters()) + """CLI for viewing/testing parameter configuration.""" + params = Parameters() + print_params(params) diff --git a/src/plot_tiles_and_copc.py b/src/plot_tiles_and_copc.py index 78f7a2b..d3cf224 100755 --- a/src/plot_tiles_and_copc.py +++ b/src/plot_tiles_and_copc.py @@ -25,14 +25,14 @@ def load_source_extents(tindex_path: Path, target_crs: str = "EPSG:32632"): """Load source point cloud file extents from tindex and transform to target CRS.""" extents = [] filenames = [] - + with fiona.open(tindex_path) as src: # Get source CRS src_crs = str(src.crs) if src.crs else "EPSG:32632" - + for feature in src: geom = feature['geometry'] - + # Get bounds from geometry if geom['type'] == 'Polygon': coords = geom['coordinates'][0] @@ -40,14 +40,14 @@ def load_source_extents(tindex_path: Path, target_crs: str = "EPSG:32632"): coords = [c for poly in geom['coordinates'] for c in poly[0]] else: continue - + xs, ys = zip(*coords) xmin, xmax = min(xs), max(xs) ymin, ymax = min(ys), max(ys) - + # Detect if coordinates are already projected (values > 360 are clearly not lat/lon) is_projected = abs(xmin) > 360 or abs(xmax) > 360 or abs(ymin) > 360 or abs(ymax) > 360 - + if is_projected and "4326" in src_crs: # CRS is misreported as WGS84, but coordinates are already projected print(f"Note: CRS reported as {src_crs} but coordinates appear projected (assuming {target_crs})") @@ -68,9 +68,9 @@ def load_source_extents(tindex_path: Path, target_crs: str = "EPSG:32632"): ymin, ymax = min(proj_ys), max(proj_ys) except Exception as e: print(f"WARNING: Could not transform coordinates: {e}", file=sys.stderr) - + extents.append((xmin, ymin, xmax, ymax)) - + # Get filename from properties file_path = feature['properties'].get('Location', '') if file_path: @@ -78,7 +78,7 @@ def load_source_extents(tindex_path: Path, target_crs: str = "EPSG:32632"): else: filename = f"file_{len(extents)}" filenames.append(filename) - + return extents, filenames @@ -86,19 +86,19 @@ def load_tile_extents(tile_bounds_json: Path): """Load tile extents from tile_bounds JSON.""" with tile_bounds_json.open() as f: data = json.load(f) - + tiles = [] for tile in data['tiles']: bounds = tile['bounds'] xmin, xmax = bounds[0] ymin, ymax = bounds[1] - + tiles.append({ 'label': f"c{tile['col']:02d}_r{tile['row']:02d}", 'bounds': (xmin, ymin, xmax, ymax), 'core': tile.get('core', None) }) - + return tiles, data.get('proj_srs', 'EPSG:32632') @@ -112,37 +112,37 @@ def plot_extents(tindex_path: Path, tile_bounds_json: Path, output_png: Path): print("Loading source file extents from tindex...") source_extents, source_names = load_source_extents(tindex_path, target_crs=proj_srs) print(f"Found {len(source_extents)} source files") - + # Calculate overall extent all_xs = [] all_ys = [] - + for xmin, ymin, xmax, ymax in source_extents: all_xs.extend([xmin, xmax]) all_ys.extend([ymin, ymax]) - + for tile in tiles: xmin, ymin, xmax, ymax = tile['bounds'] all_xs.extend([xmin, xmax]) all_ys.extend([ymin, ymax]) - + overall_xmin, overall_xmax = min(all_xs), max(all_xs) overall_ymin, overall_ymax = min(all_ys), max(all_ys) - + # Add padding x_padding = (overall_xmax - overall_xmin) * 0.05 y_padding = (overall_ymax - overall_ymin) * 0.05 - + # Create figure fig, ax = plt.subplots(1, 1, figsize=(16, 12)) - + # Plot source file extents source_patches = [] for xmin, ymin, xmax, ymax in source_extents: width = xmax - xmin height = ymax - ymin - rect = mpatches.Rectangle((xmin, ymin), width, height, - edgecolor='blue', facecolor='lightblue', + rect = mpatches.Rectangle((xmin, ymin), width, height, + edgecolor='blue', facecolor='lightblue', alpha=0.5, linewidth=1.5) source_patches.append(rect) @@ -153,11 +153,11 @@ def plot_extents(tindex_path: Path, tile_bounds_json: Path, output_png: Path): for (xmin, ymin, xmax, ymax), name in zip(source_extents, source_names): center_x = (xmin + xmax) / 2 center_y = (ymin + ymax) / 2 - ax.text(center_x, center_y, name, - ha='center', va='center', + ax.text(center_x, center_y, name, + ha='center', va='center', fontsize=8, color='darkblue', weight='bold', bbox=dict(boxstyle='round,pad=0.3', facecolor='white', alpha=0.7)) - + # Plot tile extents (full bounds including buffer) tile_patches = [] for tile in tiles: @@ -168,7 +168,7 @@ def plot_extents(tindex_path: Path, tile_bounds_json: Path, output_png: Path): edgecolor='indianred', facecolor='mistyrose', alpha=0.4, linewidth=2) tile_patches.append(rect) - + # Add tile label at center center_x = (xmin + xmax) / 2 center_y = (ymin + ymax) / 2 @@ -176,10 +176,10 @@ def plot_extents(tindex_path: Path, tile_bounds_json: Path, output_png: Path): ha='center', va='center', fontsize=10, color='darkred', weight='bold', bbox=dict(boxstyle='round,pad=0.5', facecolor='white', alpha=0.8)) - + tile_collection = PatchCollection(tile_patches, match_original=True) ax.add_collection(tile_collection) - + # Set limits and labels ax.set_xlim(overall_xmin - x_padding, overall_xmax + x_padding) ax.set_ylim(overall_ymin - y_padding, overall_ymax + y_padding) @@ -192,7 +192,7 @@ def plot_extents(tindex_path: Path, tile_bounds_json: Path, output_png: Path): # Add legend source_legend = mpatches.Patch(color='lightblue', alpha=0.5, label='Source file extent') - tile_legend = mpatches.Patch(facecolor='mistyrose', edgecolor='indianred', + tile_legend = mpatches.Patch(facecolor='mistyrose', edgecolor='indianred', alpha=0.4, linewidth=2, label='Tile extent') ax.legend(handles=[source_legend, tile_legend], loc='upper right', fontsize=10) @@ -201,16 +201,16 @@ def plot_extents(tindex_path: Path, tile_bounds_json: Path, output_png: Path): ax.text(0.02, 0.98, stats_text, transform=ax.transAxes, fontsize=10, verticalalignment='top', bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5)) - + plt.tight_layout() - + # Save figure output_png.parent.mkdir(parents=True, exist_ok=True) plt.savefig(output_png, dpi=300, bbox_inches='tight') print(f"\nVisualization saved to: {output_png}") print(f" - {len(source_extents)} source files (blue)") print(f" - {len(tiles)} tiles (light red)") - + # Optionally show plot # plt.show() @@ -236,15 +236,15 @@ def main(): help="Output PNG file path" ) args = parser.parse_args() - + if not args.tindex_path.exists(): print(f"ERROR: Tindex file not found: {args.tindex_path}") sys.exit(1) - + if not args.tile_bounds_json.exists(): print(f"ERROR: Tile bounds JSON not found: {args.tile_bounds_json}") sys.exit(1) - + plot_extents(args.tindex_path, args.tile_bounds_json, args.output) diff --git a/src/point_cloud_metadata.py b/src/point_cloud_metadata.py new file mode 100644 index 0000000..8ab86c3 --- /dev/null +++ b/src/point_cloud_metadata.py @@ -0,0 +1,272 @@ +#!/usr/bin/env python3 +"""Shared LAS/LAZ metadata helpers for SmartTile products.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import List, Optional, Set, Tuple + +import laspy +from laspy.vlrs.vlrlist import VLRList + + +# R/lidR, LAS, and laspy names can all appear in standardization summaries. +# Extra-byte names are preserved as-is via the .get() default. +DIMENSION_NAME_ALIASES = { + "Intensity": "intensity", + "intensity": "intensity", + "ReturnNumber": "return_number", + "return_number": "return_number", + "NumberOfReturns": "number_of_returns", + "number_of_returns": "number_of_returns", + "ScanDirectionFlag": "scan_direction_flag", + "scan_direction_flag": "scan_direction_flag", + "EdgeOfFlightline": "edge_of_flight_line", + "edge_of_flight_line": "edge_of_flight_line", + "Classification": "classification", + "classification": "classification", + "ScannerChannel": "scanner_channel", + "scanner_channel": "scanner_channel", + "Synthetic_flag": "synthetic", + "synthetic": "synthetic", + "Keypoint_flag": "key_point", + "key_point": "key_point", + "Withheld_flag": "withheld", + "withheld": "withheld", + "Overlap_flag": "overlap", + "overlap": "overlap", + "ScanAngle": "scan_angle", + "scan_angle": "scan_angle", + "ScanAngleRank": "scan_angle", + "scan_angle_rank": "scan_angle", + "UserData": "user_data", + "user_data": "user_data", + "PointSourceID": "point_source_id", + "point_source_id": "point_source_id", + "gpstime": "gps_time", + "gps_time": "gps_time", + "R": "red", + "red": "red", + "G": "green", + "green": "green", + "B": "blue", + "blue": "blue", +} + + +def extra_bytes_params_from_dimension_info( + dim_info, + name: Optional[str] = None, +) -> laspy.ExtraBytesParams: + """Build ExtraBytesParams from laspy DimensionInfo while preserving metadata.""" + return laspy.ExtraBytesParams( + name=name or dim_info.name, + type=dim_info.dtype, + description=getattr(dim_info, "description", "") or "", + offsets=getattr(dim_info, "offsets", None), + scales=getattr(dim_info, "scales", None), + no_data=getattr(dim_info, "no_data", None), + ) + + +def extra_bytes_params_from_params( + params: laspy.ExtraBytesParams, + name: Optional[str] = None, +) -> laspy.ExtraBytesParams: + """Clone ExtraBytesParams while optionally renaming the dimension.""" + return laspy.ExtraBytesParams( + name=name or params.name, + type=params.type, + description=getattr(params, "description", "") or "", + offsets=getattr(params, "offsets", None), + scales=getattr(params, "scales", None), + no_data=getattr(params, "no_data", None), + ) + + +def is_stale_copc_vlr(vlr) -> bool: + """Return True for COPC index VLRs that cannot be copied to a new container.""" + return getattr(vlr, "user_id", "") == "copc" and getattr(vlr, "record_id", None) in (1, 2) + + +def is_extra_bytes_vlr(vlr) -> bool: + """Return True for ExtraBytes metadata that laspy regenerates from dimensions.""" + return getattr(vlr, "user_id", "") == "LASF_Spec" and getattr(vlr, "record_id", None) == 4 + + +def copy_single_source_header( + source_header: laspy.LasHeader, + offsets=None, + scales=None, + preserve_extra_dimensions: bool = True, +) -> laspy.LasHeader: + """Copy a source header for outputs that still represent exactly that source file.""" + if preserve_extra_dimensions: + header = source_header.copy() + else: + fmt_id = getattr(source_header.point_format, "id", source_header.point_format) + if hasattr(fmt_id, "id"): + fmt_id = fmt_id.id + header = laspy.LasHeader(point_format=int(fmt_id), version=source_header.version) + for attr in ( + "file_source_id", + "global_encoding", + "uuid", + "system_identifier", + "generating_software", + "creation_date", + ): + if hasattr(source_header, attr): + setattr(header, attr, getattr(source_header, attr)) + + header.offsets = offsets if offsets is not None else source_header.offsets + header.scales = scales if scales is not None else source_header.scales + + source_vlrs = header.vlrs if preserve_extra_dimensions else source_header.vlrs + header.vlrs = VLRList([ + vlr for vlr in source_vlrs + if not is_stale_copc_vlr(vlr) + and (preserve_extra_dimensions or not is_extra_bytes_vlr(vlr)) + ]) + source_evlrs = ( + getattr(header, "evlrs", None) + if preserve_extra_dimensions + else getattr(source_header, "evlrs", None) + ) + if source_evlrs is not None: + header.evlrs = VLRList([ + vlr for vlr in source_evlrs + if not is_stale_copc_vlr(vlr) + and (preserve_extra_dimensions or not is_extra_bytes_vlr(vlr)) + ]) + + return header + + +def projection_metadata_vlrs(vlrs) -> VLRList: + """Keep CRS/projection records that remain true across a CRS-consistent run.""" + return VLRList([ + vlr for vlr in (vlrs or []) + if getattr(vlr, "user_id", "") == "LASF_Projection" + ]) + + +def _point_cloud_paths(directory: Optional[Path]) -> List[Path]: + """Return LAS/LAZ-like paths case-insensitively.""" + if directory is None or not directory.exists(): + return [] + return sorted( + ( + path + for path in directory.iterdir() + if path.is_file() + and path.name.lower().endswith((".laz", ".las")) + ), + key=lambda path: path.name.lower(), + ) + + +def point_cloud_files(directory: Optional[Path]) -> List[Path]: + """Return point-cloud files, preferring COPC over matching plain LAZ/LAS.""" + files = _point_cloud_paths(directory) + by_source = {} + for path in files: + key = point_cloud_source_key(path) + existing = by_source.get(key) + if existing is None or path.name.lower().endswith(".copc.laz"): + by_source[key] = path + return [by_source[key] for key in sorted(by_source)] + + +def raw_point_cloud_files(directory: Optional[Path]) -> List[Path]: + """Return only non-COPC LAS/LAZ point-cloud files.""" + files = _point_cloud_paths(directory) + return [path for path in files if not path.name.lower().endswith(".copc.laz")] + + +def copc_files(directory: Optional[Path]) -> List[Path]: + """Return COPC LAZ point-cloud files.""" + return [path for path in _point_cloud_paths(directory) if path.name.lower().endswith(".copc.laz")] + + +def point_cloud_source_key(path: Path) -> str: + """Return a stable source key shared by raw LAS/LAZ and derived COPC files.""" + name = path.name.lower() + if name.endswith(".copc.laz"): + return name[:-9] + if name.endswith(".laz") or name.endswith(".las"): + return name.rsplit(".", 1)[0] + return path.stem.lower() + + +def point_cloud_dimension_names(path: Path) -> Set[str]: + """Return all standard and extra dimension names in a point-cloud header.""" + with laspy.open(str(path), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + names = set(str(name) for name in reader.header.point_format.dimension_names) + names.update(dim.name for dim in reader.header.point_format.extra_dimensions) + return names + + +def load_standardization_dims(json_path: Path) -> Set[str]: + """Load expected source dimensions from a tool_standard collection summary. + + The v2.1 SmartTile contract accepted `collection_summary.json` and used its + `collection.reference_attribute_names` as the canonical source-attribute + schema. Constant/all-zero dimensions are ignored when + `global_attribute_stats` is present, matching the old behavior. + """ + with Path(json_path).open() as handle: + data = json.load(handle) + + collection = data.get("collection", data) + ref_names = collection.get("reference_attribute_names") + if not isinstance(ref_names, list): + raise ValueError( + f"standardization JSON does not contain collection.reference_attribute_names: {json_path}" + ) + + global_stats = collection.get("global_attribute_stats", []) + has_variation = set() + if isinstance(global_stats, list): + for stat in global_stats: + if not isinstance(stat, dict): + continue + name = stat.get("name", "") + variance = stat.get("variance", 0) + try: + if float(variance) > 0: + has_variation.add(name) + except (TypeError, ValueError): + pass + + expected = set() + skipped = [] + for name in ref_names: + if name in ("X", "Y", "Z"): + continue + if has_variation and name not in has_variation: + skipped.append(name) + continue + expected.add(DIMENSION_NAME_ALIASES.get(name, name)) + + if skipped: + print( + f" Standardization JSON: skipping {len(skipped)} constant/zero dims: {skipped}", + flush=True, + ) + return expected + + +def bounds_overlap_xy( + a: Tuple[float, float, float, float], + b: Tuple[float, float, float, float], + buffer: float = 0.0, +) -> bool: + """Return whether two XY bounds overlap, optionally expanding both by buffer.""" + return not ( + a[1] < b[0] - buffer + or a[0] > b[1] + buffer + or a[3] < b[2] - buffer + or a[2] > b[3] + buffer + ) diff --git a/src/point_cloud_outputs.py b/src/point_cloud_outputs.py new file mode 100644 index 0000000..471d1ea --- /dev/null +++ b/src/point_cloud_outputs.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""Point-cloud output writers for SmartTile merge/remap products. + +This module owns the boundary where in-memory SmartTile arrays become LAS/LAZ +products. Keeping that logic here makes the metadata contract easier to audit: +outputs that still describe one original source preserve that source header as +far as LAS/COPC allows, while merged multi-source products preserve only +run-true CRS/projection metadata. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Dict, Optional + +import laspy +import numpy as np + +from instance_labels import MERGED_OUTPUT_SCALES +from point_cloud_metadata import ( + copy_single_source_header, + point_cloud_files, + projection_metadata_vlrs, +) + + +def merged_product_header( + merged_points: np.ndarray, + original_input_dir: Optional[Path], + original_tiles_dir: Path, +) -> laspy.LasHeader: + """Build a merged product header from source metadata when it remains true.""" + offsets = np.min(merged_points, axis=0) + source_files = point_cloud_files(original_input_dir) or point_cloud_files(original_tiles_dir) + + if len(source_files) == 1: + with laspy.open(str(source_files[0]), laz_backend=laspy.LazBackend.LazrsParallel) as source: + header = copy_single_source_header( + source.header, + offsets=offsets, + scales=MERGED_OUTPUT_SCALES, + preserve_extra_dimensions=False, + ) + header.point_count = 0 + return header + + header = laspy.LasHeader(point_format=6, version="1.4") + header.offsets = offsets + header.scales = MERGED_OUTPUT_SCALES + + if source_files: + with laspy.open(str(source_files[0]), laz_backend=laspy.LazBackend.LazrsParallel) as source: + source_header = source.header + header.global_encoding = source_header.global_encoding + header.vlrs = projection_metadata_vlrs(source_header.vlrs) + source_evlrs = getattr(source_header, "evlrs", None) + if source_evlrs is not None: + header.evlrs = projection_metadata_vlrs(source_evlrs) + + return header + + +def write_loaded_point_cloud( + source_file: Path, + output_file: Path, + points: np.ndarray, + all_dims: Dict[str, np.ndarray], + source_indices: Optional[np.ndarray] = None, +) -> None: + """Write loaded points/dimensions back to LAZ while preserving source metadata.""" + output_file.parent.mkdir(parents=True, exist_ok=True) + with laspy.open(str(source_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + header = copy_single_source_header(reader.header, preserve_extra_dimensions=True) + source_las = reader.read() + source_points = source_las.points + + output_las = laspy.LasData(header) + if source_indices is not None: + selected_source_points = source_points[np.asarray(source_indices)] + elif len(source_points) == len(points): + selected_source_points = source_points + else: + selected_source_points = None + + output_las.points = laspy.ScaleAwarePointRecord.zeros(len(points), header=header) + if selected_source_points is not None: + source_dim_names = set(source_points.point_format.dimension_names) + for dim_name in output_las.point_format.dimension_names: + if dim_name in {"X", "Y", "Z"} or dim_name not in source_dim_names: + continue + output_las.points[dim_name] = selected_source_points[dim_name] + + output_las.x = points[:, 0] + output_las.y = points[:, 1] + output_las.z = points[:, 2] + + output_dimension_names = set(output_las.point_format.dimension_names) + output_dimension_names.update(dim.name for dim in output_las.point_format.extra_dimensions) + for dim_name, values in all_dims.items(): + if dim_name in ("X", "Y", "Z"): + continue + if dim_name not in output_dimension_names: + output_las.add_extra_dim(laspy.ExtraBytesParams(name=dim_name, type=values.dtype)) + output_dimension_names.add(dim_name) + setattr(output_las, dim_name, values) + + output_las.write( + str(output_file), + do_compress=output_file.name.lower().endswith(".laz"), + laz_backend=laspy.LazBackend.LazrsParallel, + ) diff --git a/src/prediction_collection_remap.py b/src/prediction_collection_remap.py new file mode 100644 index 0000000..689b01d --- /dev/null +++ b/src/prediction_collection_remap.py @@ -0,0 +1,573 @@ +#!/usr/bin/env python3 +"""Stream finalized prediction collections onto original point-cloud files. + +Prediction collections are expected to be finalized independently before this +step. Each collection keeps its model-specific dimension names; duplicate names +across collections fail early instead of being renamed at product time. +""" + +from __future__ import annotations + +import gc +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Dict, List, Optional, Set, Tuple + +import laspy +import numpy as np +from scipy.spatial import cKDTree + +from point_cloud_metadata import ( + bounds_overlap_xy, + copy_single_source_header, + extra_bytes_params_from_dimension_info, + point_cloud_files, + raw_point_cloud_files, +) +from worker_budget import kdtree_query_workers + + +def prediction_collection_files(path: Path) -> List[Path]: + """Return LAZ/LAS files for a prediction collection folder or single file.""" + path = Path(path) + if path.is_file(): + if path.suffix.lower() == ".las" or path.name.lower().endswith(".laz"): + return [path] + return [] + return point_cloud_files(path) + + +def scan_prediction_collection_metadata( + collections: List[Path], + target_dims: Optional[Set[str]] = None, +) -> List[Dict[str, object]]: + """Scan prediction collections and fail on duplicate output extra-dimension names.""" + collection_meta: List[Dict[str, object]] = [] + seen_dims: Dict[str, Path] = {} + + for collection in collections: + files = prediction_collection_files(collection) + if not files: + raise ValueError(f"No LAZ/LAS files found in prediction collection: {collection}") + + file_meta = [] + collection_dims: Dict[str, laspy.ExtraBytesParams] = {} + for file_path in files: + with laspy.open(str(file_path), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + header = reader.header + bounds = (header.x_min, header.x_max, header.y_min, header.y_max) + z_bounds = (header.z_min, header.z_max) + extra_names = {dim.name for dim in header.point_format.extra_dimensions} + for dim in header.point_format.extra_dimensions: + if target_dims is not None and dim.name not in target_dims: + continue + if dim.name not in collection_dims: + collection_dims[dim.name] = extra_bytes_params_from_dimension_info(dim) + file_meta.append({ + "path": file_path, + "bounds": bounds, + "z_bounds": z_bounds, + "extra_names": extra_names, + }) + + selected_names = sorted(collection_dims.keys()) + if not selected_names: + wanted = ", ".join(sorted(target_dims)) if target_dims else "any extra dimensions" + raise ValueError(f"Prediction collection {collection} has no selected dimensions ({wanted})") + + for file_info in file_meta: + missing = set(selected_names) - set(file_info["extra_names"]) + if missing: + raise ValueError( + f"Prediction collection {collection} has inconsistent schema: " + f"{Path(file_info['path']).name} is missing {', '.join(sorted(missing))}" + ) + + for dim_name in selected_names: + if dim_name in seen_dims: + raise ValueError( + "Duplicate prediction dimension name across collections: " + f"{dim_name} appears in {seen_dims[dim_name]} and {collection}. " + "Model outputs must be uniquely named before final remap." + ) + seen_dims[dim_name] = collection + + collection_meta.append( + { + "path": collection, + "files": file_meta, + "dims": selected_names, + "extra_params": collection_dims, + } + ) + + return collection_meta + + +def load_collection_subset_for_bounds( + coll_meta: Dict[str, object], + bounds: Tuple[float, float, float, float], + spatial_buffer: float, +) -> Tuple[np.ndarray, Dict[str, np.ndarray]]: + """Load prediction points/dim arrays inside a small XY bounds window.""" + min_x, max_x, min_y, max_y = bounds + min_x -= spatial_buffer + max_x += spatial_buffer + min_y -= spatial_buffer + max_y += spatial_buffer + + point_chunks = [] + dim_chunks: Dict[str, List[np.ndarray]] = {name: [] for name in coll_meta["dims"]} + + for file_info in coll_meta["files"]: + if not bounds_overlap_xy(file_info["bounds"], (min_x, max_x, min_y, max_y), 0.0): + continue + source_path = Path(file_info["path"]) + if source_path.name.lower().endswith(".copc.laz"): + z_min, z_max = file_info.get("z_bounds", (None, None)) + if z_min is None or z_max is None: + z_min, z_max = -np.inf, np.inf + query_bounds = laspy.copc.Bounds( + mins=np.array([min_x, min_y, float(z_min)], dtype=np.float64), + maxs=np.array([max_x, max_y, float(z_max)], dtype=np.float64), + ) + with laspy.CopcReader.open(str(source_path)) as copc_reader: + chunk = copc_reader.spatial_query(query_bounds) + if len(chunk) == 0: + continue + xs = np.asarray(chunk.x) + ys = np.asarray(chunk.y) + mask = (xs >= min_x) & (xs <= max_x) & (ys >= min_y) & (ys <= max_y) + if not np.any(mask): + continue + point_chunks.append(np.column_stack([xs[mask], ys[mask], np.asarray(chunk.z)[mask]])) + for dim_name in coll_meta["dims"]: + dim_chunks[dim_name].append(np.asarray(chunk[dim_name])[mask]) + else: + with laspy.open(str(source_path), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + for chunk in reader.chunk_iterator(5_000_000): + xs = np.asarray(chunk.x) + ys = np.asarray(chunk.y) + mask = (xs >= min_x) & (xs <= max_x) & (ys >= min_y) & (ys <= max_y) + if not np.any(mask): + continue + point_chunks.append(np.column_stack([xs[mask], ys[mask], np.asarray(chunk.z)[mask]])) + for dim_name in coll_meta["dims"]: + dim_chunks[dim_name].append(np.asarray(chunk[dim_name])[mask]) + + if not point_chunks: + raise ValueError(f"No prediction points found in {coll_meta['path']} for bounds {bounds}") + + points = np.concatenate(point_chunks, axis=0) + dims = {name: np.concatenate(chunks) for name, chunks in dim_chunks.items()} + return points, dims + + +def stream_add_collections_to_file( + input_file: Path, + output_file: Path, + collection_meta: List[Dict[str, object]], + spatial_buffer: float, + tolerance: float, + chunk_size: int = 5_000_000, + kdtree_workers: int = 1, +) -> Tuple[int, int]: + """Stream all prediction collections onto one original file in one write pass.""" + output_file.parent.mkdir(parents=True, exist_ok=True) + chunk_spatial_buffer = max(spatial_buffer, tolerance * 2.0, 0.25) + + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + header = copy_single_source_header(reader.header, preserve_extra_dimensions=True) + existing_names = set(header.point_format.dimension_names) + existing_names.update(dim.name for dim in header.point_format.extra_dimensions) + extra_dims_to_add = [] + for coll_meta in collection_meta: + for dim_name in coll_meta["dims"]: + if dim_name in existing_names: + raise ValueError( + f"Prediction dimension {dim_name} from collection {coll_meta['path']} " + f"collides with an existing dimension in {input_file.name}" + ) + params = coll_meta["extra_params"][dim_name] + extra_dims_to_add.append(params) + existing_names.add(dim_name) + if extra_dims_to_add: + header.add_extra_dims(extra_dims_to_add) + + n_points = 0 + matched_total = 0 + with laspy.open( + str(output_file), + mode="w", + header=header, + laz_backend=laspy.LazBackend.LazrsParallel, + ) as writer: + for chunk in reader.chunk_iterator(chunk_size): + chunk_points = np.column_stack([chunk.x, chunk.y, chunk.z]) + chunk_bounds = ( + float(np.min(chunk_points[:, 0])), + float(np.max(chunk_points[:, 0])), + float(np.min(chunk_points[:, 1])), + float(np.max(chunk_points[:, 1])), + ) + out_chunk = laspy.ScaleAwarePointRecord.zeros(len(chunk), header=header) + for dim_name in chunk.point_format.dimension_names: + if dim_name in out_chunk.point_format.dimension_names: + out_chunk[dim_name] = chunk[dim_name] + + for coll_meta in collection_meta: + source_points, source_dims = load_collection_subset_for_bounds( + coll_meta, + chunk_bounds, + chunk_spatial_buffer, + ) + tree = cKDTree(source_points) + distances, indices = tree.query(chunk_points, workers=kdtree_workers) + matched = distances <= tolerance + matched_count = int(np.count_nonzero(matched)) + if matched_count != len(chunk): + raise ValueError( + f"Prediction collection {coll_meta['path']} matched " + f"{matched_count:,}/{len(chunk):,} points in chunk from {input_file.name} " + f"within tolerance {tolerance} m" + ) + for dim_name, values in source_dims.items(): + out_chunk[dim_name] = values[indices] + matched_total += matched_count + del source_points, source_dims, tree, distances, indices + + writer.write_points(out_chunk) + n_points += len(chunk) + del chunk_points, out_chunk + if n_points % 25_000_000 < len(chunk): + print( + f" prediction collections -> {output_file.name}: " + f"{n_points:,} points", + flush=True, + ) + + gc.collect() + return n_points, matched_total + + +def _add_prediction_dims_to_header(header, collection_meta: List[Dict[str, object]], input_name: str) -> None: + """Add selected prediction dimensions to an output header, failing on collisions.""" + existing_names = set(header.point_format.dimension_names) + existing_names.update(dim.name for dim in header.point_format.extra_dimensions) + extra_dims_to_add = [] + for coll_meta in collection_meta: + for dim_name in coll_meta["dims"]: + if dim_name in existing_names: + raise ValueError( + f"Prediction dimension {dim_name} from collection {coll_meta['path']} " + f"collides with an existing dimension in {input_name}" + ) + params = coll_meta["extra_params"][dim_name] + extra_dims_to_add.append(params) + existing_names.add(dim_name) + if extra_dims_to_add: + header.add_extra_dims(extra_dims_to_add) + + +def _copy_source_record_dimensions(source_points, out_record) -> None: + """Copy dimensions present on a source point record into an output record.""" + output_dim_names = set(out_record.point_format.dimension_names) + output_dim_names.update(dim.name for dim in out_record.point_format.extra_dimensions) + for dim_name in source_points.point_format.dimension_names: + if dim_name in output_dim_names: + out_record[dim_name] = source_points[dim_name] + for dim in source_points.point_format.extra_dimensions: + if dim.name in output_dim_names: + out_record[dim.name] = source_points[dim.name] + + +def _copc_spatial_windows(header, num_spatial_chunks: int) -> List[Tuple[float, float, bool]]: + """Return X windows for COPC spatial-query enrichment.""" + min_x = float(header.x_min) + max_x = float(header.x_max) + if max_x <= min_x: + return [(min_x, max_x, True)] + + chunks = max(1, int(num_spatial_chunks or 1)) + step = (max_x - min_x) / chunks + windows = [] + for idx in range(chunks): + start = min_x + idx * step + stop = max_x if idx == chunks - 1 else min_x + (idx + 1) * step + windows.append((start, stop, idx == chunks - 1)) + return windows + + +def _query_copc_window(copc_reader, header, min_x: float, max_x: float): + """Load one COPC X window with native COPC spatial query.""" + bounds = laspy.copc.Bounds( + mins=np.array([min_x, float(header.y_min), float(header.z_min)], dtype=np.float64), + maxs=np.array([max_x, float(header.y_max), float(header.z_max)], dtype=np.float64), + ) + return copc_reader.spatial_query(bounds) + + +def stream_add_collections_to_copc_file_spatial( + input_file: Path, + output_file: Path, + collection_meta: List[Dict[str, object]], + spatial_buffer: float, + tolerance: float, + num_spatial_chunks: int = 4, + kdtree_workers: int = 1, +) -> Tuple[int, int]: + """Spatial-query a COPC original and write one enriched LAZ output.""" + output_file.parent.mkdir(parents=True, exist_ok=True) + chunk_spatial_buffer = max(spatial_buffer, tolerance * 2.0, 0.25) + + with laspy.CopcReader.open(str(input_file)) as copc_reader: + source_header = copc_reader.header + header = copy_single_source_header(source_header, preserve_extra_dimensions=True) + _add_prediction_dims_to_header(header, collection_meta, input_file.name) + windows = _copc_spatial_windows(source_header, num_spatial_chunks) + + n_points = 0 + matched_total = 0 + with laspy.open( + str(output_file), + mode="w", + header=header, + laz_backend=laspy.LazBackend.LazrsParallel, + ) as writer: + for window_idx, (min_x, max_x, include_upper) in enumerate(windows, start=1): + source_points = _query_copc_window(copc_reader, source_header, min_x, max_x) + if len(source_points) == 0: + continue + + xs = np.asarray(source_points.x) + if include_upper: + mask = (xs >= min_x) & (xs <= max_x) + else: + mask = (xs >= min_x) & (xs < max_x) + if not np.any(mask): + continue + if not np.all(mask): + source_points = source_points[mask] + + chunk_points = np.column_stack([source_points.x, source_points.y, source_points.z]) + chunk_bounds = ( + float(np.min(chunk_points[:, 0])), + float(np.max(chunk_points[:, 0])), + float(np.min(chunk_points[:, 1])), + float(np.max(chunk_points[:, 1])), + ) + out_chunk = laspy.ScaleAwarePointRecord.zeros(len(source_points), header=header) + _copy_source_record_dimensions(source_points, out_chunk) + + for coll_meta in collection_meta: + prediction_points, prediction_dims = load_collection_subset_for_bounds( + coll_meta, + chunk_bounds, + chunk_spatial_buffer, + ) + tree = cKDTree(prediction_points) + distances, indices = tree.query(chunk_points, workers=kdtree_workers) + matched = distances <= tolerance + matched_count = int(np.count_nonzero(matched)) + if matched_count != len(source_points): + raise ValueError( + f"Prediction collection {coll_meta['path']} matched " + f"{matched_count:,}/{len(source_points):,} points in COPC window " + f"{window_idx}/{len(windows)} from {input_file.name} " + f"within tolerance {tolerance} m" + ) + for dim_name, values in prediction_dims.items(): + out_chunk[dim_name] = values[indices] + matched_total += matched_count + del prediction_points, prediction_dims, tree, distances, indices + + writer.write_points(out_chunk) + n_points += len(source_points) + del source_points, chunk_points, out_chunk + print( + f" COPC spatial remap {input_file.name}: " + f"window {window_idx}/{len(windows)}, {n_points:,} points", + flush=True, + ) + + gc.collect() + return n_points, matched_total + + +def _existing_output_is_reusable( + input_file: Path, + output_file: Path, + expected_prediction_dims: List[str], +) -> bool: + """Return True when an existing enriched original matches this remap request.""" + if not output_file.exists() or output_file.stat().st_size == 0: + return False + try: + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as input_reader: + expected_count = int(input_reader.header.point_count) + input_dims = set(input_reader.header.point_format.dimension_names) + input_dims.update(dim.name for dim in input_reader.header.point_format.extra_dimensions) + if input_dims & set(expected_prediction_dims): + return False + with laspy.open(str(output_file), laz_backend=laspy.LazBackend.LazrsParallel) as output_reader: + if int(output_reader.header.point_count) != expected_count: + return False + output_dims = set(output_reader.header.point_format.dimension_names) + output_dims.update(dim.name for dim in output_reader.header.point_format.extra_dimensions) + return set(expected_prediction_dims).issubset(output_dims) + except Exception as exc: + print(f" Existing output is not reusable ({output_file.name}): {exc}", flush=True) + return False + + +def remap_prediction_collections_to_original_files( + collections: List[Path], + original_input_dir: Path, + output_dir: Path, + tolerance: float = 0.1, + num_threads: int = 4, + retile_buffer: float = 2.0, + target_dims: Optional[Set[str]] = None, + chunk_size: int = 5_000_000, + num_spatial_chunks: Optional[int] = None, + prefer_copc_sources: bool = True, +) -> None: + """Remap finalized prediction collections onto original files.""" + print(f"\n{'=' * 60}", flush=True) + print("Remapping prediction collections to original input files", flush=True) + print(f"{'=' * 60}", flush=True) + + if not collections: + raise ValueError("At least one prediction collection is required") + original_files = ( + point_cloud_files(original_input_dir) + if prefer_copc_sources + else raw_point_cloud_files(original_input_dir) + ) + if not original_files: + raise ValueError(f"No LAZ/LAS files found in original input dir: {original_input_dir}") + if not prefer_copc_sources: + print(" Raw-original mode: ignoring COPC twins in original input dir", flush=True) + + collection_meta = scan_prediction_collection_metadata(collections, target_dims=target_dims) + for coll_meta in collection_meta: + print( + f" Collection {coll_meta['path']}: {', '.join(coll_meta['dims'])}", + flush=True, + ) + + output_dir.mkdir(parents=True, exist_ok=True) + spatial_buffer = max(tolerance * 2, 1.0) + retile_buffer + expected_prediction_dims = [ + dim_name + for coll_meta in collection_meta + for dim_name in coll_meta["dims"] + ] + files_to_process = [] + skipped = 0 + stale = 0 + for input_file in original_files: + output_name = input_file.name.replace(".copc.laz", ".laz") + output_file = output_dir / output_name + if output_file.exists(): + if _existing_output_is_reusable(input_file, output_file, expected_prediction_dims): + skipped += 1 + continue + stale += 1 + try: + output_file.unlink() + except OSError as exc: + raise RuntimeError(f"Could not replace stale output {output_file}: {exc}") from exc + files_to_process.append((input_file, output_file)) + + if skipped: + print(f" Skipping {skipped} already processed files", flush=True) + if stale: + print(f" Reprocessing {stale} stale existing output file(s)", flush=True) + if not files_to_process: + print(" All original files already processed", flush=True) + return + + def process_one(args): + input_file, output_file = args + try: + with laspy.open(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + n_points = reader.header.point_count + + if input_file.name.lower().endswith(".copc.laz"): + spatial_chunks = num_spatial_chunks or max(1, num_threads) + print( + f" COPC original fast path: {input_file.name} " + f"with {spatial_chunks} spatial query window(s)", + flush=True, + ) + stage_points, matched_total = stream_add_collections_to_copc_file_spatial( + input_file, + output_file, + collection_meta, + spatial_buffer, + tolerance, + num_spatial_chunks=spatial_chunks, + kdtree_workers=query_workers, + ) + else: + stage_points, matched_total = stream_add_collections_to_file( + input_file, + output_file, + collection_meta, + spatial_buffer, + tolerance, + chunk_size=chunk_size, + kdtree_workers=query_workers, + ) + if stage_points != n_points: + raise ValueError( + f"Streaming remap wrote {stage_points:,}/{n_points:,} points " + f"for {input_file.name}" + ) + + gc.collect() + return (input_file.name, n_points, matched_total, True, "Success") + except Exception as exc: + return (input_file.name, 0, 0, False, str(exc)) + + total_points = 0 + total_matches = 0 + parallel_workers = min(max(1, num_threads), len(files_to_process)) + query_workers = kdtree_query_workers(num_threads, parallel_workers) + print( + f" Processing {len(files_to_process)} original files with {parallel_workers} worker(s); " + f"{query_workers} KDTree query worker(s) each", + flush=True, + ) + if parallel_workers > 1: + with ThreadPoolExecutor(max_workers=parallel_workers) as executor: + results = list(executor.map(process_one, files_to_process)) + else: + results = [process_one(args) for args in files_to_process] + + failures = [] + for idx, (filename, n_points, matched_total, success, message) in enumerate(results, start=1): + if success: + expected = n_points * len(collection_meta) + match_pct = (matched_total / expected * 100) if expected else 0.0 + print( + f" [{idx}/{len(results)}] {filename}: " + f"{matched_total:,}/{expected:,} collection-point matches ({match_pct:.1f}%)", + flush=True, + ) + total_points += expected + total_matches += matched_total + else: + failures.append(f"{filename}: {message}") + print(f" [{idx}/{len(results)}] FAILED {filename}: {message}", flush=True) + + if failures: + raise RuntimeError("Multi-collection remap failed:\n " + "\n ".join(failures)) + + overall = (total_matches / total_points * 100) if total_points else 0.0 + print( + f"\n ✓ Multi-collection remap complete: " + f"{total_matches:,}/{total_points:,} matches ({overall:.1f}%)", + flush=True, + ) diff --git a/src/prepare_tile_jobs.py b/src/prepare_tile_jobs.py index 0a1c4ac..94eb1f6 100755 --- a/src/prepare_tile_jobs.py +++ b/src/prepare_tile_jobs.py @@ -24,7 +24,7 @@ raise -DEFAULT_BOUNDS_JSON = Path("/home/kg281/data/output/pdal_experiments/tile_bounds_tindex.json") +DEFAULT_BOUNDS_JSON = Path("tile_bounds_tindex.json") def run_get_bounds(tindex_path: Path, tile_length: float, tile_buffer: float, bounds_json_path: Path) -> dict: @@ -38,7 +38,7 @@ def run_get_bounds(tindex_path: Path, tile_length: float, tile_buffer: float, bo ] print(f"[prepare_tile_jobs] running: {' '.join(cmd)}", file=sys.stderr) completed = subprocess.run(cmd, capture_output=True, text=True, check=False) - + if completed.returncode != 0: # Print the full error message if completed.stderr: @@ -52,7 +52,7 @@ def run_get_bounds(tindex_path: Path, tile_length: float, tile_buffer: float, bo f"stderr: {completed.stderr}\n" f"stdout: {completed.stdout}" ) - + env = {} for line in completed.stdout.splitlines(): if "=" in line: @@ -68,7 +68,7 @@ def write_job_list(bounds_json: Path, job_file: Path) -> None: # Get SRS from tile bounds data # get_bounds_from_tindex.py uses 'proj_srs' for the working projection srs = data.get("proj_srs", data.get("tindex_srs", "missing")) - + transformer = None if srs != "missing": try: @@ -80,7 +80,7 @@ def to_geo_bounds(bx: List[float], by: List[float]) -> Tuple[List[float], List[f if transformer is None: # Fallback: if no transformer, return original bounds (planar units) return bx, by - + corners = [ transformer.transform(bx[0], by[0]), transformer.transform(bx[0], by[1]), @@ -109,10 +109,16 @@ def main(): parser.add_argument("tindex_path", type=Path, help="Path to the tindex shapefile") parser.add_argument("--tile-length", type=float, default=40.0) parser.add_argument("--tile-buffer", type=float, default=5.0) + parser.add_argument( + "--grid-offset", + type=float, + default=1.0, + help="Accepted for compatibility with tile_tindex.py; tiling remains data-aligned.", + ) parser.add_argument( "--jobs-out", type=Path, - default=Path("/home/kg281/data/output/pdal_experiments/tile_jobs.txt"), + default=Path("tile_jobs.txt"), ) parser.add_argument( "--bounds-out", diff --git a/src/run.py b/src/run.py old mode 100644 new mode 100755 index 7c8674b..201de98 --- a/src/run.py +++ b/src/run.py @@ -1,88 +1,921 @@ #!/usr/bin/env python3 """ -CLI orchestrator for the 3DTrees smart tile pipeline. +Main orchestrator script for the 3DTrees smart tiling pipeline. -This file intentionally stays thin: it parses CLI arguments, builds task -dependencies, and dispatches to the dedicated task modules. -""" +Routes to appropriate task modules based on --task parameter: +- tile: XYZ reduction, COPC conversion, tiling, and subsampling (1cm and 10cm) +- merge: Remap predictions and merge tiles with instance matching +- filter: Remove duplicate buffer-zone instances from segmented/remapped tiles +- remap: Remap merged file dimensions to original input files +- create_merged_file: Create prod-merged files from original_with_predictions -from __future__ import annotations +Usage: + python src/run.py --task tile --input-dir /path/to/input --output-dir /path/to/output + python src/run.py --task merge --subsampled-10cm-folder /path/to/10cm --original-input-dir /path/to/input + python src/run.py --task filter --input-dir /path/to/segmented_remapped --output-dir /path/to/filtered + python src/run.py --task remap --merged-laz /path/to/merged.laz --original-laz-input-dir /path/to/originals --original-laz-output-dir /path/to/output + python src/run.py --task create_merged_file --original-with-predictions-dir /path/to/original_with_predictions --output-dir /path/to/output +""" -import argparse import sys +import argparse from pathlib import Path +# Add src directory to path for imports when run from project root +_src_dir = Path(__file__).parent.resolve() +if str(_src_dir) not in sys.path: + sys.path.insert(0, str(_src_dir)) -_SRC_DIR = Path(__file__).parent.resolve() -if str(_SRC_DIR) not in sys.path: - sys.path.insert(0, str(_SRC_DIR)) - +# Import Pydantic-based parameters try: - from parameters import Parameters, print_params + from parameters import Parameters, print_params, get_tile_params, get_merge_params, get_remap_params except ImportError as e: print(f"Error: Could not import parameters.py: {e}") print("Please install required dependencies: pip install pydantic pydantic-settings") sys.exit(1) -from filter_task import FilterTaskDependencies, run_filter_task as _run_filter_task_impl -from filter_tree_files import update_trees_files_with_global_ids -from remap_task import RemapTaskDependencies, run_remap_task as _run_remap_task_impl -from remap_task_support import ( - _collect_pointcloud_outputs, - _concat_laz_files, - _convert_collection_file_to_copc, - _describe_laz_dimensions, - _dir_has_pointcloud_outputs, - _ensure_collections_copc, - _fuse_aligned_collections_to_copc, - _fuse_collections_by_spatial_chunks_to_copc, - _pointcloud_file_key, - _resolve_remap_target_dims, -) -from tile_task import run_tile_task +def _create_prod_merged_outputs( + original_with_predictions_dir: Path, + output_dir: Path, + params: Parameters, +) -> None: + """Create prod-merged outputs using the shared create_merged_file implementation.""" + if not original_with_predictions_dir.exists(): + print(f" Skipping prod-merged outputs; missing {original_with_predictions_dir}") + return -def run_filter_task(params: Parameters): - """Build dependencies and dispatch the filter task.""" - deps = FilterTaskDependencies( - update_trees_files_with_global_ids=update_trees_files_with_global_ids, - pointcloud_key_fn=_pointcloud_file_key, - convert_collection_file_to_copc=_convert_collection_file_to_copc, - run_remap_task=run_remap_task, + try: + from main_create_merged_file import create_prod_merged_files + except ImportError as e: + print(f"Error: Could not import main_create_merged_file.py: {e}") + sys.exit(1) + + print() + print("=" * 60) + print("Creating Prod-Merged Files") + print("=" * 60) + print(f"Original-with-predictions dir: {original_with_predictions_dir}") + print(f"Output dir: {output_dir}") + if params.staged_copc_dir: + print(f"Staged COPC dir: {params.staged_copc_dir}") + if params.standardization_json: + print(f"Standardization JSON: {params.standardization_json}") + print(f"Selected resolutions: {params.merged_resolutions}") + print(f"Selected output formats: {params.merged_output_formats}") + print("Product subsampling method: nearest-to-centroid") + print() + + outputs = create_prod_merged_files( + original_with_predictions_dir=original_with_predictions_dir, + output_dir=output_dir, + resolution_selector=params.merged_resolutions, + output_format_selector=params.merged_output_formats, + res1=params.resolution_1, + res2=params.resolution_2, + num_spatial_chunks=params.num_spatial_chunks or params.workers, + staged_copc_dir=params.staged_copc_dir, + standardization_json=params.standardization_json, ) - return _run_filter_task_impl(params, deps) + print(" Prod-merged outputs:") + for output in outputs: + print(f" {output}") -def run_remap_task(params: Parameters): - """Build dependencies and dispatch the remap task.""" - deps = RemapTaskDependencies( - concat_laz_files=_concat_laz_files, - describe_laz_dimensions=_describe_laz_dimensions, - dir_has_pointcloud_outputs=_dir_has_pointcloud_outputs, - collect_pointcloud_outputs=_collect_pointcloud_outputs, - resolve_remap_target_dims=_resolve_remap_target_dims, - ensure_collections_copc=_ensure_collections_copc, - fuse_aligned_collections_to_copc=_fuse_aligned_collections_to_copc, - fuse_collections_by_spatial_chunks_to_copc=_fuse_collections_by_spatial_chunks_to_copc, +def _raw_original_output_dir(params: Parameters, raw_input_dir: Path) -> Path: + """Return the output directory for raw uploaded Original-with-predictions files.""" + if params.original_raw_output_dir: + return Path(params.original_raw_output_dir) + if params.output_dir: + return Path(params.output_dir) + return raw_input_dir.parent / "original_with_predictions" + + +def _validate_raw_original_lane( + raw_input_dir: Path, + raw_output_dir: Path, +) -> None: + """Fail early on ambiguous raw-download lane configuration.""" + if not raw_input_dir.exists(): + print(f"Error: LAZ original input directory not found: {raw_input_dir}") + sys.exit(1) + if raw_output_dir.resolve() == raw_input_dir.resolve(): + print( + "Error: --original-laz-output-dir/--output-dir must differ from --original-laz-input-dir." + ) + sys.exit(1) + + +def _validate_copc_original_lane(copc_input_dir: Path) -> None: + """Fail early when the explicit COPC lane does not contain COPC LAZ files.""" + if not copc_input_dir.exists(): + print(f"Error: COPC original input directory not found: {copc_input_dir}") + sys.exit(1) + try: + from point_cloud_metadata import copc_files + except ImportError as e: + print(f"Error: Could not import COPC discovery helpers: {e}") + sys.exit(1) + if not copc_files(copc_input_dir): + print( + f"Error: --original-copc-input-dir must contain *.copc.laz files: {copc_input_dir}" + ) + sys.exit(1) + + +def _validate_copc_laz_source_pairs(copc_input_dir: Path, laz_input_dir: Path) -> None: + """Validate that explicit COPC and LAZ original dirs describe the same sources.""" + try: + import laspy + from point_cloud_metadata import copc_files, point_cloud_source_key, raw_point_cloud_files + except ImportError as e: + print(f"Error: Could not import source-pair validation helpers: {e}") + sys.exit(1) + + copc_by_key = {point_cloud_source_key(path): path for path in copc_files(copc_input_dir)} + laz_by_key = {point_cloud_source_key(path): path for path in raw_point_cloud_files(laz_input_dir)} + if not laz_by_key: + print(f"Error: --original-laz-input-dir must contain non-COPC LAZ/LAS files: {laz_input_dir}") + sys.exit(1) + missing = sorted(set(laz_by_key) - set(copc_by_key)) + if missing: + print( + "Error: --original-copc-input-dir is missing COPC twins for uploaded LAZ/LAS sources: " + + ", ".join(missing) + ) + sys.exit(1) + + for key, laz_path in laz_by_key.items(): + copc_path = copc_by_key[key] + with laspy.open(str(laz_path), laz_backend=laspy.LazBackend.LazrsParallel) as laz_reader: + laz_header = laz_reader.header + laz_bounds = ( + float(laz_header.x_min), + float(laz_header.x_max), + float(laz_header.y_min), + float(laz_header.y_max), + float(laz_header.z_min), + float(laz_header.z_max), + ) + laz_count = int(laz_header.point_count) + with laspy.open(str(copc_path), laz_backend=laspy.LazBackend.LazrsParallel) as copc_reader: + copc_header = copc_reader.header + copc_bounds = ( + float(copc_header.x_min), + float(copc_header.x_max), + float(copc_header.y_min), + float(copc_header.y_max), + float(copc_header.z_min), + float(copc_header.z_max), + ) + copc_count = int(copc_header.point_count) + if laz_count != copc_count: + print( + f"Error: COPC/LAZ source pair point-count mismatch for {key}: " + f"{copc_path.name} has {copc_count:,}, {laz_path.name} has {laz_count:,}" + ) + sys.exit(1) + if any(abs(a - b) > 0.02 for a, b in zip(laz_bounds, copc_bounds)): + print( + f"Error: COPC/LAZ source pair bounds mismatch for {key}: " + f"{copc_path.name} {copc_bounds} vs {laz_path.name} {laz_bounds}" + ) + sys.exit(1) + + print( + f" COPC/LAZ source validation: matched {len(laz_by_key)} source pair(s)", + flush=True, + ) + + +def run_tile_task(params: Parameters): + """ + Run the tile task: COPC conversion, tiling, and subsampling. + + Pipeline: + 1. Convert LAZ/LAS inputs to intermediate COPC with standard LAS dimensions by default + 2. Build spatial index + 3. Calculate tile bounds + 4. Create overlapping tiles + 5. Subsample to resolution 1 (1cm by default) + 6. Subsample to resolution 2 (10cm) + """ + # Import Python modules + try: + from main_tile import run_tiling_pipeline + from main_subsample import run_subsample_pipeline + except ImportError as e: + print(f"Error: Could not import required modules: {e}") + print("Make sure main_tile.py and main_subsample.py exist.") + sys.exit(1) + + # Required arguments + if not params.input_dir: + print("Error: --input-dir is required for tile task") + sys.exit(1) + if not params.output_dir: + print("Error: --output-dir is required for tile task") + sys.exit(1) + + # Validate input directory + input_dir = Path(params.input_dir) + output_dir = Path(params.output_dir) + + if not input_dir.exists(): + print(f"Error: Input directory does not exist: {input_dir}") + sys.exit(1) + + # Get parameters from Pydantic model + tile_length = params.tile_length + tile_buffer = params.tile_buffer + threads = params.threads + workers = params.workers + # Coerce to bool so CLI/env string "True"/"true"/"1" is respected (Pydantic usually does this; be explicit) + skip_dimension_reduction = bool( + params.skip_dimension_reduction if isinstance(params.skip_dimension_reduction, bool) + else str(params.skip_dimension_reduction).strip().lower() in ("true", "1", "yes") ) - return _run_remap_task_impl(params, deps) + num_spatial_chunks = params.num_spatial_chunks + subsampling_chunks = num_spatial_chunks or workers + res1 = params.resolution_1 + res2 = params.resolution_2 + output_copc_res1 = params.output_copc_res1 + output_copc_res2 = params.output_copc_res2 + subsampling_method = params.subsampling_method + tiling_threshold = params.tiling_threshold + chunk_size = params.chunk_size + print("=" * 60) + print("Running Tile Task (Python Pipeline)") + print("=" * 60) + print(f"Input directory: {input_dir}") + print(f"Output directory: {output_dir}") + print(f"Tile length: {tile_length}m") + print(f"Tile buffer: {tile_buffer}m") + print(f"Workers: {workers}") + print(f"Threads per writer: {threads}") + print(f"Subsampling spatial chunks/window workers: {subsampling_chunks}") + print(f"Keep extra dimensions in LAZ intermediates: {skip_dimension_reduction}") + dimension_reduction = not skip_dimension_reduction + print(f"Subsampling dimensions: {'minimal (standard dims only)' if dimension_reduction else 'keep all (including extra_dims)'}") + print(f"Subsampling method: {subsampling_method}") + print(f"Resolutions: {res1}m ({int(res1*100)}cm), {res2}m ({int(res2*100)}cm)") + print(f"Resolution 1 output: {'COPC LAZ' if output_copc_res1 else 'LAZ'}") + print(f"Resolution 2 output: {'COPC LAZ' if output_copc_res2 else 'LAZ'}") + if tiling_threshold is not None: + print(f"Tiling threshold: {tiling_threshold} MB") + print(f"Chunk size: {chunk_size:,} points") + print() + + try: + # Step 1-4: Tiling pipeline (dimension_reduction not used in tiling; only in subsampling below) + tiles_dir = run_tiling_pipeline( + input_dir=input_dir, + output_dir=output_dir, + tile_length=tile_length, + tile_buffer=tile_buffer, + num_workers=workers, + threads=threads, + max_tile_procs=workers, + dimension_reduction=dimension_reduction, + tiling_threshold=tiling_threshold, + chunk_size=chunk_size, + ) + + # Check if tiling was skipped (returns copc_dir instead of tiles_dir) + tiling_skipped = tiles_dir.name.startswith("copc_") + + if tiling_skipped: + # Single file case - create tiles_* directory structure for consistency + # Move COPC files to tiles_* directory so subsampling creates consistent structure + tiles_dir_normalized = output_dir / f"tiles_{int(tile_length)}m" + tiles_dir_normalized.mkdir(exist_ok=True) + + # Copy/move COPC files to tiles directory + import shutil + for copc_file in tiles_dir.glob("*.copc.laz"): + dest_file = tiles_dir_normalized / copc_file.name + if not dest_file.exists(): + try: + shutil.copy2(copc_file, dest_file) + except OSError as exc: + print( + " Warning: metadata-preserving copy failed " + f"({exc}); retrying as data-only copy" + ) + shutil.copyfile(copc_file, dest_file) + + # Update tiles_dir to use normalized structure + tiles_dir = tiles_dir_normalized + output_prefix = f"{output_dir.name}_{int(tile_length)}m" + print(f" Note: Tiling was skipped, using normalized directory structure: {tiles_dir}") + else: + # Normal tiled case + output_prefix = f"{output_dir.name}_{int(tile_length)}m" + + # Step 5-6: Subsampling pipeline + res1_dir, res2_dir = run_subsample_pipeline( + tiles_dir=tiles_dir, + res1=res1, + res2=res2, + num_cores=workers, + num_threads=subsampling_chunks, + output_prefix=output_prefix, + output_base_dir=output_dir, # Output directly to output_dir, not under tiles_dir + dimension_reduction=dimension_reduction, # True = minimal (standard dims only); False = keep extra_dims + subsampling_method=subsampling_method, + output_copc_res1=output_copc_res1, + output_copc_res2=output_copc_res2, + ) + + # Step 7: Update tile_bounds_tindex.json with actual bounds from created tiles + # (so remap/merge matching uses file extent instead of nominal grid) + bounds_json = output_dir / "tile_bounds_tindex.json" + if bounds_json.exists(): + from main_tile import update_tile_bounds_json_from_files + num_updated = update_tile_bounds_json_from_files(bounds_json, res1_dir) + if num_updated > 0: + print(f" Updated tile_bounds_tindex.json with bounds from {num_updated} tile(s) in {res1_dir.name}") + + print() + print("=" * 60) + print("Tile Task Complete") + print("=" * 60) + print(f"Tiles: {tiles_dir}") + print(f"Subsampled {int(res1*100)}cm: {res1_dir}") + print(f"Subsampled {int(res2*100)}cm: {res2_dir}") + + # Return the input_dir for use in merge task if needed + return input_dir + + except Exception as e: + print(f"Error: {e}") + import traceback + traceback.print_exc() + sys.exit(1) + + +def run_merge_task(params: Parameters): + """ + Run the merge task: remap predictions and merge tiles. + + Pipeline: + 1. Remap predictions from 10cm to target resolution (via main_remap.py) + 2. Merge tiles with instance matching (via main_merge.py) + 3. Remap to original input files (if original_input_dir provided) + """ + # Import Python modules + try: + from instance_labels import MERGED_OUTPUT_SCALES + from main_remap import remap_all_tiles + from main_merge import run_merge + except ImportError as e: + print(f"Error: Could not import required modules: {e}") + print("Make sure main_remap.py and main_merge.py exist.") + sys.exit(1) + + # Required arguments - need either subsampled_10cm_folder or segmented_remapped_folder + # Note: subsampled_10cm_folder is populated by --subsampled-segmented-folder via alias + if not params.subsampled_10cm_folder and not params.segmented_remapped_folder: + print("Error: --subsampled-segmented-folder (or --subsampled-10cm-folder) or --segmented-remapped-folder is required for merge task") + sys.exit(1) + + # Get parameters from Pydantic model + workers = params.workers + buffer = params.buffer + overlap_threshold = params.overlap_threshold + max_centroid_distance = params.max_centroid_distance + max_volume_for_merge = params.max_volume_for_merge + border_zone_width = params.border_zone_width + min_cluster_size = params.min_cluster_size + retile_buffer = 2.0 # Fixed to 2.0m + + print("=" * 60) + print("Running Merge Task (Python Pipeline)") + print("=" * 60) + + try: + # Step 1: Remap predictions (if subsampled_10cm_folder provided) + segmented_remapped_folder = None + + if params.subsampled_10cm_folder: + subsampled_10cm_dir = Path(params.subsampled_10cm_folder) + + if not subsampled_10cm_dir.exists(): + print(f"Error: Input directory does not exist: {subsampled_10cm_dir}") + sys.exit(1) + + print(f"Input (10cm): {subsampled_10cm_dir}") + print() + + # Derive target folder and output folder + # The resolution folders are now at: tiles_*/subsampled_res1 and tiles_*/subsampled_res2 + # For backward compatibility, also check old naming: subsampled_{resolution}cm + parent_dir = subsampled_10cm_dir.parent + target_folder = params.subsampled_target_folder + + if target_folder is None: + # Try new naming first (subsampled_res1) as default target + target_folder_res1 = parent_dir / "subsampled_res1" + if target_folder_res1.exists(): + target_folder = target_folder_res1 + else: + # Fallback or error + pass + + output_folder = params.output_folder + if output_folder is None: + output_folder = parent_dir / "segmented_remapped" + + if target_folder is None or not target_folder.exists(): + print(f"Error: Target resolution folder does not exist or not specified") + if target_folder: + print(f"Path: {target_folder}") + print(f"Please provide --subsampled-target-folder") + sys.exit(1) + + # Optional: tile_bounds_tindex.json for remap matching (use --tile_bounds_json first) + remap_tile_bounds_json = None + if params.tile_bounds_json and params.tile_bounds_json.exists(): + remap_tile_bounds_json = params.tile_bounds_json + if remap_tile_bounds_json is None: + remap_json_candidates = [ + parent_dir / "tile_bounds_tindex.json", + subsampled_10cm_dir / "tile_bounds_tindex.json", + ] + if params.original_tiles_dir: + remap_json_candidates.insert(0, Path(params.original_tiles_dir) / "tile_bounds_tindex.json") + for p in remap_json_candidates: + if p.exists(): + remap_tile_bounds_json = p + break + + # Remap - source is segmented, target is the configured resolution-1 subsample + segmented_remapped_folder = remap_all_tiles( + source_folder=subsampled_10cm_dir, + target_folder=target_folder, + output_folder=output_folder, + tile_bounds_json=remap_tile_bounds_json, + verbose=bool(params.verbose), + num_workers=workers, + instance_dimension=params.instance_dimension, + output_scales=tuple(MERGED_OUTPUT_SCALES), + ) + + # Step 2: Merge tiles + if params.segmented_remapped_folder: + segmented_remapped_folder = Path(params.segmented_remapped_folder) + + if segmented_remapped_folder is None: + print("Error: No segmented remapped folder available for merge") + sys.exit(1) + + if not segmented_remapped_folder.exists(): + print(f"Error: Segmented folder does not exist: {segmented_remapped_folder}") + sys.exit(1) + + print() + print(f"Segmented folder: {segmented_remapped_folder}") + print(f"Buffer: {buffer}m") + print(f"Overlap threshold: {overlap_threshold}") + print(f"Workers: {workers}") + if params.original_raw_input_dir or params.original_input_dir: + print(f"LAZ original input dir: {params.original_raw_input_dir or params.original_input_dir}") + if params.original_copc_input_dir: + print(f"COPC matching original input dir: {params.original_copc_input_dir}") + print() + + output_merged = params.output_merged_laz + output_tiles_dir = params.output_tiles_folder + original_tiles_dir = params.original_tiles_dir + original_input_dir = params.original_raw_input_dir or params.original_input_dir + if params.original_copc_input_dir and original_input_dir: + _validate_copc_original_lane(Path(params.original_copc_input_dir)) + _validate_copc_laz_source_pairs(Path(params.original_copc_input_dir), Path(original_input_dir)) + elif params.original_copc_input_dir: + print( + "Error: --original-laz-input-dir is required when --original-copc-input-dir " + "is used for merge/remap-to-originals." + ) + sys.exit(1) + + # Auto-derive paths if not provided + parent_dir = segmented_remapped_folder.parent + if output_tiles_dir is None: + # Use segmented folder's parent, but ensure it's writable + # If parent is root or not writable, use segmented folder itself + if parent_dir == Path('/') or not os.access(parent_dir, os.W_OK): + output_tiles_dir = segmented_remapped_folder / "output_tiles" + else: + output_tiles_dir = parent_dir / "output_tiles" + if original_tiles_dir is None: + # Try to find the tiles directory (parent of subsampled folders) + original_tiles_dir = parent_dir + if original_input_dir: + _validate_raw_original_lane( + Path(original_input_dir), + Path(output_tiles_dir).parent / "original_with_predictions", + ) + + # tile_bounds_json is required for merge (no fallback) + tile_bounds_json = params.tile_bounds_json + if tile_bounds_json is None: + raise ValueError( + "Merge task requires --tile_bounds_json /path/to/tile_bounds_tindex.json (e.g. from Tile task output)." + ) + if not tile_bounds_json.exists(): + raise FileNotFoundError( + f"tile_bounds_tindex.json not found: {tile_bounds_json}. Pass a valid --tile_bounds_json path." + ) + + # Parse 3DTrees dimension branding params + threedtrees_dims = [d.strip() for d in params.threedtrees_dims.split(",") if d.strip()] if params.threedtrees_dims else None + threedtrees_suffix = params.threedtrees_suffix + + merged_output = run_merge( + segmented_dir=segmented_remapped_folder, + output_tiles_dir=output_tiles_dir, + original_tiles_dir=original_tiles_dir, + tile_bounds_json=tile_bounds_json, + original_input_dir=original_input_dir, + output_merged=output_merged, + buffer=buffer, + overlap_threshold=overlap_threshold, + max_centroid_distance=max_centroid_distance, + max_volume_for_merge=max_volume_for_merge, + border_zone_width=border_zone_width, + min_cluster_size=min_cluster_size, + num_threads=workers, + enable_matching=not params.disable_matching, + require_overlap=True, + enable_volume_merge=not params.disable_volume_merge, + skip_merged_file=params.skip_merged_file, + verbose=params.verbose, + retile_buffer=retile_buffer, + instance_dimension=params.instance_dimension, + transfer_original_dims_to_merged=False, + threedtrees_dims=threedtrees_dims, + threedtrees_suffix=threedtrees_suffix, + ) + + if original_input_dir and params.transfer_original_dims_to_merged: + original_with_predictions_dir = Path(output_tiles_dir).parent / "original_with_predictions" + product_output_dir = Path(merged_output).parent if merged_output else Path(output_tiles_dir).parent + _create_prod_merged_outputs( + original_with_predictions_dir=original_with_predictions_dir, + output_dir=product_output_dir, + params=params, + ) + elif original_input_dir: + print(" Skipping prod-merged output creation (disabled).") + + print() + print("=" * 60) + print("Merge Task Complete") + print("=" * 60) + print(f"Merged output: {merged_output}") + + except Exception as e: + print(f"Error: {e}") + import traceback + traceback.print_exc() + sys.exit(1) + + +def run_remap_task(params: Parameters): + """ + Run the remap task. + + Supported modes: + - --segmented-folders: finalized prediction collections -> original files folder. + Extra dimensions are preserved as-is and duplicate prediction names fail. + - --merged-laz: one merged LAZ/COPC LAZ file -> original files folder. + 3DTrees dimensions are suffixed during transfer. + """ + try: + from prediction_collection_remap import remap_prediction_collections_to_original_files + from merge_tiles import ( + load_merged_file, + reassign_small_instances_in_dims, + remap_to_original_input_files, + ) + from output_remap import remap_merged_file_to_original_input_files + from point_cloud_outputs import write_loaded_point_cloud + except ImportError as e: + print(f"Error: Could not import required modules: {e}") + sys.exit(1) + + if not params.original_raw_input_dir and not params.original_input_dir: + print("Error: --original-laz-input-dir is required for remap task") + print(" Legacy --original-input-dir is still accepted as a LAZ/LAS source for compatibility.") + sys.exit(1) + if not params.segmented_folders and not params.merged_laz: + print("Error: --segmented-folders or --merged-laz is required for remap task") + sys.exit(1) + + laz_input_dir = Path(params.original_raw_input_dir or params.original_input_dir) + laz_output_dir = _raw_original_output_dir(params, laz_input_dir) + _validate_raw_original_lane(laz_input_dir, laz_output_dir) + + copc_input_dir = Path(params.original_copc_input_dir) if params.original_copc_input_dir else None + using_explicit_copc_dir = params.original_copc_input_dir is not None + if using_explicit_copc_dir: + _validate_copc_original_lane(copc_input_dir) + _validate_copc_laz_source_pairs(copc_input_dir, laz_input_dir) + + workers = max(1, params.workers) + retile_buffer = 2.0 + tolerance = 0.1 + + if params.segmented_folders: + collections = [Path(p.strip()) for p in params.segmented_folders.split(",") if p.strip()] + target_dims = {d.strip() for d in params.remap_dims.split(",") if d.strip()} if params.remap_dims else None + + print("=" * 60) + print("Remap: prediction collections -> original files") + print("=" * 60) + print(f"Collections: {[str(c) for c in collections]}") + if copc_input_dir is not None: + print(f"COPC matching original input dir: {copc_input_dir}") + else: + print("COPC matching original input dir: not provided") + print(f"LAZ original input dir: {laz_input_dir}") + print(f"LAZ output dir: {laz_output_dir}") + print("COPC-original enrichment output: disabled") + print(f"Remap dims: {sorted(target_dims) if target_dims else 'all extra dimensions'}") + print() + + try: + print() + print("=" * 60) + print("Remap: prediction collections -> uploaded LAZ originals") + print("=" * 60) + remap_prediction_collections_to_original_files( + collections, + laz_input_dir, + laz_output_dir, + tolerance=tolerance, + num_threads=workers, + retile_buffer=retile_buffer, + target_dims=target_dims, + chunk_size=params.chunk_size or 5_000_000, + num_spatial_chunks=params.num_spatial_chunks or params.workers, + prefer_copc_sources=False, + ) + except Exception as e: + print(f"Error: {e}") + sys.exit(1) + + if params.transfer_original_dims_to_merged: + product_output_dir = laz_output_dir.parent + _create_prod_merged_outputs( + original_with_predictions_dir=laz_output_dir, + output_dir=product_output_dir, + params=params, + ) + else: + print(" Skipping prod-merged output creation (disabled).") + + print() + print("Remap complete.") + return + + merged_laz = Path(params.merged_laz) + if not merged_laz.exists(): + print(f"Error: Merged file not found: {merged_laz}") + sys.exit(1) + + print("=" * 60) + print("Remap: merged file -> original files") + print("=" * 60) + print(f"Merged file: {merged_laz}") + if copc_input_dir is not None: + print(f"COPC matching original input dir: {copc_input_dir}") + else: + print("COPC matching original input dir: not provided") + print(f"LAZ original input dir: {laz_input_dir}") + print(f"LAZ output dir: {laz_output_dir}") + print() + + # Parse 3DTrees dimension branding params + threedtrees_dims = [d.strip() for d in params.threedtrees_dims.split(",") if d.strip()] if params.threedtrees_dims else None + threedtrees_suffix = params.threedtrees_suffix + + if not params.pre_remap_reassign_instances: + remap_merged_file_to_original_input_files( + merged_laz, + laz_input_dir, + laz_output_dir, + tolerance=tolerance, + num_threads=workers, + retile_buffer=retile_buffer, + threedtrees_dims=threedtrees_dims, + threedtrees_suffix=threedtrees_suffix, + num_spatial_chunks=params.num_spatial_chunks or params.workers, + chunk_size=params.chunk_size or 5_000_000, + prefer_copc_sources=False, + ) + else: + merged_points, merged_extra_dims, merged_extra_dim_params = load_merged_file(merged_laz) + candidate_dims = [d.strip() for d in params.threedtrees_dims.split(",") if d.strip()] + instance_dimension = params.pre_remap_reassign_instance_dimension + if instance_dimension is None: + instance_dimension = next((d for d in candidate_dims if "instance" in d.lower()), None) + if instance_dimension is None: + print("Error: --pre-remap-reassign-instances requires an instance dimension") + sys.exit(1) + + print() + print("Pre-remap small instance reassignment") + print(f" Instance dimension: {instance_dimension}") + print(f" Reassign point-count clusters below: {params.pre_remap_reassign_min_cluster_size}") + print(f" Hull check for clusters below: {params.pre_remap_reassign_hull_point_threshold}") + print(f" Reassign hull volume below: {params.pre_remap_reassign_max_volume} m3") + reassignment_stats = reassign_small_instances_in_dims( + merged_points, + merged_extra_dims, + instance_dimension=instance_dimension, + min_cluster_size=params.pre_remap_reassign_min_cluster_size, + hull_point_threshold=params.pre_remap_reassign_hull_point_threshold, + max_volume_for_merge=params.pre_remap_reassign_max_volume, + max_search_radius=float("inf"), + num_threads=workers, + verbose=bool(params.verbose), + ) + print( + " Reassignment result: " + f"{reassignment_stats['changed_points']:,} points changed, " + f"{reassignment_stats['instances_before']:,} -> " + f"{reassignment_stats['instances_after']:,} instances" + ) + if params.pre_remap_reassigned_laz: + reassigned_laz = Path(params.pre_remap_reassigned_laz) + print(f" Saving reassigned segmented point cloud: {reassigned_laz}") + write_loaded_point_cloud( + source_file=merged_laz, + output_file=reassigned_laz, + points=merged_points, + all_dims=merged_extra_dims, + ) + + remap_to_original_input_files( + merged_points, + merged_extra_dims, + merged_extra_dim_params, + laz_input_dir, + laz_output_dir, + tolerance=tolerance, + num_threads=workers, + retile_buffer=retile_buffer, + threedtrees_dims=threedtrees_dims, + threedtrees_suffix=threedtrees_suffix, + num_spatial_chunks=params.num_spatial_chunks or params.workers, + prefer_copc_sources=False, + ) + + # Create prod-merged files from the enriched original outputs (optional). + if params.transfer_original_dims_to_merged: + product_output_dir = laz_output_dir.parent + _create_prod_merged_outputs( + original_with_predictions_dir=laz_output_dir, + output_dir=product_output_dir, + params=params, + ) + else: + print(" Skipping prod-merged output creation (disabled).") + + print() + print("Remap complete.") + + +def run_create_merged_file_task(params: Parameters): + """Create prod-merged files from Original-with-predictions files.""" + try: + from main_create_merged_file import create_prod_merged_files + except ImportError as e: + print(f"Error: Could not import main_create_merged_file.py: {e}") + sys.exit(1) + + input_dir = params.original_with_predictions_dir or params.input_dir + if input_dir is None: + print("Error: --original-with-predictions-dir or --input-dir is required for create_merged_file task") + sys.exit(1) + + output_dir = params.output_dir + if output_dir is None: + output_dir = Path(input_dir).parent + else: + output_dir = Path(output_dir) + + print("=" * 60) + print("Create Prod-Merged Files") + print("=" * 60) + print(f"Original-with-predictions dir: {input_dir}") + print(f"Output dir: {output_dir}") + if params.staged_copc_dir: + print(f"Staged COPC dir: {params.staged_copc_dir}") + if params.standardization_json: + print(f"Standardization JSON: {params.standardization_json}") + print(f"Selected resolutions: {params.merged_resolutions}") + print(f"Selected output formats: {params.merged_output_formats}") + print(f"Resolution 1: {params.resolution_1:g}m") + print(f"Resolution 2: {params.resolution_2:g}m") + print("Product subsampling method: nearest-to-centroid") + print() + + try: + outputs = create_prod_merged_files( + original_with_predictions_dir=Path(input_dir), + output_dir=output_dir, + resolution_selector=params.merged_resolutions, + output_format_selector=params.merged_output_formats, + res1=params.resolution_1, + res2=params.resolution_2, + num_spatial_chunks=params.num_spatial_chunks or params.workers, + staged_copc_dir=params.staged_copc_dir, + standardization_json=params.standardization_json, + ) + except Exception as e: + print(f"Error: {e}") + import traceback + traceback.print_exc() + sys.exit(1) + + print() + print("Create merged file complete.") + for output in outputs: + print(f" {output}") + + +def run_filter_task(params: Parameters): + """Filter buffer-zone duplicate instances from segmented/remapped tiles.""" + try: + from filter_buffer_instances import filter_buffer_instances_dir + except ImportError as e: + print(f"Error: Could not import filter_buffer_instances.py: {e}") + sys.exit(1) + + if not params.input_dir: + print("Error: --input-dir is required for filter task") + sys.exit(1) + if not params.output_dir: + print("Error: --output-dir is required for filter task") + sys.exit(1) + + input_dir = Path(params.input_dir) + output_dir = Path(params.output_dir) + if not input_dir.exists(): + print(f"Error: Input directory does not exist: {input_dir}") + sys.exit(1) + if output_dir.resolve() == input_dir.resolve() and params.filter_suffix == "": + print("Error: --output-dir must differ from --input-dir when --filter-suffix is empty") + sys.exit(1) + + print("=" * 60) + print("Running Filter Task") + print("=" * 60) + print(f"Input directory: {input_dir}") + print(f"Output directory: {output_dir}") + print(f"Buffer: {params.buffer}m") + print(f"Instance dimension: {params.instance_dimension}") + print(f"Output suffix: {params.filter_suffix!r}") + if params.filter_output_extension: + print(f"Output extension: {params.filter_output_extension}") + print() + + try: + summary = filter_buffer_instances_dir( + input_dir=input_dir, + output_dir=output_dir, + buffer=params.buffer, + suffix=params.filter_suffix, + instance_dimension=params.instance_dimension, + output_extension=params.filter_output_extension, + ) + except Exception as e: + print(f"Error: {e}") + sys.exit(1) + + print() + print("Filter task complete.") + print(f" Files processed: {summary['input_files']}") + print(f" Output files: {len(summary['output_files'])}") def preprocess_boolean_flags(args_list): """ - Convert bare boolean flags into explicit True/False values for Pydantic. + Preprocess CLI args to convert boolean flags to explicit True/False for Pydantic. + Pydantic expects --flag True/False, but we want --flag to work like argparse. """ boolean_flags = [ - "--show-params", "--show_params", - "--dimension-reduction", "--dimension_reduction", - "--skip-dimension-reduction", "--skip_dimension_reduction", - "--output-copc-res1", "--output_copc_res1", - "--output-copc-res2", "--output_copc_res2", - "--chunkwise-copc-source-creation", "--chunkwise_copc_source_creation", - "--enable-volume-merge", "--enable_volume_merge", - "--produce-merged-file", "--produce_merged_file", - "--transfer-original-dims-to-merged", "--transfer_original_dims_to_merged", - "--remap-merge", "--remap_merge", + '--show-params', '--show_params', + '--skip-dimension-reduction', '--skip_dimension_reduction', + '--disable-matching', '--disable_matching', + '--disable-volume-merge', '--disable_volume_merge', + '--pre-remap-reassign-instances', '--pre_remap_reassign_instances', + '--output-copc-res1', '--output_copc_res1', + '--output-copc-res2', '--output_copc_res2', + '--skip-merged-file', '--skip_merged_file', + '--verbose', '-v' ] processed = [] @@ -90,11 +923,13 @@ def preprocess_boolean_flags(args_list): while i < len(args_list): arg = args_list[i] if arg in boolean_flags: - if i + 1 < len(args_list) and args_list[i + 1] in ["True", "False"]: + # Check if next arg is already True/False + if i + 1 < len(args_list) and args_list[i + 1].lower() in ['true', 'false']: processed.extend([arg, args_list[i + 1]]) i += 2 else: - processed.extend([arg, "True"]) + # Add explicit True for boolean flag + processed.extend([arg, 'True']) i += 1 else: processed.append(arg) @@ -102,44 +937,120 @@ def preprocess_boolean_flags(args_list): return processed +def _accepted_cli_flags() -> set[str]: + """Return long CLI flags accepted by Parameters or the run.py preprocessor.""" + accepted = {"show-params", "show_params", "no-transfer-original-dims-to-merged"} + for field_name, field in Parameters.model_fields.items(): + accepted.add(field_name) + accepted.add(field_name.replace("_", "-")) + validation_alias = field.validation_alias + if validation_alias is None: + continue + choices = getattr(validation_alias, "choices", None) + if choices is None: + accepted.add(str(validation_alias)) + else: + accepted.update(str(choice) for choice in choices) + return accepted + + +def _unknown_cli_flags(args_list) -> list[str]: + """Return unknown long CLI flags from args_list.""" + accepted = _accepted_cli_flags() + unknown = [] + for raw_arg in args_list: + if not raw_arg.startswith("--") or raw_arg == "--": + continue + flag = raw_arg[2:].split("=", 1)[0] + if flag and flag not in accepted: + unknown.append(flag) + return sorted(set(unknown)) + + +def _validate_known_cli_flags(args_list) -> None: + """Fail fast on typoed or unsupported long CLI flags.""" + unknown = _unknown_cli_flags(args_list) + if unknown: + print( + "Error: Unknown SmartTile CLI argument(s): " + + ", ".join(f"--{flag}" for flag in unknown) + ) + sys.exit(1) + + def main(): + # Handle --show-params flag first using argparse (before Pydantic parsing) + # This avoids Pydantic's boolean flag parsing issues pre_parser = argparse.ArgumentParser(add_help=False) - pre_parser.add_argument("--show-params", "--show_params", action="store_true") + pre_parser.add_argument('--show-params', '--show_params', action='store_true') pre_args, remaining_args = pre_parser.parse_known_args() + # If --show-params was found, add it back to remaining_args for Pydantic + if pre_args.show_params: + remaining_args = ['--show-params'] + remaining_args + + # Manually map aliases that Pydantic might not generate flags for + # --subsampled-segmented-folder -> --subsampled-10cm-folder + # --no-transfer-original-dims-to-merged -> --transfer-original-dims-to-merged False + mapped_args = [] + for arg in remaining_args: + if arg == '--subsampled-segmented-folder': + mapped_args.append('--subsampled-10cm-folder') + elif arg == '--no-transfer-original-dims-to-merged': + mapped_args.extend(['--transfer-original-dims-to-merged', 'False']) + else: + mapped_args.append(arg) + remaining_args = mapped_args + + _validate_known_cli_flags(remaining_args) + + # Preprocess boolean flags for Pydantic processed_args = [sys.argv[0]] + preprocess_boolean_flags(remaining_args) + + # Temporarily replace sys.argv for Pydantic parsing original_argv = sys.argv sys.argv = processed_args + + # Parse parameters using Pydantic (handles CLI automatically) try: params = Parameters() except Exception as e: print(f"Error parsing parameters: {e}") sys.exit(1) finally: + # Restore original argv sys.argv = original_argv + # Show parameters if requested (flag handled by pre-parser; not in Parameters) if pre_args.show_params: print_params(params) sys.exit(0) + # Task is required if not showing params if not params.task: print("Error: --task is required (unless using --show-params)") - print(" python run.py --task tile --input-dir /path/to/input --output-dir /path/to/output") - print(" python run.py --task filter --segmented-folders /path/to/tiles --tile-bounds-json /path/to/tindex.json --output-dir /path/to/output") - print(" python run.py --task remap --segmented-folders /path/to/tiles --original-input-dir /path/to/originals --output-dir /path/to/output") - print(" python run.py --task filter --segmented-folders /path/to/tiles --tile-bounds-json /path/to/tindex.json --remap-merge --original-input-dir /path/to/originals --output-dir /path/to/output") + print("Usage: python run.py --task tile --input-dir /path/to/input --output-dir /path/to/output") + print(" python run.py --task merge --subsampled-10cm-folder /path/to/10cm") + print(" python run.py --task filter --input-dir /path/to/segmented_remapped --output-dir /path/to/filtered") + print(" python run.py --task remap --merged-laz /path/to/merged.laz --original-input-dir /path/to/originals") + print(" python run.py --task create_merged_file --original-with-predictions-dir /path/to/original_with_predictions --output-dir /path/to/output") print(" python run.py --show-params") sys.exit(1) + # Route to appropriate task function if params.task == "tile": run_tile_task(params) - elif params.task == "remap": - run_remap_task(params) + elif params.task == "merge": + run_merge_task(params) elif params.task == "filter": run_filter_task(params) + elif params.task == "remap": + run_remap_task(params) + elif params.task == "create_merged_file": + run_create_merged_file_task(params) else: print(f"Error: Unknown task: {params.task}") - print("Valid tasks: tile, filter, remap") + print("Valid tasks: tile, merge, filter, remap, create_merged_file") sys.exit(1) diff --git a/src/subsample_chunk_worker.py b/src/subsample_chunk_worker.py new file mode 100644 index 0000000..e5c6163 --- /dev/null +++ b/src/subsample_chunk_worker.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +"""PDAL-backed spatial chunk worker for SmartTile subsampling.""" + +from __future__ import annotations + +import json +import re +import shutil +import subprocess +from pathlib import Path +from typing import Optional, Tuple + +from subsample_com import center_of_mass_subsample_las +from subsample_methods import ( + SUBSAMPLING_METHOD_CENTER_OF_MASS, + is_copc_file, + normalize_subsampling_method, + voxel_subsampling_filter, +) + + +def get_pdal_path() -> str: + """Return the PDAL executable path.""" + pdal_path = shutil.which("pdal") + return pdal_path if pdal_path else "pdal" + + +def crop_input_to_laz( + input_file: Path, + crop_file: Path, + bounds_str: str, + dimension_reduction: bool, +) -> bool: + """Crop a chunk with PDAL before SmartTile center-of-mass aggregation.""" + is_copc = is_copc_file(input_file) + reader_type = "readers.copc" if is_copc else "readers.las" + writer_opts = { + "type": "writers.las", + "filename": str(crop_file), + "compression": True, + "forward": "all", + } + if dimension_reduction: + writer_opts["minor_version"] = 2 + writer_opts["dataformat_id"] = 0 + else: + writer_opts["minor_version"] = 4 + writer_opts["extra_dims"] = "all" + + if is_copc: + stages = [{"type": reader_type, "filename": str(input_file), "bounds": bounds_str}] + else: + stages = [ + {"type": reader_type, "filename": str(input_file)}, + {"type": "filters.crop", "bounds": bounds_str}, + ] + stages.append(writer_opts) + + pipeline_file = crop_file.parent / f"_{crop_file.stem}_crop.json" + with open(pipeline_file, "w") as f: + json.dump({"pipeline": stages}, f, indent=2) + try: + result = subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + finally: + if pipeline_file.exists(): + pipeline_file.unlink() + + return result.returncode == 0 and crop_file.exists() and crop_file.stat().st_size > 0 + + +def _chunk_writer_options(chunk_file: Path, dimension_reduction: bool) -> dict: + writer_opts = { + "type": "writers.las", + "filename": str(chunk_file), + "compression": True, + "forward": "all", + } + if dimension_reduction: + writer_opts["minor_version"] = 2 + writer_opts["dataformat_id"] = 0 + else: + writer_opts["minor_version"] = 4 + writer_opts["extra_dims"] = "all" + return writer_opts + + +def _voxel_chunk_pipeline( + input_file: Path, + bounds_str: str, + resolution: float, + subsampling_method: str, + writer_opts: dict, + use_copc_bounds: bool, +) -> dict: + reader_type = "readers.copc" if use_copc_bounds else "readers.las" + if use_copc_bounds: + stages = [ + {"type": reader_type, "filename": str(input_file), "bounds": bounds_str}, + voxel_subsampling_filter(resolution, subsampling_method), + writer_opts, + ] + else: + stages = [ + {"type": reader_type, "filename": str(input_file)}, + {"type": "filters.crop", "bounds": bounds_str}, + voxel_subsampling_filter(resolution, subsampling_method), + writer_opts, + ] + return {"pipeline": stages} + + +def _run_pdal_pipeline(pipeline: dict, pipeline_file: Path) -> subprocess.CompletedProcess: + with open(pipeline_file, "w") as f: + json.dump(pipeline, f, indent=2) + try: + return subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + finally: + if pipeline_file.exists(): + pipeline_file.unlink() + + +def _point_count(path: Path) -> int: + try: + info_result = subprocess.run( + [get_pdal_path(), "info", "--metadata", str(path)], + capture_output=True, + text=True, + check=True, + ) + match = re.search(r'"count":\s*(\d+)', info_result.stdout) + return int(match.group(1)) if match else 0 + except Exception: + return 0 + + +def subsample_tile_chunk( + args: Tuple[Path, str, float, Path, int, int, bool, str] +) -> Tuple[Optional[Path], int]: + """Subsample one spatial chunk using COPC bounds where possible.""" + input_file, bounds_str, resolution, output_dir, chunk_idx, total_chunks, dimension_reduction, method = args + subsampling_method = normalize_subsampling_method(method) + + try: + chunk_file = output_dir / f"{input_file.stem}_chunk{chunk_idx}.laz" + + if subsampling_method == SUBSAMPLING_METHOD_CENTER_OF_MASS: + crop_file = output_dir / f"{input_file.stem}_chunk{chunk_idx}_crop.laz" + try: + if not crop_input_to_laz(input_file, crop_file, bounds_str, dimension_reduction): + print(f" ⚠ Chunk {chunk_idx}/{total_chunks}: crop failed for center-of-mass") + return (None, 0) + point_count = center_of_mass_subsample_las( + crop_file, + chunk_file, + resolution, + dimension_reduction=dimension_reduction, + ) + print(f" ✓ Chunk {chunk_idx}/{total_chunks}: {point_count:,} points") + return (chunk_file, point_count) + finally: + if crop_file.exists(): + try: + crop_file.unlink() + except Exception: + pass + + pdal_cmd = get_pdal_path() + writer_opts = _chunk_writer_options(chunk_file, dimension_reduction) + input_is_copc = is_copc_file(input_file) + pipeline = _voxel_chunk_pipeline( + input_file, + bounds_str, + resolution, + subsampling_method, + writer_opts, + use_copc_bounds=input_is_copc, + ) + result = _run_pdal_pipeline(pipeline, output_dir / f"_pipeline_chunk{chunk_idx}.json") + + if result.returncode != 0 and input_is_copc and ("copc" in result.stderr.lower() or "vlr" in result.stderr.lower()): + print(f" ⚠ Chunk {chunk_idx}/{total_chunks}: COPC reader failed, falling back to readers.las") + pipeline = _voxel_chunk_pipeline( + input_file, + bounds_str, + resolution, + subsampling_method, + writer_opts, + use_copc_bounds=False, + ) + result = _run_pdal_pipeline(pipeline, output_dir / f"_pipeline_chunk{chunk_idx}_fallback.json") + if result.returncode != 0: + print(f" ⚠ Chunk {chunk_idx}/{total_chunks} fallback error: {result.stderr[:100]}") + return (None, 0) + elif result.returncode != 0: + print(f" ⚠ Chunk {chunk_idx}/{total_chunks} error: {result.stderr[:100]}") + return (None, 0) + + if not chunk_file.exists() or chunk_file.stat().st_size == 0: + return (None, 0) + + point_count = _point_count(chunk_file) + print(f" ✓ Chunk {chunk_idx}/{total_chunks}: {point_count:,} points") + return (chunk_file, point_count) + except Exception as e: + print(f" ✗ Chunk {chunk_idx}/{total_chunks} failed: {e}") + return (None, 0) diff --git a/src/subsample_com.py b/src/subsample_com.py new file mode 100644 index 0000000..3d8f709 --- /dev/null +++ b/src/subsample_com.py @@ -0,0 +1,379 @@ +#!/usr/bin/env python3 +"""Center-of-mass subsampling strategies for SmartTile.""" + +from __future__ import annotations + +import math +import multiprocessing +from concurrent.futures import ProcessPoolExecutor, as_completed +from pathlib import Path +from typing import Iterable, List, Optional, Tuple + +import laspy +import numpy as np + + +COPC_COM_TARGET_WINDOW_CELLS = 500 +COPC_COM_MAX_WINDOW_SIZE = 20.0 +COPC_COM_DENSE_BIN_LIMIT = 5_000_000 + + +def process_pool_kwargs() -> dict: + """Use a clean worker start method to avoid inherited COPC/lazrs thread state.""" + return {"mp_context": multiprocessing.get_context("spawn")} + + +def _is_stale_copc_vlr(vlr) -> bool: + return getattr(vlr, "user_id", "") == "copc" and getattr(vlr, "record_id", None) in (1, 2) + + +def _is_extra_bytes_vlr(vlr) -> bool: + return getattr(vlr, "user_id", "") == "LASF_Spec" and getattr(vlr, "record_id", None) == 4 + + +def make_center_of_mass_header(source_header: laspy.LasHeader, dimension_reduction: bool) -> laspy.LasHeader: + """Create an output header for center-of-mass subsampling.""" + from laspy.vlrs.vlrlist import VLRList + + if dimension_reduction: + header = laspy.LasHeader(point_format=0, version="1.2") + for attr in ( + "file_source_id", + "global_encoding", + "uuid", + "system_identifier", + "generating_software", + "creation_date", + ): + if hasattr(source_header, attr): + try: + setattr(header, attr, getattr(source_header, attr)) + except Exception: + pass + source_vlrs = source_header.vlrs + else: + header = source_header.copy() + source_vlrs = header.vlrs + + header.offsets = source_header.offsets + header.scales = source_header.scales + header.vlrs = VLRList([ + vlr for vlr in (source_vlrs or []) + if not _is_stale_copc_vlr(vlr) + and (not dimension_reduction or not _is_extra_bytes_vlr(vlr)) + ]) + + source_evlrs = getattr(source_header, "evlrs", None) + if source_evlrs is not None and not dimension_reduction: + header.evlrs = VLRList([vlr for vlr in source_evlrs if not _is_stale_copc_vlr(vlr)]) + + return header + + +def copy_non_xyz_dimensions(source_las: laspy.LasData, output_las: laspy.LasData, indices: np.ndarray) -> None: + """Copy all output-supported non-XYZ dimensions from selected source points.""" + for dim_name in output_las.point_format.dimension_names: + if dim_name in {"X", "Y", "Z"}: + continue + if not hasattr(source_las, dim_name): + continue + try: + setattr(output_las, dim_name, np.asarray(getattr(source_las, dim_name))[indices]) + except Exception: + pass + + +def point_record_to_xyz(points) -> np.ndarray: + coords = np.empty((len(points), 3), dtype=np.float64) + coords[:, 0] = points.x + coords[:, 1] = points.y + coords[:, 2] = points.z + return coords + + +def aggregate_center_of_mass_sparse(coords: np.ndarray, resolution: float) -> np.ndarray: + voxel_keys = np.floor(coords / resolution).astype(np.int64) + _, inverse, counts = np.unique( + voxel_keys, + axis=0, + return_inverse=True, + return_counts=True, + ) + sums = np.zeros((len(counts), 3), dtype=np.float64) + np.add.at(sums, inverse, coords) + return sums / counts[:, None] + + +def aggregate_center_of_mass_bincount(coords: np.ndarray, resolution: float) -> Optional[np.ndarray]: + keys = np.floor(coords / resolution).astype(np.int64) + key_min = keys.min(axis=0) + key_max = keys.max(axis=0) + shape = (key_max - key_min + 1).astype(np.int64) + bins = int(shape[0] * shape[1] * shape[2]) + if bins <= 0 or bins > COPC_COM_DENSE_BIN_LIMIT: + return None + + yz = int(shape[1] * shape[2]) + z = int(shape[2]) + linear = ( + (keys[:, 0] - key_min[0]) * yz + + (keys[:, 1] - key_min[1]) * z + + (keys[:, 2] - key_min[2]) + ) + counts = np.bincount(linear, minlength=bins) + occupied = np.flatnonzero(counts) + sums_x = np.bincount(linear, weights=coords[:, 0], minlength=bins)[occupied] + sums_y = np.bincount(linear, weights=coords[:, 1], minlength=bins)[occupied] + sums_z = np.bincount(linear, weights=coords[:, 2], minlength=bins)[occupied] + occupied_counts = counts[occupied].astype(np.float64) + + centers = np.empty((len(occupied), 3), dtype=np.float64) + centers[:, 0] = sums_x / occupied_counts + centers[:, 1] = sums_y / occupied_counts + centers[:, 2] = sums_z / occupied_counts + return centers + + +def aggregate_center_of_mass_xyz(points, resolution: float) -> np.ndarray: + coords = point_record_to_xyz(points) + if len(coords) == 0: + return np.empty((0, 3), dtype=np.float64) + centers = aggregate_center_of_mass_bincount(coords, resolution) + if centers is not None: + return centers + return aggregate_center_of_mass_sparse(coords, resolution) + + +def aligned_edges(start: float, stop: float, step: float, align: float) -> List[Tuple[float, float]]: + first = math.floor(start / align) * align + final = math.ceil(stop / align) * align + edges = [] + cur = first + while cur < stop: + nxt = min(cur + step, final) + if nxt > start and cur < stop: + edges.append((max(cur, start), min(nxt, stop))) + cur = nxt + return edges + + +def _copc_com_window_size(resolution: float) -> float: + return max(resolution, min(COPC_COM_MAX_WINDOW_SIZE, resolution * COPC_COM_TARGET_WINDOW_CELLS)) + + +def iter_copc_center_of_mass_windows(header: laspy.LasHeader, resolution: float) -> Iterable[object]: + """Yield voxel-aligned, half-open XY bounds for COPC center-of-mass queries.""" + from laspy.copc import Bounds + + step = _copc_com_window_size(resolution) + x_edges = aligned_edges(header.x_min, header.x_max, step, resolution) + y_edges = aligned_edges(header.y_min, header.y_max, step, resolution) + eps_x = float(header.scales[0]) * 0.5 + eps_y = float(header.scales[1]) * 0.5 + + for xi, (xmin, xmax) in enumerate(x_edges): + for yi, (ymin, ymax) in enumerate(y_edges): + qmaxx = xmax if xi == len(x_edges) - 1 else xmax - eps_x + qmaxy = ymax if yi == len(y_edges) - 1 else ymax - eps_y + if qmaxx < xmin or qmaxy < ymin: + continue + yield Bounds( + np.array([xmin, ymin, header.z_min], dtype=np.float64), + np.array([qmaxx, qmaxy, header.z_max], dtype=np.float64), + ) + + +def _copc_center_of_mass_window_worker( + args: Tuple[int, str, Tuple[float, float, float], Tuple[float, float, float], float] +) -> Tuple[int, int, np.ndarray]: + window_idx, input_file, mins, maxs, resolution = args + from laspy.copc import Bounds, CopcReader + + with CopcReader.open(input_file) as reader: + points = reader.query( + Bounds( + np.array(mins, dtype=np.float64), + np.array(maxs, dtype=np.float64), + ) + ) + point_count = len(points) + if point_count == 0: + return (window_idx, 0, np.empty((0, 3), dtype=np.float64)) + return (window_idx, point_count, aggregate_center_of_mass_xyz(points, resolution)) + + +def write_center_of_mass_points(writer, header: laspy.LasHeader, centers: np.ndarray) -> int: + """Write one batch of XYZ center points to an open LAS/LAZ writer.""" + if len(centers) == 0: + return 0 + points = laspy.ScaleAwarePointRecord.zeros(len(centers), header=header) + points.x = centers[:, 0] + points.y = centers[:, 1] + points.z = centers[:, 2] + writer.write_points(points) + return len(centers) + + +def center_of_mass_subsample_copc( + input_file: Path, + output_file: Path, + resolution: float, + num_workers: int = 1, +) -> int: + """Subsample one COPC file by querying voxel-aligned windows and averaging XYZ.""" + from laspy.copc import CopcReader + + if resolution <= 0: + raise ValueError("resolution must be positive") + num_workers = max(1, int(num_workers or 1)) + + pending_centers = {} + total_input_points = 0 + total_output_points = 0 + + with CopcReader.open(input_file) as reader: + header = reader.header + windows = list(iter_copc_center_of_mass_windows(header, resolution)) + + if not windows: + raise ValueError(f"No COPC query windows available for {input_file}") + + output_header = make_center_of_mass_header(header, dimension_reduction=True) + output_file.parent.mkdir(parents=True, exist_ok=True) + + def make_task(window_idx: int): + bounds = windows[window_idx] + return ( + window_idx, + str(input_file), + tuple(float(value) for value in bounds.mins), + tuple(float(value) for value in bounds.maxs), + resolution, + ) + + with laspy.open( + str(output_file), + mode="w", + header=output_header, + do_compress=output_file.suffix.lower() == ".laz", + ) as writer: + if num_workers == 1 or len(windows) <= 1: + with CopcReader.open(input_file) as reader: + for bounds in windows: + points = reader.query(bounds) + if len(points) == 0: + continue + total_input_points += len(points) + centers = aggregate_center_of_mass_xyz(points, resolution) + total_output_points += write_center_of_mass_points(writer, output_header, centers) + else: + worker_count = min(num_workers, len(windows)) + max_in_flight = min(len(windows), worker_count * 2) + print(f" → COPC COM window parallelism: {worker_count} workers") + + with ProcessPoolExecutor(max_workers=worker_count, **process_pool_kwargs()) as executor: + futures = {} + next_submit = 0 + next_write = 0 + + def submit_until_capacity(): + nonlocal next_submit + while next_submit < len(windows) and len(futures) < max_in_flight: + future = executor.submit(_copc_center_of_mass_window_worker, make_task(next_submit)) + futures[future] = next_submit + next_submit += 1 + + submit_until_capacity() + + while futures or next_submit < len(windows): + if not futures: + submit_until_capacity() + for future in as_completed(futures): + futures.pop(future) + window_idx, point_count, centers = future.result() + total_input_points += point_count + pending_centers[window_idx] = centers + + while next_write in pending_centers: + centers_to_write = pending_centers.pop(next_write) + total_output_points += write_center_of_mass_points( + writer, + output_header, + centers_to_write, + ) + next_write += 1 + submit_until_capacity() + break + + while next_write in pending_centers: + centers_to_write = pending_centers.pop(next_write) + total_output_points += write_center_of_mass_points( + writer, + output_header, + centers_to_write, + ) + next_write += 1 + + if total_output_points == 0: + raise ValueError(f"No points available in {input_file}") + + print( + f" COPC COM windows: {len(windows):,}; " + f"input points read: {total_input_points:,}; output points: {total_output_points:,}" + ) + return total_output_points + + +def center_of_mass_subsample_las( + input_file: Path, + output_file: Path, + resolution: float, + dimension_reduction: bool = True, +) -> int: + """Subsample one LAS/LAZ file by averaging XYZ per voxel. + + Non-coordinate attributes are copied from the real point nearest to the averaged XYZ + within each voxel. They are never averaged. + """ + if resolution <= 0: + raise ValueError("resolution must be positive") + + source_las = laspy.read(str(input_file), laz_backend=laspy.LazBackend.LazrsParallel) + point_count = len(source_las.points) + if point_count == 0: + raise ValueError(f"No points available in {input_file}") + + coords = point_record_to_xyz(source_las) + voxel_keys = np.floor(coords / resolution).astype(np.int64) + _, inverse, counts = np.unique( + voxel_keys, + axis=0, + return_inverse=True, + return_counts=True, + ) + + sums = np.zeros((len(counts), 3), dtype=np.float64) + np.add.at(sums, inverse, coords) + means = sums / counts[:, None] + + dist2 = np.sum((coords - means[inverse]) ** 2, axis=1) + order = np.lexsort((dist2, inverse)) + sorted_inverse = inverse[order] + first_in_voxel = np.empty(len(order), dtype=bool) + first_in_voxel[0] = True + first_in_voxel[1:] = sorted_inverse[1:] != sorted_inverse[:-1] + selected_indices = order[first_in_voxel] + selected_voxels = inverse[selected_indices] + mean_coords = means[selected_voxels] + + header = make_center_of_mass_header(source_las.header, dimension_reduction) + output_las = laspy.LasData(header) + output_las.points = laspy.ScaleAwarePointRecord.zeros(len(selected_indices), header=header) + copy_non_xyz_dimensions(source_las, output_las, selected_indices) + output_las.x = mean_coords[:, 0] + output_las.y = mean_coords[:, 1] + output_las.z = mean_coords[:, 2] + + output_file.parent.mkdir(parents=True, exist_ok=True) + output_las.write(str(output_file), do_compress=output_file.suffix.lower() == ".laz") + return len(selected_indices) diff --git a/src/subsample_methods.py b/src/subsample_methods.py new file mode 100644 index 0000000..a7a8793 --- /dev/null +++ b/src/subsample_methods.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Shared SmartTile subsampling method policy.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Optional + + +SUBSAMPLING_METHOD_CENTER_OF_MASS = "center-of-mass" +SUBSAMPLING_METHOD_NEAREST_TO_CENTROID = "nearest-to-centroid" +SUBSAMPLING_METHODS = { + SUBSAMPLING_METHOD_CENTER_OF_MASS, + SUBSAMPLING_METHOD_NEAREST_TO_CENTROID, +} + + +def normalize_subsampling_method(method: Optional[str]) -> str: + """Return a canonical SmartTile subsampling method name.""" + normalized = (method or SUBSAMPLING_METHOD_CENTER_OF_MASS).strip().lower() + if normalized in {"centroid", "voxelcentroidnearestneighbor", "voxel-centroid-nearest-neighbor"}: + return SUBSAMPLING_METHOD_NEAREST_TO_CENTROID + if normalized in {"com", "center_of_mass", "center-of-mass"}: + return SUBSAMPLING_METHOD_CENTER_OF_MASS + if normalized not in SUBSAMPLING_METHODS: + allowed = ", ".join(sorted(SUBSAMPLING_METHODS)) + raise ValueError(f"Unknown subsampling method {method!r}; expected one of: {allowed}") + return normalized + + +def voxel_subsampling_filter(resolution: float, method: str) -> dict: + """Return the PDAL filter stage for methods implemented by PDAL.""" + method = normalize_subsampling_method(method) + if method == SUBSAMPLING_METHOD_NEAREST_TO_CENTROID: + return {"type": "filters.voxelcentroidnearestneighbor", "cell": resolution} + raise ValueError(f"{method} is implemented by SmartTile, not a PDAL filter stage") + + +def is_copc_file(path: Path) -> bool: + return path.name.lower().endswith(".copc.laz") diff --git a/src/subsample_outputs.py b/src/subsample_outputs.py new file mode 100644 index 0000000..1c3fb62 --- /dev/null +++ b/src/subsample_outputs.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""Output-path and COPC-conversion helpers for SmartTile subsampling products.""" + +from __future__ import annotations + +import json +import shutil +import subprocess +from pathlib import Path +from typing import List, Optional + + +def get_pdal_path() -> str: + """Return the PDAL executable path.""" + pdal_path = shutil.which("pdal") + return pdal_path if pdal_path else "pdal" + + +def laz_output_path(output_file: Path) -> Path: + """Normalize an output path to regular compressed LAZ.""" + name = output_file.name + if name.endswith(".copc.laz"): + name = name[: -len(".copc.laz")] + ".laz" + elif not name.endswith(".laz"): + name = output_file.stem + ".laz" + return output_file.parent / name + + +def copc_output_path(output_file: Path) -> Path: + """Normalize an output path to COPC LAZ.""" + name = output_file.name + if name.endswith(".copc.laz"): + return output_file + if name.endswith(".laz"): + name = name[: -len(".laz")] + ".copc.laz" + return output_file.parent / f"{output_file.stem}.copc.laz" + + +def temporary_laz_path(target_output: Path, pipeline_dir: Path) -> Path: + """Return a temporary LAZ path outside chunk directories for COPC conversion.""" + pipeline_dir.mkdir(parents=True, exist_ok=True) + stem = target_output.name + if stem.endswith(".copc.laz"): + stem = stem[: -len(".copc.laz")] + elif stem.endswith(".laz"): + stem = stem[: -len(".laz")] + return pipeline_dir / f"_{stem}_for_copc.laz" + + +def subsample_output_files(output_dir: Path, output_copc: bool) -> List[Path]: + """List subsampling outputs for the selected output format.""" + if output_copc: + return sorted(output_dir.glob("*.copc.laz")) + return sorted(path for path in output_dir.glob("*.laz") if not path.name.endswith(".copc.laz")) + + +def convert_laz_output_to_copc( + input_laz: Path, + output_copc: Path, + source_metadata_file: Optional[Path] = None, + preserve_extra_dims: bool = False, +) -> bool: + """Convert a reduced LAZ to COPC using the tiling converter and CRS checks.""" + output_copc.parent.mkdir(parents=True, exist_ok=True) + if output_copc.exists(): + try: + output_copc.unlink() + except OSError: + pass + + try: + from copc_metadata import ( + append_source_geotiff_projection_evlrs, + copc_preserves_source_crs, + ) + from main_tile import _convert_laz_to_copc + + converted = _convert_laz_to_copc( + input_laz, + output_copc, + preserve_extra_dims=preserve_extra_dims, + ) + if not converted: + return False + if source_metadata_file is not None: + preserved_geotiff, message = append_source_geotiff_projection_evlrs( + source_metadata_file, + output_copc, + ) + if not preserved_geotiff: + print(f" COPC GeoTIFF projection preservation failed: {message}") + return False + valid_crs, message = copc_preserves_source_crs(source_metadata_file, output_copc) + if not valid_crs: + print(f" COPC CRS validation failed: {message}") + return False + return True + except Exception as exc: + print(f" Warning: shared COPC converter unavailable ({exc}); falling back to PDAL") + + writer_opts = { + "type": "writers.copc", + "filename": str(output_copc), + "forward": "all", + } + if preserve_extra_dims: + writer_opts["extra_dims"] = "all" + pipeline = { + "pipeline": [ + {"type": "readers.las", "filename": str(input_laz)}, + writer_opts, + ] + } + pipeline_file = output_copc.parent / f"_{output_copc.stem}_convert_copc.json" + with open(pipeline_file, "w") as f: + json.dump(pipeline, f, indent=2) + try: + result = subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + print(f" COPC conversion failed: {result.stderr[:200]}") + return False + converted = output_copc.exists() and output_copc.stat().st_size > 0 + if converted and source_metadata_file is not None: + try: + from copc_metadata import ( + append_source_geotiff_projection_evlrs, + copc_preserves_source_crs, + ) + + preserved_geotiff, message = append_source_geotiff_projection_evlrs( + source_metadata_file, + output_copc, + ) + if not preserved_geotiff: + print(f" COPC GeoTIFF projection preservation failed: {message}") + return False + valid_crs, message = copc_preserves_source_crs(source_metadata_file, output_copc) + if not valid_crs: + print(f" COPC CRS validation failed: {message}") + return False + except Exception as exc: + print(f" COPC CRS validation unavailable after PDAL conversion: {exc}") + return False + return converted + finally: + if pipeline_file.exists(): + pipeline_file.unlink() diff --git a/src/tile_bounds_graph.py b/src/tile_bounds_graph.py new file mode 100644 index 0000000..c68c6fd --- /dev/null +++ b/src/tile_bounds_graph.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Tile-bound JSON neighbor graph and matching helpers.""" + +from __future__ import annotations + +import json +import math +from pathlib import Path +from typing import Dict, List, Optional, Set, Tuple + + +def build_neighbor_graph_from_bounds_json( + tile_bounds_json: Path, +) -> Tuple[List[Tuple[float, float, float, float]], List[Tuple[float, float]], List[Dict[str, Optional[int]]]]: + """ + Build a neighbor graph from tile_bounds_tindex.json. + + Returns: + json_bounds: list of (minx, maxx, miny, maxy) for each JSON tile + centers: list of (cx, cy) centers for each JSON tile + neighbors_idx: list of dicts {dir -> neighbor_index or None} for each JSON tile + """ + if not tile_bounds_json.exists(): + raise FileNotFoundError(f"tile_bounds_tindex.json not found: {tile_bounds_json}") + + with tile_bounds_json.open() as f: + data = json.load(f) + + tiles = data.get("tiles", []) + if not tiles: + raise ValueError(f"No tiles found in tile bounds JSON: {tile_bounds_json}") + + json_bounds: List[Tuple[float, float, float, float]] = [] + centers: List[Tuple[float, float]] = [] + + for tile in tiles: + bx, by = tile["bounds"] + minx, maxx = float(bx[0]), float(bx[1]) + miny, maxy = float(by[0]), float(by[1]) + json_bounds.append((minx, maxx, miny, maxy)) + centers.append(((minx + maxx) * 0.5, (miny + maxy) * 0.5)) + + n = len(json_bounds) + neighbors_idx: List[Dict[str, Optional[int]]] = [ + {"east": None, "west": None, "north": None, "south": None} for _ in range(n) + ] + + col_row_to_idx: Dict[Tuple[int, int], int] = {} + for i, tile in enumerate(tiles): + if "col" in tile and "row" in tile: + col_row_to_idx[(int(tile["col"]), int(tile["row"]))] = i + + if col_row_to_idx: + for i, tile in enumerate(tiles): + if "col" not in tile or "row" not in tile: + continue + c, r = int(tile["col"]), int(tile["row"]) + neighbors_idx[i]["east"] = col_row_to_idx.get((c + 1, r)) + neighbors_idx[i]["west"] = col_row_to_idx.get((c - 1, r)) + neighbors_idx[i]["north"] = col_row_to_idx.get((c, r + 1)) + neighbors_idx[i]["south"] = col_row_to_idx.get((c, r - 1)) + return json_bounds, centers, neighbors_idx + + for i in range(n): + minx_i, maxx_i, miny_i, maxy_i = json_bounds[i] + cx_i, cy_i = centers[i] + best_east = None + best_west = None + best_north = None + best_south = None + + for j in range(n): + if i == j: + continue + minx_j, maxx_j, miny_j, maxy_j = json_bounds[j] + cx_j, cy_j = centers[j] + overlap_y = not (maxy_i <= miny_j or maxy_j <= miny_i) + overlap_x = not (maxx_i <= minx_j or maxx_j <= minx_i) + + if cx_j > cx_i and overlap_y: + dx = cx_j - cx_i + if best_east is None or dx < best_east[0]: + best_east = (dx, j) + if cx_j < cx_i and overlap_y: + dx = cx_i - cx_j + if best_west is None or dx < best_west[0]: + best_west = (dx, j) + if cy_j > cy_i and overlap_x: + dy = cy_j - cy_i + if best_north is None or dy < best_north[0]: + best_north = (dy, j) + if cy_j < cy_i and overlap_x: + dy = cy_i - cy_j + if best_south is None or dy < best_south[0]: + best_south = (dy, j) + + if best_east is not None: + neighbors_idx[i]["east"] = best_east[1] + if best_west is not None: + neighbors_idx[i]["west"] = best_west[1] + if best_north is not None: + neighbors_idx[i]["north"] = best_north[1] + if best_south is not None: + neighbors_idx[i]["south"] = best_south[1] + + return json_bounds, centers, neighbors_idx + + +def match_tiles_to_json_bounds( + tile_boundaries: Dict[str, Tuple[float, float, float, float]], + json_bounds: List[Tuple[float, float, float, float]], + centers: List[Tuple[float, float]], +) -> Tuple[Dict[str, int], Dict[int, str]]: + """Match loaded tiles to JSON tiles with stepwise bounds/centroid tolerances.""" + tile_items = list(tile_boundaries.items()) + tile_to_json: Dict[str, int] = {} + json_to_tile: Dict[int, str] = {} + used_json: Set[int] = set() + + if len(tile_items) == 1 and len(json_bounds) == 1: + name = tile_items[0][0] + return {name: 0}, {0: name} + + for tol in [0.1, 0.5, 1.0, 2.0, 5.0]: + for name, bounds in tile_items: + if name in tile_to_json: + continue + minx_a, maxx_a, miny_a, maxy_a = bounds + best_j = None + best_l1 = None + for j, (minx_b, maxx_b, miny_b, maxy_b) in enumerate(json_bounds): + if j in used_json: + continue + if ( + abs(minx_a - minx_b) <= tol + and abs(maxx_a - maxx_b) <= tol + and abs(miny_a - miny_b) <= tol + and abs(maxy_a - maxy_b) <= tol + ): + l1 = ( + abs(minx_a - minx_b) + + abs(maxx_a - maxx_b) + + abs(miny_a - miny_b) + + abs(maxy_a - maxy_b) + ) + if best_l1 is None or l1 < best_l1: + best_l1 = l1 + best_j = j + if best_j is not None: + tile_to_json[name] = best_j + json_to_tile[best_j] = name + used_json.add(best_j) + + for name, bounds in tile_items: + if name in tile_to_json: + continue + minx_a, maxx_a, miny_a, maxy_a = bounds + cx_a = (minx_a + maxx_a) * 0.5 + cy_a = (miny_a + maxy_a) * 0.5 + best_j = None + best_dist = None + for j, (cx_b, cy_b) in enumerate(centers): + if j in used_json: + continue + dist = math.hypot(cx_b - cx_a, cy_b - cy_a) + if dist <= tol and (best_dist is None or dist < best_dist): + best_dist = dist + best_j = j + if best_j is not None: + tile_to_json[name] = best_j + json_to_tile[best_j] = name + used_json.add(best_j) + + if len(tile_to_json) == len(tile_boundaries): + break + + if len(tile_to_json) != len(tile_boundaries): + unmatched = sorted(set(tile_boundaries.keys()) - set(tile_to_json.keys())) + raise ValueError( + "Failed to match all tiles to entries in tile_bounds_tindex.json. " + f"Unmatched tiles: {', '.join(unmatched)}" + ) + + return tile_to_json, json_to_tile diff --git a/src/tile_copc.py b/src/tile_copc.py new file mode 100644 index 0000000..66614db --- /dev/null +++ b/src/tile_copc.py @@ -0,0 +1,496 @@ +#!/usr/bin/env python3 +"""COPC finalization and conversion helpers for SmartTile tiling.""" + +from __future__ import annotations + +import json +import shutil +import subprocess +import tempfile +from pathlib import Path +from typing import List, Optional, Tuple + +from copc_metadata import ( + append_source_geotiff_projection_evlrs, + copc_preserves_source_crs, + first_crs_source, + first_srs_assignment, + srs_assignment_from_file, +) + +UNTWINE_STRIP_EXTRA_DIMS_ARG = "" +_UNTWINE_EMPTY_DIMS_UNSUPPORTED = "Missing value for argument 'dims'" +_UNTWINE_EMPTY_DIMS_SUPPORTED: Optional[bool] = None + + +def get_pdal_path() -> str: + """Return the PDAL executable path.""" + pdal_path = shutil.which("pdal") + return pdal_path if pdal_path else "pdal" + + +def _run_untwine( + inputs: List[Path], + output_copc: Path, + srs_arg: Optional[str], + strip_extra_dims: bool = False, +) -> Tuple[bool, str]: + """Run untwine for one or more inputs.""" + global _UNTWINE_EMPTY_DIMS_SUPPORTED + + untwine_cmd = shutil.which("untwine") + if not untwine_cmd: + return (False, "untwine not available") + if ( + strip_extra_dims + and UNTWINE_STRIP_EXTRA_DIMS_ARG == "" + and _UNTWINE_EMPTY_DIMS_SUPPORTED is False + ): + return (False, "untwine empty --dims unsupported") + + input_args = [] + for path in inputs: + input_args.extend(["-i", str(path)]) + # Untwine treats --dims as an extra-dimension keep-list while X/Y/Z and the + # standard LAS fields remain loaded. SmartTile asks for an empty keep-list + # in the default tiling/staging path. Untwine 1.5.1 rejects this form, so + # callers validate the output and fall back to stripped-LAZ staging. + dims_args = ["--dims", UNTWINE_STRIP_EXTRA_DIMS_ARG] if strip_extra_dims else [] + srs_args = ["--a_srs", srs_arg] if srs_arg else [] + try: + result = subprocess.run( + [untwine_cmd] + input_args + ["-o", str(output_copc)] + dims_args + srs_args, + capture_output=True, + text=True, + check=False, + ) + except Exception as exc: + return (False, f"untwine error: {exc}") + if strip_extra_dims and UNTWINE_STRIP_EXTRA_DIMS_ARG == "": + if result.returncode == 0: + _UNTWINE_EMPTY_DIMS_SUPPORTED = True + elif _UNTWINE_EMPTY_DIMS_UNSUPPORTED in result.stderr: + _UNTWINE_EMPTY_DIMS_SUPPORTED = False + + if result.returncode != 0: + return (False, f"untwine failed: {result.stderr[:200]}") + if not output_copc.exists() or output_copc.stat().st_size == 0: + return (False, "untwine produced no output") + return (True, "untwine") + + +def _has_extra_dimensions(path: Path) -> bool: + import laspy + + from copc_metadata import laspy_laz_backend + + with laspy.open(str(path), laz_backend=laspy_laz_backend()) as reader: + return bool(list(reader.header.point_format.extra_dimensions)) + + +def _output_has_no_extra_dimensions(path: Path) -> bool: + try: + return not _has_extra_dimensions(path) + except Exception: + return False + + +def _strip_las_to_standard_dims( + input_files: List[Path], + output_laz: Path, + tile_bounds: Optional[Tuple[float, float, float, float]] = None, +) -> Tuple[bool, str]: + """Write a temporary LAZ with standard LAS dimensions only.""" + writer_opts = { + "type": "writers.las", + "filename": str(output_laz), + "compression": True, + "forward": "all", + "minor_version": 2, + "dataformat_id": 0, + } + if tile_bounds is not None: + bxmin, bymin, bxmax, bymax = tile_bounds + writer_opts["offset_x"] = (bxmin + bxmax) / 2.0 + writer_opts["offset_y"] = (bymin + bymax) / 2.0 + + stages = [{"type": "readers.las", "filename": str(path)} for path in input_files] + if len(stages) > 1: + stages.append({"type": "filters.merge"}) + stages.append(writer_opts) + pipeline = {"pipeline": stages} + + output_laz.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as handle: + json.dump(pipeline, handle) + pipeline_file = Path(handle.name) + try: + result = subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + finally: + if pipeline_file.exists(): + pipeline_file.unlink() + if result.returncode != 0: + return (False, f"standard-dimension LAZ strip failed: {result.stderr[:200]}") + if not output_laz.exists() or output_laz.stat().st_size == 0: + return (False, "standard-dimension LAZ strip produced no output") + return (True, "stripped-laz") + + +def finalize_tile_to_copc(args: Tuple) -> Tuple[str, bool, str]: + """Merge a tile's LAS part files into one COPC tile.""" + label, tiles_dir, log_dir, tile_bounds = args + final_tile = tiles_dir / f"{label}.copc.laz" + + if final_tile.exists() and final_tile.stat().st_size > 0: + return (label, True, "Already exists") + + tile_dir = tiles_dir / label + if not tile_dir.exists(): + return (label, True, "No data in bounds") + + parts = sorted(tile_dir.glob("part_*.las")) + if not parts: + if not any(tile_dir.iterdir()): + tile_dir.rmdir() + return (label, True, "No data in bounds") + + try: + success, message = finalize_tile_to_copc_untwine( + parts, + final_tile, + log_dir, + label, + tile_bounds=tile_bounds, + ) + if not success: + return (label, False, message) + + crs_source = first_crs_source(parts) + if crs_source is not None: + preserved_geotiff, geotiff_message = append_source_geotiff_projection_evlrs( + crs_source, + final_tile, + ) + if not preserved_geotiff: + return (label, False, geotiff_message) + valid_crs, crs_message = copc_preserves_source_crs(crs_source, final_tile) + if not valid_crs and message == "untwine": + try: + final_tile.unlink(missing_ok=True) + except OSError: + pass + success, message = finalize_tile_to_copc_pdal( + parts, + final_tile, + log_dir, + label, + tile_bounds, + ) + if not success: + return (label, False, f"{message}; after untwine CRS validation failed: {crs_message}") + preserved_geotiff, geotiff_message = append_source_geotiff_projection_evlrs( + crs_source, + final_tile, + ) + if not preserved_geotiff: + return (label, False, geotiff_message) + valid_crs, crs_message = copc_preserves_source_crs(crs_source, final_tile) + if not valid_crs: + return (label, False, f"COPC CRS validation failed: {crs_message}") + + for part in parts: + if part.exists(): + part.unlink() + if tile_dir.exists() and not any(tile_dir.iterdir()): + tile_dir.rmdir() + + return (label, True, f"{len(parts)} parts merged ({message})") + except Exception as exc: + return (label, False, str(exc)) + + +def finalize_tile_to_copc_pdal( + parts: List[Path], + final_tile: Path, + log_dir: Path, + label: str, + tile_bounds: Optional[Tuple[float, float, float, float]] = None, + preserve_extra_dims: bool = False, +) -> Tuple[bool, str]: + """Finalize a tile with PDAL writers.copc.""" + writer_opts = { + "type": "writers.copc", + "filename": str(final_tile), + "forward": "all", + } + if preserve_extra_dims: + writer_opts["extra_dims"] = "all" + if tile_bounds is not None: + bxmin, bymin, bxmax, bymax = tile_bounds + writer_opts["offset_x"] = (bxmin + bxmax) / 2.0 + writer_opts["offset_y"] = (bymin + bymax) / 2.0 + + if len(parts) == 1: + pipeline = { + "pipeline": [ + {"type": "readers.las", "filename": str(parts[0])}, + writer_opts, + ] + } + else: + readers = [{"type": "readers.las", "filename": str(path)} for path in parts] + pipeline = {"pipeline": readers + [{"type": "filters.merge"}, writer_opts]} + + pipeline_file = log_dir / f"{label}_pipeline.json" + with open(pipeline_file, "w") as handle: + json.dump(pipeline, handle) + + try: + result = subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + finally: + if pipeline_file.exists(): + pipeline_file.unlink() + + if result.returncode != 0: + return (False, f"COPC conversion failed: {result.stderr[:200]}") + return (True, "OK") + + +def finalize_tile_to_copc_untwine( + parts: List[Path], + final_tile: Path, + log_dir: Path, + label: str, + tile_bounds: Optional[Tuple[float, float, float, float]] = None, + preserve_extra_dims: bool = False, +) -> Tuple[bool, str]: + """Finalize a tile using untwine, falling back to PDAL when unavailable.""" + if not preserve_extra_dims: + untwine_cmd = shutil.which("untwine") + if untwine_cmd: + success, message = _run_untwine( + parts, + final_tile, + first_srs_assignment(parts), + strip_extra_dims=True, + ) + if success and _output_has_no_extra_dimensions(final_tile): + return (True, "untwine-stripped") + try: + final_tile.unlink(missing_ok=True) + except OSError: + pass + with tempfile.TemporaryDirectory(prefix=f"_{label}_strip_", dir=final_tile.parent) as tmpdir: + stripped_laz = Path(tmpdir) / f"{label}.stripped.laz" + success, message = _strip_las_to_standard_dims( + parts, + stripped_laz, + tile_bounds=tile_bounds, + ) + if success: + success, message = _run_untwine( + [stripped_laz], + final_tile, + first_srs_assignment(parts), + strip_extra_dims=False, + ) + if success and _output_has_no_extra_dimensions(final_tile): + return (True, "pdal-strip+untwine") + try: + final_tile.unlink(missing_ok=True) + except OSError: + pass + success, message = finalize_tile_to_copc_pdal( + parts, + final_tile, + log_dir, + label, + tile_bounds=tile_bounds, + preserve_extra_dims=False, + ) + if success: + return (True, "pdal-stripped") + return (False, message) + + untwine_cmd = shutil.which("untwine") + if not untwine_cmd: + success, message = finalize_tile_to_copc_pdal( + parts, + final_tile, + log_dir, + label, + tile_bounds=tile_bounds, + preserve_extra_dims=True, + ) + if success: + return (True, "pdal") + return (False, message) + + try: + input_args = [] + for part in parts: + input_args.extend(["-i", str(part)]) + srs_arg = first_srs_assignment(parts) + srs_args = ["--a_srs", srs_arg] if srs_arg else [] + + result = subprocess.run( + [untwine_cmd] + input_args + ["-o", str(final_tile)] + srs_args, + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + return (False, f"untwine failed: {result.stderr[:200]}") + if not final_tile.exists() or final_tile.stat().st_size == 0: + return (False, "untwine produced no output") + return (True, "untwine") + except Exception as exc: + return (False, f"untwine error: {exc}") + + +def convert_laz_to_copc( + input_laz: Path, + output_copc: Path, + preserve_extra_dims: bool = False, +) -> bool: + """Convert a single LAZ/LAS file to COPC with CRS/GeoTIFF validation. + + The default SmartTile COPC conversion strips extra point attributes while + forwarding header metadata/CRS. Prod-merged creation passes + preserve_extra_dims=True because those products must retain enriched + prediction and source attributes. + """ + if not preserve_extra_dims: + untwine_cmd = shutil.which("untwine") + if untwine_cmd: + success, _ = _run_untwine( + [input_laz], + output_copc, + srs_assignment_from_file(input_laz), + strip_extra_dims=True, + ) + if success and _output_has_no_extra_dimensions(output_copc): + append_source_geotiff_projection_evlrs(input_laz, output_copc) + valid_crs, message = copc_preserves_source_crs(input_laz, output_copc) + if valid_crs: + return True + print(f" Warning: untwine COPC CRS validation failed for {output_copc.name}: {message}; retrying with stripped LAZ + untwine") + elif success: + print(f" Warning: untwine dimension limiting kept extra dimensions for {output_copc.name}; retrying with stripped LAZ + untwine") + else: + print(f" Warning: untwine dimension limiting failed for {output_copc.name}; retrying with stripped LAZ + untwine") + try: + output_copc.unlink(missing_ok=True) + except OSError: + pass + if untwine_cmd: + with tempfile.TemporaryDirectory(prefix=f"_{output_copc.stem}_strip_", dir=output_copc.parent) as tmpdir: + stripped_laz = Path(tmpdir) / f"{output_copc.stem}.stripped.laz" + success, message = _strip_las_to_standard_dims([input_laz], stripped_laz) + if success: + success, message = _run_untwine( + [stripped_laz], + output_copc, + srs_assignment_from_file(input_laz), + strip_extra_dims=False, + ) + if success and _output_has_no_extra_dimensions(output_copc): + append_source_geotiff_projection_evlrs(input_laz, output_copc) + valid_crs, message = copc_preserves_source_crs(input_laz, output_copc) + if valid_crs: + return True + print(f" Warning: stripped LAZ + untwine COPC CRS validation failed for {output_copc.name}: {message}; retrying with PDAL") + try: + output_copc.unlink(missing_ok=True) + except OSError: + pass + elif success: + print(f" Warning: stripped LAZ + untwine kept extra dimensions for {output_copc.name}; retrying with PDAL") + try: + output_copc.unlink(missing_ok=True) + except OSError: + pass + if not convert_laz_to_copc_pdal(input_laz, output_copc, preserve_extra_dims=False): + return False + preserved_geotiff, message = append_source_geotiff_projection_evlrs(input_laz, output_copc) + if not preserved_geotiff: + print(f" Warning: COPC GeoTIFF projection preservation failed for {output_copc.name}: {message}") + return False + valid_crs, message = copc_preserves_source_crs(input_laz, output_copc) + if not valid_crs: + print(f" Warning: COPC CRS validation failed for {output_copc.name}: {message}") + return valid_crs + + untwine_cmd = shutil.which("untwine") + if untwine_cmd: + success, _ = _run_untwine( + [input_laz], + output_copc, + srs_assignment_from_file(input_laz), + strip_extra_dims=False, + ) + if success: + append_source_geotiff_projection_evlrs(input_laz, output_copc) + valid_crs, _ = copc_preserves_source_crs(input_laz, output_copc) + if valid_crs: + return True + try: + output_copc.unlink(missing_ok=True) + except OSError: + pass + + if not convert_laz_to_copc_pdal(input_laz, output_copc, preserve_extra_dims=True): + return False + preserved_geotiff, message = append_source_geotiff_projection_evlrs(input_laz, output_copc) + if not preserved_geotiff: + print(f" Warning: COPC GeoTIFF projection preservation failed for {output_copc.name}: {message}") + return False + valid_crs, message = copc_preserves_source_crs(input_laz, output_copc) + if not valid_crs: + print(f" Warning: COPC CRS validation failed for {output_copc.name}: {message}") + return valid_crs + + +def convert_laz_to_copc_pdal( + input_laz: Path, + output_copc: Path, + preserve_extra_dims: bool = False, +) -> bool: + """Convert a single LAZ/LAS file to COPC using PDAL writers.copc.""" + writer_opts = { + "type": "writers.copc", + "filename": str(output_copc), + "forward": "all", + } + if preserve_extra_dims: + writer_opts["extra_dims"] = "all" + pipeline = { + "pipeline": [ + {"type": "readers.las", "filename": str(input_laz)}, + writer_opts, + ] + } + with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as handle: + json.dump(pipeline, handle) + pipeline_file = Path(handle.name) + try: + result = subprocess.run( + [get_pdal_path(), "pipeline", str(pipeline_file)], + capture_output=True, + text=True, + check=False, + ) + return result.returncode == 0 and output_copc.exists() and output_copc.stat().st_size > 0 + finally: + if pipeline_file.exists(): + pipeline_file.unlink() diff --git a/src/tile_spatial.py b/src/tile_spatial.py new file mode 100644 index 0000000..1179d80 --- /dev/null +++ b/src/tile_spatial.py @@ -0,0 +1,234 @@ +#!/usr/bin/env python3 +"""Spatial geometry helpers for SmartTile tile merge/remap workflows.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Dict, Optional, Set, Tuple + +import laspy +import numpy as np + + +Bounds = Tuple[float, float, float, float] + + +def compute_tile_bounds(points: np.ndarray) -> Bounds: + """Return the XY bounding box of a point cloud.""" + return ( + points[:, 0].min(), + points[:, 0].max(), + points[:, 1].min(), + points[:, 1].max(), + ) + + +def get_tile_bounds_from_header(filepath: Path) -> Optional[Bounds]: + """Read XY bounds from a LAS/LAZ header without loading points.""" + try: + with laspy.open(str(filepath), laz_backend=laspy.LazBackend.LazrsParallel) as las: + return (las.header.x_min, las.header.x_max, las.header.y_min, las.header.y_max) + except Exception: + return None + + +def find_overlap_region(bounds_a: Bounds, bounds_b: Bounds) -> Optional[Bounds]: + """Return the XY overlap region between two bounding boxes.""" + minx_a, maxx_a, miny_a, maxy_a = bounds_a + minx_b, maxx_b, miny_b, maxy_b = bounds_b + overlap = ( + max(minx_a, minx_b), + min(maxx_a, maxx_b), + max(miny_a, miny_b), + min(maxy_a, maxy_b), + ) + if overlap[0] < overlap[1] and overlap[2] < overlap[3]: + return overlap + return None + + +def compute_centroids_vectorized(points: np.ndarray, instances: np.ndarray) -> Dict[int, np.ndarray]: + """Compute positive-instance centroids with sorting and cumulative sums.""" + valid_mask = instances > 0 + valid_points = points[valid_mask] + valid_instances = instances[valid_mask] + if len(valid_instances) == 0: + return {} + + sort_idx = np.argsort(valid_instances) + sorted_instances = valid_instances[sort_idx] + sorted_points = valid_points[sort_idx] + unique_instances, first_indices, counts = np.unique( + sorted_instances, return_index=True, return_counts=True + ) + + cumsum = np.zeros((len(sorted_points) + 1, 3), dtype=np.float64) + cumsum[1:] = np.cumsum(sorted_points, axis=0) + centroids = {} + for inst_id, start_idx, count in zip(unique_instances, first_indices, counts): + end_idx = start_idx + count + centroids[int(inst_id)] = (cumsum[end_idx] - cumsum[start_idx]) / count + return centroids + + +def _edge_alignment_score( + overlap_span: float, + size_a: float, + size_b: float, + min_a: float, + max_a: float, + min_b: float, + max_b: float, +) -> Tuple[float, float]: + ratio_a = overlap_span / size_a if size_a > 0 else 0.0 + ratio_b = overlap_span / size_b if size_b > 0 else 0.0 + max_ratio = max(ratio_a, ratio_b) + axis_alignment = ( + 1.0 + if (ratio_a > 0.8 and ratio_b > 0.8) + or (min_b == min_a and max_b == max_a) + or (max_ratio > 0.8) + else 0.5 + ) + edge_tolerance = 0.1 + low_edge_align = abs(min_a - min_b) < edge_tolerance + high_edge_align = abs(max_a - max_b) < edge_tolerance + edge_alignment = ( + 2.0 + if (low_edge_align and high_edge_align) + else (1.0 if (low_edge_align or high_edge_align) else 0.0) + ) + return axis_alignment, edge_alignment + + +def find_spatial_neighbors( + tile_boundary: Bounds, + tile_name: str, + all_tiles: Dict[str, Bounds], + tolerance: float = 1.0, +) -> Dict[str, Optional[str]]: + """Find east/west/north/south neighbors from actual spatial overlaps.""" + minx_a, maxx_a, miny_a, maxy_a = tile_boundary + tile_width_a = maxx_a - minx_a + tile_height_a = maxy_a - miny_a + neighbors = {"east": None, "west": None, "north": None, "south": None} + candidates = {"east": [], "west": [], "north": [], "south": []} + + for other_name, other_bounds in all_tiles.items(): + if other_name == tile_name: + continue + overlap = find_overlap_region(tile_boundary, other_bounds) + if overlap is None: + continue + + minx_b, maxx_b, miny_b, maxy_b = other_bounds + overlap_minx, overlap_maxx, overlap_miny, overlap_maxy = overlap + overlap_width = overlap_maxx - overlap_minx + overlap_height = overlap_maxy - overlap_miny + overlap_area = overlap_width * overlap_height + + if minx_b > minx_a and minx_b <= maxx_a + tolerance and overlap_width >= tolerance: + if not (maxy_b < miny_a or miny_b > maxy_a): + axis_alignment, edge_alignment = _edge_alignment_score( + overlap_height, tile_height_a, maxy_b - miny_b, miny_a, maxy_a, miny_b, maxy_b + ) + candidates["east"].append((overlap_area, axis_alignment, minx_b - minx_a, edge_alignment, other_name)) + + if minx_b < minx_a and maxx_b >= minx_a - tolerance and overlap_width >= tolerance: + if not (maxy_b < miny_a or miny_b > maxy_a): + axis_alignment, edge_alignment = _edge_alignment_score( + overlap_height, tile_height_a, maxy_b - miny_b, miny_a, maxy_a, miny_b, maxy_b + ) + candidates["west"].append((overlap_area, axis_alignment, minx_a - minx_b, edge_alignment, other_name)) + + if miny_b > miny_a and miny_b <= maxy_a + tolerance and overlap_height >= tolerance: + if not (maxx_b < minx_a or minx_b > maxx_a): + axis_alignment, edge_alignment = _edge_alignment_score( + overlap_width, tile_width_a, maxx_b - minx_b, minx_a, maxx_a, minx_b, maxx_b + ) + candidates["north"].append((overlap_area, axis_alignment, miny_b - miny_a, edge_alignment, other_name)) + + if miny_b < miny_a and maxy_b >= miny_a - tolerance and overlap_height >= tolerance: + if not (maxx_b < minx_a or minx_b > maxx_a): + axis_alignment, edge_alignment = _edge_alignment_score( + overlap_width, tile_width_a, maxx_b - minx_b, minx_a, maxx_a, minx_b, maxx_b + ) + candidates["south"].append((overlap_area, axis_alignment, miny_a - miny_b, edge_alignment, other_name)) + + for direction, values in candidates.items(): + if not values: + continue + best = max(values, key=lambda x: (x[1], x[3], x[0], -x[2])) + neighbors[direction] = best[4] + + return neighbors + + +def filter_by_centroid_in_buffer( + points: np.ndarray, + instances: np.ndarray, + boundary: Bounds, + tile_name: str, + all_tiles: Dict[str, Bounds], + buffer: float = 10.0, + precomputed_neighbors: Optional[Dict[str, Optional[str]]] = None, +) -> Tuple[Set[int], Dict[int, str]]: + """Return instances whose centroid falls in an overlapping tile buffer zone.""" + min_x, max_x, min_y, max_y = boundary + neighbors = ( + {direction: precomputed_neighbors.get(direction) for direction in ("east", "west", "north", "south")} + if precomputed_neighbors is not None + else find_spatial_neighbors(boundary, tile_name, all_tiles, tolerance=buffer) + ) + + buf_min_x = min_x + (buffer if neighbors["west"] is not None else 0) + buf_max_x = max_x - (buffer if neighbors["east"] is not None else 0) + buf_min_y = min_y + (buffer if neighbors["south"] is not None else 0) + buf_max_y = max_y - (buffer if neighbors["north"] is not None else 0) + + instances_to_remove = set() + instance_buffer_direction = {} + for inst_id, centroid in compute_centroids_vectorized(points, instances).items(): + if inst_id <= 0: + continue + cx, cy = centroid[0], centroid[1] + in_west_buffer = neighbors["west"] is not None and cx < buf_min_x + in_east_buffer = neighbors["east"] is not None and cx > buf_max_x + in_south_buffer = neighbors["south"] is not None and cy < buf_min_y + in_north_buffer = neighbors["north"] is not None and cy > buf_max_y + + if in_west_buffer or in_east_buffer or in_south_buffer or in_north_buffer: + instances_to_remove.add(inst_id) + if in_west_buffer: + instance_buffer_direction[inst_id] = "west" + elif in_south_buffer: + instance_buffer_direction[inst_id] = "south" + elif in_east_buffer: + instance_buffer_direction[inst_id] = "east" + else: + instance_buffer_direction[inst_id] = "north" + + return instances_to_remove, instance_buffer_direction + + +def get_border_region_mask( + points: np.ndarray, + boundary: Bounds, + inner_dist: float, + outer_dist: float, + neighbors: Dict[str, Optional[str]], +) -> np.ndarray: + """Return a mask for points in the edge band for directions with neighbors.""" + min_x, max_x, min_y, max_y = boundary + x, y = points[:, 0], points[:, 1] + mask = np.zeros(len(points), dtype=bool) + + if neighbors.get("east") is not None: + mask |= (x > max_x - outer_dist) & (x <= max_x - inner_dist) + if neighbors.get("west") is not None: + mask |= (x >= min_x + inner_dist) & (x < min_x + outer_dist) + if neighbors.get("north") is not None: + mask |= (y > max_y - outer_dist) & (y <= max_y - inner_dist) + if neighbors.get("south") is not None: + mask |= (y >= min_y + inner_dist) & (y < min_y + outer_dist) + return mask diff --git a/src/tile_tindex.py b/src/tile_tindex.py new file mode 100644 index 0000000..2267fd6 --- /dev/null +++ b/src/tile_tindex.py @@ -0,0 +1,319 @@ +#!/usr/bin/env python3 +"""Tindex and tile-bounds helpers for SmartTile tiling.""" + +from __future__ import annotations + +import json +import shutil +import sqlite3 +import struct +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +from point_cloud_metadata import point_cloud_files + + +def get_pdal_path() -> str: + """Return the PDAL executable path.""" + pdal_path = shutil.which("pdal") + return pdal_path if pdal_path else "pdal" + + +def get_pdal_wrench_path() -> str: + """Return the pdal_wrench executable path.""" + wrench_path = shutil.which("pdal_wrench") + return wrench_path if wrench_path else "pdal_wrench" + + +def build_tindex(input_dir: Path, output_gpkg: Path) -> Path: + """Build a GeoPackage tindex from LAZ/LAS/COPC source files.""" + print() + print("=" * 60) + print("Step 1: Building spatial index (tindex)") + print("=" * 60) + + if output_gpkg.exists(): + if output_gpkg.stat().st_size > 0: + print(f" Using existing tindex: {output_gpkg}") + return output_gpkg + print(f" Removing empty tindex from previous failed run: {output_gpkg}") + output_gpkg.unlink() + + output_gpkg.parent.mkdir(parents=True, exist_ok=True) + source_files = point_cloud_files(input_dir) + if not source_files: + raise ValueError(f"No LAZ/LAS files found in {input_dir}") + + tindex_srs = None + try: + info_result = subprocess.run( + [get_pdal_path(), "info", "--metadata", str(source_files[0])], + capture_output=True, + text=True, + check=False, + ) + if info_result.returncode == 0: + meta = json.loads(info_result.stdout) + tindex_srs = ( + meta.get("metadata", {}).get("srs", {}).get("compoundwkt") + or meta.get("metadata", {}).get("spatialreference") + ) + except Exception as exc: + print(f" Warning: Could not extract SRS for tindex: {exc}") + + print(f" Found {len(source_files)} source files") + print(f" Output: {output_gpkg}") + + with tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False) as handle: + for source_file in source_files: + handle.write(f"{source_file.absolute()}\n") + file_list_path = Path(handle.name) + + try: + with tempfile.TemporaryDirectory(prefix="smarttile_tindex_") as tmp_dir: + tmp_gpkg = Path(tmp_dir) / output_gpkg.name + tmp_cmd = [ + get_pdal_path(), + "tindex", + "create", + str(tmp_gpkg), + "--filelist", + str(file_list_path), + "--tindex_name=Location", + "--ogrdriver=GPKG", + "--fast_boundary", + "--write_absolute_path", + ] + if tindex_srs: + tmp_cmd.append(f"--t_srs={tindex_srs}") + + result = subprocess.run(tmp_cmd, capture_output=True, text=True, check=False) + if result.returncode != 0 and "Unexpected argument 'filelist'" in (result.stderr or result.stdout): + stdin_cmd = [ + get_pdal_path(), + "tindex", + "create", + str(tmp_gpkg), + "--stdin", + "--tindex_name=Location", + "--ogrdriver=GPKG", + "--fast_boundary", + "--write_absolute_path", + ] + if tindex_srs: + stdin_cmd.append(f"--t_srs={tindex_srs}") + result = subprocess.run( + stdin_cmd, + input=file_list_path.read_text(), + capture_output=True, + text=True, + check=False, + ) + + if result.returncode != 0: + raise RuntimeError(f"pdal tindex failed: {result.stderr or result.stdout or 'unknown error'}") + if not tmp_gpkg.exists() or tmp_gpkg.stat().st_size == 0: + raise RuntimeError(f"pdal tindex produced an empty GeoPackage: {tmp_gpkg}") + + shutil.copy2(tmp_gpkg, output_gpkg) + + if not output_gpkg.exists() or output_gpkg.stat().st_size == 0: + raise RuntimeError(f"Copied tindex is missing or empty: {output_gpkg}") + + print(f" ✓ Tindex created: {output_gpkg}") + finally: + if file_list_path.exists(): + file_list_path.unlink() + + return output_gpkg + + +def calculate_tile_bounds( + tindex_file: Path, + tile_length: float, + tile_buffer: float, + output_dir: Path, + grid_offset: float = 1.0, +) -> Tuple[Path, Path, dict]: + """Calculate tile jobs and bounds JSON from a tindex.""" + print() + print("=" * 60) + print("Step 2: Calculating tile bounds") + print("=" * 60) + + prepare_jobs_script = Path(__file__).parent / "prepare_tile_jobs.py" + jobs_file = output_dir / f"tile_jobs_{int(tile_length)}m.txt" + bounds_json = output_dir / "tile_bounds_tindex.json" + cmd = [ + sys.executable, + str(prepare_jobs_script), + str(tindex_file), + f"--tile-length={tile_length}", + f"--tile-buffer={tile_buffer}", + f"--jobs-out={jobs_file}", + f"--bounds-out={bounds_json}", + f"--grid-offset={grid_offset}", + ] + + print(f" Tile length: {tile_length}m") + print(f" Tile buffer: {tile_buffer}m") + result = subprocess.run(cmd, capture_output=True, text=True, check=False) + if result.returncode != 0: + raise RuntimeError(f"prepare_tile_jobs.py failed: {result.stderr}") + + env = {} + for line in result.stdout.splitlines(): + if "=" in line: + key, value = line.split("=", 1) + env[key.strip()] = value.strip().strip('"') + + print(f" ✓ Calculated {env.get('tile_count', 'unknown')} tiles") + print(f" Jobs file: {jobs_file}") + print(f" Bounds file: {bounds_json}") + return jobs_file, bounds_json, env + + +def update_tile_bounds_json_from_files( + tile_bounds_json: Path, + files_dir: Path, + file_glob: str = "*.laz", +) -> int: + """Update tile_bounds_tindex.json from created tile file headers.""" + from tile_spatial import get_tile_bounds_from_header + + if not tile_bounds_json.exists(): + return 0 + with tile_bounds_json.open() as handle: + data = json.load(handle) + tiles = data.get("tiles", []) + if not tiles: + return 0 + + label_to_path: Dict[str, Path] = {} + for path in files_dir.glob(file_glob): + stem = path.stem + for sep in ("_subsampled", "_chunk", "."): + if sep in stem: + stem = stem.split(sep)[0] + break + if stem and stem not in label_to_path: + label_to_path[stem] = path + + updated = 0 + for tile in tiles: + label = f"c{tile['col']:02d}_r{tile['row']:02d}" + path = label_to_path.get(label) + if path is None: + continue + bounds = get_tile_bounds_from_header(path) + if bounds is None: + continue + minx, maxx, miny, maxy = bounds + tile["bounds"] = [[minx, maxx], [miny, maxy]] + updated += 1 + + if updated > 0: + with tile_bounds_json.open("w") as handle: + json.dump(data, handle, indent=2) + return updated + + +def get_source_files_from_tindex(tindex_file: Path) -> List[str]: + """Return source point-cloud paths from a tindex database.""" + conn = sqlite3.connect(str(tindex_file)) + cursor = conn.cursor() + cursor.execute('SELECT table_name FROM gpkg_contents WHERE data_type = "features" LIMIT 1') + result = cursor.fetchone() + if not result: + conn.close() + return [] + + table_name = result[0] + cursor.execute(f'SELECT DISTINCT Location FROM "{table_name}"') + files = [row[0] for row in cursor.fetchall()] + conn.close() + return files + + +def get_source_bounds_from_tindex(tindex_file: Path) -> Dict[str, Tuple[float, float, float, float]]: + """Return source-file bounds from tindex GeoPackage geometries.""" + conn = sqlite3.connect(str(tindex_file)) + cursor = conn.cursor() + cursor.execute("SELECT table_name, column_name FROM gpkg_geometry_columns LIMIT 1") + row = cursor.fetchone() + if not row: + conn.close() + return {} + table_name, geom_col = row + + cursor.execute(f'SELECT Location, "{geom_col}" FROM "{table_name}"') + bounds_map = {} + for filepath, geom_blob in cursor.fetchall(): + if not geom_blob or not filepath: + continue + try: + flags = geom_blob[3] + envelope_type = (flags >> 1) & 0x07 + header_size = 8 + if envelope_type in (1, 2): + minx, maxx, miny, maxy = struct.unpack_from(" Optional[Tuple[float, float, float, float]]: + """Parse '([xmin,xmax],[ymin,ymax])' into (xmin, ymin, xmax, ymax).""" + try: + string_value = proj_bounds.strip().strip("()") + parts = string_value.split("],[") + xpart = parts[0].strip("([])").split(",") + ypart = parts[1].strip("([])").split(",") + xmin, xmax = float(xpart[0]), float(xpart[1]) + ymin, ymax = float(ypart[0]), float(ypart[1]) + return (xmin, ymin, xmax, ymax) + except (ValueError, IndexError): + return None + + +def bounds_overlap( + a: Tuple[float, float, float, float], + b: Tuple[float, float, float, float], +) -> bool: + """Return whether two (minx, miny, maxx, maxy) boxes overlap.""" + return a[0] < b[2] and a[2] > b[0] and a[1] < b[3] and a[3] > b[1] + + +def get_bounds( + filepath: str, + source_bounds: Dict[str, Tuple[float, float, float, float]], + bounds_by_basename: Optional[Dict[str, Tuple[float, float, float, float]]] = None, +) -> Optional[Tuple[float, float, float, float]]: + """Look up source bounds by exact path, with basename fallback.""" + file_bounds = source_bounds.get(filepath) + if file_bounds is not None: + return file_bounds + if bounds_by_basename is not None: + return bounds_by_basename.get(Path(filepath).name) + return None + + +def filter_source_files_for_tile( + source_files: List[str], + source_bounds: Dict[str, Tuple[float, float, float, float]], + tile_bounds: Tuple[float, float, float, float], + bounds_by_basename: Optional[Dict[str, Tuple[float, float, float, float]]] = None, +) -> List[str]: + """Return source files whose bounds overlap tile bounds.""" + result = [] + for source_file in source_files: + file_bounds = get_bounds(source_file, source_bounds, bounds_by_basename) + if file_bounds is None or bounds_overlap(file_bounds, tile_bounds): + result.append(source_file) + return result diff --git a/src/union_find.py b/src/union_find.py new file mode 100644 index 0000000..87c2a24 --- /dev/null +++ b/src/union_find.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Union-Find data structure with size-biased roots.""" + +from __future__ import annotations + +from collections import defaultdict +from typing import Dict, List + + +class UnionFind: + """Disjoint-set structure that keeps the larger component as root.""" + + def __init__(self): + self.parent = {} + self.rank = {} + self.size = {} + + def make_set(self, x, size: int = 0): + """Create a new set containing only x.""" + if x not in self.parent: + self.parent[x] = x + self.rank[x] = 0 + self.size[x] = size + + def find(self, x) -> int: + """Find the root of the set containing x with path compression.""" + if x not in self.parent: + self.make_set(x) + if self.parent[x] != x: + self.parent[x] = self.find(self.parent[x]) + return self.parent[x] + + def union(self, x, y) -> int: + """Merge two sets and return the size-biased root.""" + root_x, root_y = self.find(x), self.find(y) + if root_x == root_y: + return root_x + if self.size.get(root_x, 0) >= self.size.get(root_y, 0): + self.parent[root_y] = root_x + self.size[root_x] = self.size.get(root_x, 0) + self.size.get(root_y, 0) + return root_x + self.parent[root_x] = root_y + self.size[root_y] = self.size.get(root_x, 0) + self.size.get(root_y, 0) + return root_y + + def get_components(self) -> Dict[int, List[int]]: + """Return connected components as {root: [members]}.""" + components = defaultdict(list) + for x in self.parent: + components[self.find(x)].append(x) + return dict(components) diff --git a/src/worker_budget.py b/src/worker_budget.py new file mode 100644 index 0000000..1733591 --- /dev/null +++ b/src/worker_budget.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 +"""Shared worker-budget helpers for nested SmartTile parallelism.""" + +from __future__ import annotations + + +def kdtree_query_workers(total_workers: int, outer_workers: int) -> int: + """Return per-task cKDTree query workers without oversubscribing CPUs.""" + total = max(1, int(total_workers or 1)) + outer = max(1, int(outer_workers or 1)) + return max(1, total // outer) diff --git a/tests/test_copc_staging.py b/tests/test_copc_staging.py new file mode 100644 index 0000000..afa764c --- /dev/null +++ b/tests/test_copc_staging.py @@ -0,0 +1,99 @@ +import sys +import tempfile +import unittest +from pathlib import Path + +import laspy +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from copc_staging import ( # noqa: E402 + staged_copc_matches_source, + write_copc_stage_manifest_entry, +) + + +def _write_las(path: Path, n_points: int = 3) -> None: + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([100.0, 200.0, 10.0]) + las = laspy.LasData(header) + las.x = np.arange(n_points, dtype=np.float64) * 0.1 + 100.0 + las.y = np.arange(n_points, dtype=np.float64) * 0.1 + 200.0 + las.z = np.arange(n_points, dtype=np.float64) * 0.1 + 10.0 + las.write(path) + + +def _write_las_with_extra_description(path: Path, description: str, n_points: int = 3) -> None: + header = laspy.LasHeader(point_format=6, version="1.4") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([100.0, 200.0, 10.0]) + las = laspy.LasData(header) + las.x = np.arange(n_points, dtype=np.float64) * 0.1 + 100.0 + las.y = np.arange(n_points, dtype=np.float64) * 0.1 + 200.0 + las.z = np.arange(n_points, dtype=np.float64) * 0.1 + 10.0 + las.add_extra_dim( + laspy.ExtraBytesParams( + name="PredInstance", + type=np.uint16, + description=description, + ) + ) + las.PredInstance = np.arange(n_points, dtype=np.uint16) + las.write(path) + + +class CopcStagingTests(unittest.TestCase): + def test_manifest_allows_exact_source_and_staged_header_match(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source.las" + staged = root / "source.copc.las" + _write_las(source) + _write_las(staged) + + write_copc_stage_manifest_entry(source, staged) + + self.assertTrue(staged_copc_matches_source(staged, source)) + + def test_manifest_rejects_changed_source_or_changed_staged_file(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source.las" + staged = root / "source.copc.las" + _write_las(source) + _write_las(staged) + write_copc_stage_manifest_entry(source, staged) + + _write_las(source, n_points=4) + self.assertFalse(staged_copc_matches_source(staged, source)) + + _write_las(source) + _write_las(staged, n_points=4) + self.assertFalse(staged_copc_matches_source(staged, source)) + + def test_no_manifest_allows_structural_header_match(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source.las" + staged = root / "source.copc.las" + _write_las_with_extra_description(source, "source description") + _write_las_with_extra_description(staged, "") + + self.assertTrue(staged_copc_matches_source(staged, source)) + + def test_no_manifest_rejects_changed_structural_header(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source.las" + staged = root / "source.copc.las" + _write_las_with_extra_description(source, "source description", n_points=3) + _write_las_with_extra_description(staged, "", n_points=4) + + self.assertFalse(staged_copc_matches_source(staged, source)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_create_merged_file.py b/tests/test_create_merged_file.py new file mode 100644 index 0000000..dd95cd0 --- /dev/null +++ b/tests/test_create_merged_file.py @@ -0,0 +1,813 @@ +import os +import sys +import tempfile +import unittest +from unittest import mock +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from main_create_merged_file import ( # noqa: E402 + _merge_chunk_files_pipeline, + _merge_prod_chunks, + _untwine_chunk_files_to_copc, + _validate_expected_dims, + _validate_preserved_product_dims, + create_chunked_prod_merged_file, + create_chunked_prod_merged_files_for_resolution, + create_prod_merged_files, + create_prod_merged_file, + expensive_prod_merged_warning, + parse_merged_output_formats, + parse_merged_resolutions, + point_cloud_files, + prepare_copc_inputs, + prod_merged_output_path, + prod_merged_pipeline, + source_key, +) + + +class CreateMergedFileTests(unittest.TestCase): + def test_parse_merged_resolutions_defaults_to_res1_and_res2(self): + self.assertEqual( + parse_merged_resolutions("res1,res2", 0.01, 0.1), + [("1cm", 0.01), ("10cm", 0.1)], + ) + + def test_parse_merged_resolutions_accepts_numeric_and_centimeter_tokens(self): + self.assertEqual( + parse_merged_resolutions("1cm,0.1,res2", 0.01, 0.1), + [("1cm", 0.01), ("10cm", 0.1)], + ) + + def test_parse_merged_output_formats_accepts_supported_formats(self): + self.assertEqual( + parse_merged_output_formats("laz,copc,ply,copc.laz"), + ["laz", "copc.laz", "ply"], + ) + + def test_parse_merged_output_formats_accepts_list_like_values(self): + self.assertEqual( + parse_merged_output_formats(["copc.laz", "laz", "ply"]), + ["copc.laz", "laz", "ply"], + ) + self.assertEqual( + parse_merged_output_formats("['copc.laz', 'laz']"), + ["copc.laz", "laz"], + ) + + def test_parse_merged_output_formats_defaults_to_copc_laz(self): + self.assertEqual(parse_merged_output_formats(""), ["copc.laz"]) + + def test_parse_merged_output_formats_rejects_unknown_format(self): + with self.assertRaisesRegex(ValueError, "Unsupported merged output format"): + parse_merged_output_formats("laz,txt") + + def test_prod_merged_output_path_uses_resolution_label(self): + self.assertEqual( + prod_merged_output_path(Path("/tmp/out"), "10cm"), + Path("/tmp/out/prod_merged_10cm.copc.laz"), + ) + + def test_prod_merged_output_path_uses_selected_format(self): + self.assertEqual( + prod_merged_output_path(Path("/tmp/out"), "10cm", "copc.laz"), + Path("/tmp/out/prod_merged_10cm.copc.laz"), + ) + self.assertEqual( + prod_merged_output_path(Path("/tmp/out"), "10cm", "ply"), + Path("/tmp/out/prod_merged_10cm.ply"), + ) + + def test_expensive_warning_only_for_full_resolution_copc(self): + warning = expensive_prod_merged_warning("1cm", 0.01, "copc.laz") + self.assertIsNotNone(warning) + self.assertIn("scratch-disk", warning) + self.assertIsNone(expensive_prod_merged_warning("10cm", 0.1, "copc.laz")) + self.assertIsNone(expensive_prod_merged_warning("1cm", 0.01, "laz")) + self.assertIsNone(expensive_prod_merged_warning("1cm", 0.01, "ply")) + + def test_point_cloud_files_excludes_copc_derivatives(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + raw_laz = root / "a.laz" + raw_las = root / "b.las" + copc = root / "c.copc.laz" + raw_laz.write_text("raw") + raw_las.write_text("raw") + copc.write_text("copc") + + self.assertEqual(point_cloud_files(root), [raw_laz, raw_las]) + + def test_source_key_treats_raw_and_copc_as_same_source(self): + self.assertEqual(source_key(Path("/tmp/original.laz")), "original") + self.assertEqual(source_key(Path("/tmp/original.copc.laz")), "original") + self.assertEqual(source_key(Path("/tmp/SOURCE.COPC.LAZ")), "source") + self.assertEqual(source_key(Path("/tmp/source.laz")), "source") + + def test_standardization_dim_validation_fails_for_missing_dims(self): + with mock.patch( + "main_create_merged_file.point_cloud_dimension_names", + return_value={"X", "Y", "Z", "intensity"}, + ): + with self.assertRaisesRegex(RuntimeError, "missing 1 standardized dimension"): + _validate_expected_dims( + [Path("/tmp/source.copc.laz")], + {"intensity", "Amplitude"}, + "staged inputs", + ) + + def test_product_dim_validation_fails_when_writer_drops_dims(self): + with mock.patch( + "main_create_merged_file.point_cloud_dimension_names", + side_effect=[ + {"X", "Y", "Z", "PredInstance", "Amplitude"}, + {"X", "Y", "Z", "Amplitude"}, + ], + ): + valid, message = _validate_preserved_product_dims( + [Path("/tmp/chunk.laz")], + Path("/tmp/prod_merged.copc.laz"), + ) + + self.assertFalse(valid) + self.assertIn("PredInstance", message) + + def test_create_prod_merged_files_validates_standardization_dims(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + summary = tmp_path / "collection_summary.json" + summary.write_text( + """ + { + "collection": { + "reference_attribute_names": ["X", "Y", "Z", "Intensity", "Amplitude", "ZeroDim"], + "global_attribute_stats": [ + {"name": "Intensity", "variance": 1.0}, + {"name": "Amplitude", "variance": 2.0}, + {"name": "ZeroDim", "variance": 0.0} + ] + } + } + """ + ) + created = tmp_path / "prod_merged_10cm.laz" + + def create_file(*args, **kwargs): + created.write_text("laz") + return created + + with mock.patch( + "main_create_merged_file.prepare_copc_inputs", + return_value=[tmp_path / "source.copc.laz"], + ) as prepare: + with mock.patch( + "main_create_merged_file.point_cloud_dimension_names", + return_value={"X", "Y", "Z", "intensity", "Amplitude"}, + ) as dims: + with mock.patch( + "main_create_merged_file.create_prod_merged_file", + side_effect=create_file, + ): + outputs = create_prod_merged_files( + tmp_path / "original_with_predictions", + tmp_path, + "10cm", + "laz", + 0.01, + 0.1, + standardization_json=summary, + ) + + self.assertEqual(outputs, [created]) + prepare.assert_called_once() + self.assertGreaterEqual(dims.call_count, 2) + + def test_prod_merged_pipeline_uses_nearest_to_centroid_and_forwards_dimensions(self): + pipeline = prod_merged_pipeline( + [Path("/tmp/a.copc.laz"), Path("/tmp/b.copc.laz")], + Path("/tmp/out/prod_merged_1cm.laz"), + 0.01, + "laz", + )["pipeline"] + + self.assertEqual(pipeline[0]["type"], "readers.copc") + self.assertEqual(pipeline[1]["type"], "readers.copc") + self.assertEqual(pipeline[2], {"type": "filters.merge"}) + self.assertEqual( + pipeline[3], + {"type": "filters.voxelcentroidnearestneighbor", "cell": 0.01}, + ) + self.assertEqual(pipeline[4]["type"], "writers.las") + self.assertEqual(pipeline[4]["forward"], "all") + self.assertEqual(pipeline[4]["extra_dims"], "all") + + def test_prod_merged_pipeline_defaults_to_copc_laz(self): + pipeline = prod_merged_pipeline( + [Path("/tmp/a.copc.laz")], + Path("/tmp/out/prod_merged_1cm.copc.laz"), + 0.01, + )["pipeline"] + + self.assertEqual(pipeline[-1]["type"], "writers.copc") + + def test_prod_merged_pipeline_can_write_copc_laz(self): + pipeline = prod_merged_pipeline( + [Path("/tmp/a.copc.laz")], + Path("/tmp/out/prod_merged_1cm.copc.laz"), + 0.01, + "copc.laz", + )["pipeline"] + + self.assertEqual(pipeline[-1]["type"], "writers.copc") + self.assertEqual(pipeline[-1]["forward"], "all") + self.assertEqual(pipeline[-1]["extra_dims"], "all") + + def test_prod_merged_pipeline_can_write_ply(self): + pipeline = prod_merged_pipeline( + [Path("/tmp/a.copc.laz")], + Path("/tmp/out/prod_merged_1cm.ply"), + 0.01, + "ply", + )["pipeline"] + + self.assertEqual(pipeline[-1]["type"], "writers.ply") + self.assertEqual(pipeline[-1]["storage_mode"], "little endian") + self.assertNotIn("forward", pipeline[-1]) + + def test_chunked_ply_merge_does_not_receive_las_scale_options(self): + pipeline = _merge_chunk_files_pipeline( + [Path("/tmp/chunk.laz")], + Path("/tmp/out/prod_merged_1cm.ply"), + "ply", + {"scale_x": 0.01, "offset_x": 500000.0}, + )["pipeline"] + + self.assertEqual(pipeline[-1]["type"], "writers.ply") + self.assertNotIn("scale_x", pipeline[-1]) + self.assertNotIn("offset_x", pipeline[-1]) + + def test_chunked_copc_merge_prefers_direct_untwine_path(self): + with mock.patch( + "main_create_merged_file._untwine_chunk_files_to_copc", + return_value=(True, "untwine"), + ) as untwine: + with mock.patch("main_create_merged_file._run_pdal_pipeline") as pdal: + _merge_prod_chunks( + [Path("/tmp/chunk.laz")], + Path("/tmp/out/prod_merged_1cm.copc.laz"), + "copc.laz", + Path("/tmp/work"), + Path("/tmp/source.copc.laz"), + {"scale_x": 0.01}, + ) + + untwine.assert_called_once() + pdal.assert_not_called() + + def test_chunked_copc_pdal_fallback_validates_metadata(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + output = tmp_path / "prod_merged_1cm.copc.laz" + + def run_pipeline(*_, **__): + output.write_text("copc") + return mock.Mock(returncode=0, stdout="", stderr="") + + with mock.patch( + "main_create_merged_file._untwine_chunk_files_to_copc", + return_value=(False, "untwine unavailable"), + ): + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=run_pipeline): + with mock.patch( + "main_create_merged_file._validate_preserved_product_dims", + return_value=(True, "ok"), + ): + with mock.patch( + "main_create_merged_file._preserve_and_validate_las_metadata", + return_value=(True, "ok"), + ) as validate: + _merge_prod_chunks( + [tmp_path / "chunk.laz"], + output, + "copc.laz", + tmp_path / "work", + tmp_path / "source.copc.laz", + {"scale_x": 0.01}, + ) + + validate.assert_called_once_with(tmp_path / "source.copc.laz", output) + + def test_chunked_multiple_formats_reuse_one_chunk_set(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.copc.laz" + outputs = [ + (tmp_path / "prod_merged_10cm.laz", "laz"), + (tmp_path / "prod_merged_10cm.copc.laz", "copc.laz"), + (tmp_path / "prod_merged_10cm.ply", "ply"), + ] + merge_calls = [] + + def run_pipeline(pipeline, *_): + Path(pipeline["pipeline"][-1]["filename"]).write_text("chunk") + return mock.Mock(returncode=0, stdout="", stderr="") + + def merge_chunks(chunk_files, output_file, output_format, *_args, **_kwargs): + merge_calls.append((list(chunk_files), output_file, output_format)) + output_file.write_text(output_format) + + with mock.patch( + "main_create_merged_file._prod_merged_chunk_bounds", + return_value=["bounds-a", "bounds-b"], + ): + with mock.patch("main_create_merged_file._scale_offset_options", return_value={}): + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=run_pipeline) as pdal: + with mock.patch( + "main_create_merged_file._merge_prod_chunks", + side_effect=merge_chunks, + ): + created = create_chunked_prod_merged_files_for_resolution( + [source], + outputs, + 0.1, + 2, + ) + + self.assertEqual(created, [path for path, _ in outputs]) + self.assertEqual(pdal.call_count, 2) + self.assertEqual(len(merge_calls), 3) + first_chunk_set = merge_calls[0][0] + self.assertTrue(first_chunk_set) + for chunk_set, _, _ in merge_calls: + self.assertEqual(chunk_set, first_chunk_set) + + def test_single_copc_pipeline_validates_metadata(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.copc.laz" + output = tmp_path / "prod_merged_10cm.copc.laz" + source.write_text("copc") + + def run_pipeline(*_, **__): + output.write_text("copc") + return mock.Mock(returncode=0, stdout="", stderr="") + + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=run_pipeline): + with mock.patch( + "main_create_merged_file._validate_preserved_product_dims", + return_value=(True, "ok"), + ) as validate_dims: + with mock.patch( + "main_create_merged_file._preserve_and_validate_las_metadata", + return_value=(True, "ok"), + ) as validate: + created = create_prod_merged_file( + [source], + output, + 0.1, + "copc.laz", + num_spatial_chunks=1, + ) + + self.assertEqual(created, output) + validate_dims.assert_called_once_with([source], output) + validate.assert_called_once_with(source, output) + + def test_single_laz_pipeline_validates_metadata(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.copc.laz" + output = tmp_path / "prod_merged_10cm.laz" + source.write_text("copc") + + def run_pipeline(*_, **__): + output.write_text("laz") + return mock.Mock(returncode=0, stdout="", stderr="") + + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=run_pipeline): + with mock.patch( + "main_create_merged_file._validate_preserved_product_dims", + return_value=(True, "ok"), + ) as validate_dims: + with mock.patch( + "main_create_merged_file._preserve_and_validate_las_metadata", + return_value=(True, "ok"), + ) as validate: + created = create_prod_merged_file( + [source], + output, + 0.1, + "laz", + num_spatial_chunks=1, + ) + + self.assertEqual(created, output) + validate_dims.assert_called_once_with([source], output) + validate.assert_called_once_with(source, output) + + def test_single_product_removes_stale_output_before_rewrite(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.copc.laz" + output = tmp_path / "prod_merged_10cm.laz" + source.write_text("copc") + output.write_text("stale") + + def run_pipeline(*_, **__): + self.assertFalse(output.exists()) + output.write_text("fresh") + return mock.Mock(returncode=0, stdout="", stderr="") + + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=run_pipeline): + with mock.patch( + "main_create_merged_file._validate_preserved_product_dims", + return_value=(True, "ok"), + ): + with mock.patch( + "main_create_merged_file._preserve_and_validate_las_metadata", + return_value=(True, "ok"), + ): + created = create_prod_merged_file([source], output, 0.1, "laz") + + self.assertEqual(created, output) + self.assertEqual(output.read_text(), "fresh") + + def test_chunked_product_failure_cleans_scratch_by_default(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.copc.laz" + output = tmp_path / "prod_merged_10cm.laz" + source.write_text("copc") + + def fail_chunk(pipeline, *_): + Path(pipeline["pipeline"][-1]["filename"]).write_text("partial") + return mock.Mock(returncode=1, stdout="", stderr="chunk failed") + + with mock.patch("main_create_merged_file._prod_merged_chunk_bounds", return_value=["bounds-a"]): + with mock.patch("main_create_merged_file._scale_offset_options", return_value={}): + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=fail_chunk): + with self.assertRaisesRegex(RuntimeError, "chunk 1 failed"): + create_chunked_prod_merged_file([source], output, 0.1, "laz", 2) + + self.assertFalse((tmp_path / "_prod_merged_10cm_chunks").exists()) + + def test_chunked_product_failure_can_keep_scratch_for_debug(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.copc.laz" + output = tmp_path / "prod_merged_10cm.laz" + source.write_text("copc") + + def fail_chunk(pipeline, *_): + Path(pipeline["pipeline"][-1]["filename"]).write_text("partial") + return mock.Mock(returncode=1, stdout="", stderr="chunk failed") + + with mock.patch.dict(os.environ, {"SMARTTILE_KEEP_FAILED_CHUNKS": "1"}): + with mock.patch("main_create_merged_file._prod_merged_chunk_bounds", return_value=["bounds-a"]): + with mock.patch("main_create_merged_file._scale_offset_options", return_value={}): + with mock.patch("main_create_merged_file._run_pdal_pipeline", side_effect=fail_chunk): + with self.assertRaisesRegex(RuntimeError, "chunk 1 failed"): + create_chunked_prod_merged_file([source], output, 0.1, "laz", 2) + + self.assertTrue((tmp_path / "_prod_merged_10cm_chunks").exists()) + + def test_direct_untwine_uses_configured_temp_dir(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + output = tmp_path / "out.copc.laz" + temp_dir = tmp_path / "untwine_tmp" + output.write_text("copc") + fake_copc_metadata = mock.Mock( + append_source_geotiff_projection_evlrs=mock.Mock(return_value=(True, "ok")), + copc_preserves_source_crs=mock.Mock(return_value=(True, "ok")), + srs_assignment_from_file=mock.Mock(return_value="EPSG:32632"), + ) + with mock.patch("main_create_merged_file.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch.dict(sys.modules, {"copc_metadata": fake_copc_metadata}): + with mock.patch("main_create_merged_file.subprocess.run") as run: + def run_untwine(*_, **__): + output.write_text("copc") + return mock.Mock(returncode=0, stdout="", stderr="") + + run.side_effect = run_untwine + with mock.patch( + "main_create_merged_file._point_cloud_point_count", + side_effect=[1, 1], + ): + with mock.patch( + "main_create_merged_file._validate_preserved_product_dims", + return_value=(True, "ok"), + ): + success, _ = _untwine_chunk_files_to_copc( + [tmp_path / "chunk.laz"], + output, + tmp_path / "source.copc.laz", + temp_dir=temp_dir, + ) + + self.assertTrue(success) + command = run.call_args.args[0] + self.assertIn("--temp_dir", command) + self.assertEqual(command[command.index("--temp_dir") + 1], str(temp_dir)) + + def test_direct_untwine_rejects_point_count_mismatch(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + output = tmp_path / "out.copc.laz" + fake_copc_metadata = mock.Mock( + append_source_geotiff_projection_evlrs=mock.Mock(return_value=(True, "ok")), + copc_preserves_source_crs=mock.Mock(return_value=(True, "ok")), + srs_assignment_from_file=mock.Mock(return_value="EPSG:32632"), + ) + + def run_untwine(*_, **__): + output.write_text("copc") + return mock.Mock(returncode=0, stdout="", stderr="") + + with mock.patch("main_create_merged_file.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch.dict(sys.modules, {"copc_metadata": fake_copc_metadata}): + with mock.patch("main_create_merged_file.subprocess.run", side_effect=run_untwine): + with mock.patch( + "main_create_merged_file._point_cloud_point_count", + side_effect=[3, 2], + ): + success, message = _untwine_chunk_files_to_copc( + [tmp_path / "chunk.laz"], + output, + tmp_path / "source.copc.laz", + ) + + self.assertFalse(success) + self.assertIn("point-count mismatch", message) + self.assertFalse(output.exists()) + + def test_direct_untwine_rejects_dimension_loss(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + output = tmp_path / "out.copc.laz" + fake_copc_metadata = mock.Mock( + append_source_geotiff_projection_evlrs=mock.Mock(return_value=(True, "ok")), + copc_preserves_source_crs=mock.Mock(return_value=(True, "ok")), + srs_assignment_from_file=mock.Mock(return_value="EPSG:32632"), + ) + + def run_untwine(*_, **__): + output.write_text("copc") + return mock.Mock(returncode=0, stdout="", stderr="") + + with mock.patch("main_create_merged_file.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch.dict(sys.modules, {"copc_metadata": fake_copc_metadata}): + with mock.patch("main_create_merged_file.subprocess.run", side_effect=run_untwine): + with mock.patch( + "main_create_merged_file._point_cloud_point_count", + side_effect=[3, 3], + ): + with mock.patch( + "main_create_merged_file._validate_preserved_product_dims", + return_value=(False, "prod-merged output dropped dimensions"), + ): + success, message = _untwine_chunk_files_to_copc( + [tmp_path / "chunk.laz"], + output, + tmp_path / "source.copc.laz", + ) + + self.assertFalse(success) + self.assertIn("dropped dimensions", message) + self.assertFalse(output.exists()) + + def test_prod_merged_pipeline_can_still_read_plain_laz(self): + pipeline = prod_merged_pipeline( + [Path("/tmp/a.laz")], + Path("/tmp/out/prod_merged_1cm.laz"), + 0.01, + )["pipeline"] + + self.assertEqual(pipeline[0]["type"], "readers.las") + + def test_prepare_copc_inputs_uses_tiling_converter(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + input_dir.mkdir() + + convert = mock.Mock(side_effect=lambda _, output, **__: output.write_text("copc") or True) + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + with mock.patch("main_create_merged_file.point_cloud_files") as files: + files.return_value = [input_dir / "original.laz"] + with mock.patch("main_create_merged_file.copc_files", return_value=[]): + with mock.patch("main_create_merged_file.write_copc_stage_manifest_entry"): + outputs = prepare_copc_inputs(input_dir, output_dir) + + expected = output_dir / "original_with_predictions_copc/original.copc.laz" + self.assertEqual(outputs, [expected]) + convert.assert_called_once_with( + input_dir / "original.laz", + expected, + preserve_extra_dims=True, + ) + + def test_prepare_copc_inputs_prefers_existing_copc_over_matching_raw_file(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + input_dir.mkdir() + + raw = input_dir / "original.laz" + copc = input_dir / "original.copc.laz" + raw.write_text("raw") + copc.write_text("copc") + + convert = mock.Mock() + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + with mock.patch("main_create_merged_file.staged_copc_matches_source", return_value=True): + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir) + + self.assertEqual(outputs, [copc]) + convert.assert_not_called() + + def test_prepare_copc_inputs_rebuilds_existing_copc_without_fresh_manifest(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + input_dir.mkdir() + + raw = input_dir / "original.laz" + stale_copc = input_dir / "original.copc.laz" + raw.write_text("raw") + stale_copc.write_text("stale copc") + + def convert_input(_, output, **__): + output.write_text("converted") + return True + + convert = mock.Mock(side_effect=convert_input) + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + with mock.patch("main_create_merged_file.staged_copc_matches_source", return_value=False): + with mock.patch("main_create_merged_file.write_copc_stage_manifest_entry") as manifest: + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir) + + expected = output_dir / "original_with_predictions_copc/original.copc.laz" + self.assertEqual(outputs, [expected]) + convert.assert_called_once_with(raw, expected, preserve_extra_dims=True) + manifest.assert_called_once_with(raw, expected) + + def test_prepare_copc_inputs_accepts_existing_copc_without_converter_import(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + input_dir.mkdir() + + copc = input_dir / "original.copc.laz" + copc.write_text("copc") + + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + outputs = prepare_copc_inputs(input_dir, output_dir) + + self.assertEqual(outputs, [copc]) + + def test_prepare_copc_inputs_keeps_raw_and_copc_only_sources(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + input_dir.mkdir() + + raw = input_dir / "alpha.laz" + copc = input_dir / "bravo.copc.laz" + raw.write_text("raw") + copc.write_text("copc") + + def convert_input(_, output, **__): + output.write_text("converted") + return True + + convert = mock.Mock(side_effect=convert_input) + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + with mock.patch("main_create_merged_file.write_copc_stage_manifest_entry"): + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir) + + converted = output_dir / "original_with_predictions_copc/alpha.copc.laz" + self.assertEqual(outputs, [converted, copc]) + convert.assert_called_once_with(raw, converted, preserve_extra_dims=True) + + def test_prepare_copc_inputs_reuses_staged_copc_for_raw_file(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + input_dir.mkdir() + + raw = input_dir / "original.laz" + staged = output_dir / "original_with_predictions_copc/original.copc.laz" + raw.write_text("raw") + staged.parent.mkdir(parents=True) + staged.write_text("copc") + + convert = mock.Mock() + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + with mock.patch("main_create_merged_file.staged_copc_matches_source", return_value=True): + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir) + + self.assertEqual(outputs, [staged]) + convert.assert_not_called() + + def test_prepare_copc_inputs_reuses_explicit_staged_copc_dir(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + staged_dir = tmp_path / "staged" + input_dir.mkdir() + staged_dir.mkdir() + + raw = input_dir / "original.laz" + staged = staged_dir / "original.copc.laz" + raw.write_text("raw") + staged.write_text("copc") + + convert = mock.Mock() + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + with mock.patch("main_create_merged_file.staged_copc_matches_source", return_value=True): + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir, staged_copc_dir=staged_dir) + + self.assertEqual(outputs, [staged]) + convert.assert_not_called() + + def test_prepare_copc_inputs_ignores_unmatched_staged_copc(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + staged_dir = tmp_path / "staged" + input_dir.mkdir() + staged_dir.mkdir() + + raw = input_dir / "original.laz" + staged = staged_dir / "original.copc.laz" + unrelated = staged_dir / "unrelated.copc.laz" + raw.write_text("raw") + staged.write_text("copc") + unrelated.write_text("stale copc") + + convert = mock.Mock() + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=True): + with mock.patch("main_create_merged_file.staged_copc_matches_source", return_value=True): + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir, staged_copc_dir=staged_dir) + + self.assertEqual(outputs, [staged]) + self.assertNotIn(unrelated, outputs) + convert.assert_not_called() + + def test_prepare_copc_inputs_ignores_unreadable_staged_copc(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "input" + output_dir = tmp_path / "out" + staged_dir = tmp_path / "staged" + input_dir.mkdir() + staged_dir.mkdir() + + raw = input_dir / "original.laz" + staged = staged_dir / "original.copc.laz" + raw.write_text("raw") + staged.write_text("partial") + + def convert_input(_, output, **__): + output.write_text("converted") + return True + + convert = mock.Mock(side_effect=convert_input) + fake_main_tile = mock.Mock(_convert_laz_to_copc=convert) + with mock.patch("main_create_merged_file._is_reusable_copc", return_value=False): + with mock.patch("main_create_merged_file.write_copc_stage_manifest_entry"): + with mock.patch.dict(sys.modules, {"main_tile": fake_main_tile}): + outputs = prepare_copc_inputs(input_dir, output_dir, staged_copc_dir=staged_dir) + + expected = output_dir / "original_with_predictions_copc/original.copc.laz" + self.assertEqual(outputs, [expected]) + convert.assert_called_once_with(raw, expected, preserve_extra_dims=True) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_dimension_transfer.py b/tests/test_dimension_transfer.py new file mode 100644 index 0000000..4ad663d --- /dev/null +++ b/tests/test_dimension_transfer.py @@ -0,0 +1,94 @@ +import sys +import unittest +from pathlib import Path + +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from dimension_transfer import ( # noqa: E402 + next_available_suffix, + plan_dimension_transfer, + suffixes_for_collision, +) + + +class DimensionTransferTests(unittest.TestCase): + def test_plans_add_overwrite_rename_and_skip_core_dimensions(self): + source_dims = { + "X": np.dtype(np.float64), + "OriginalOnly": np.dtype(np.uint16), + "EmptyTarget": np.dtype(np.uint8), + "RealCollision": np.dtype(np.float32), + } + target_names = {"X", "Y", "Z", "EmptyTarget", "RealCollision"} + target_arrays = { + "EmptyTarget": np.zeros(4, dtype=np.uint8), + "RealCollision": np.array([0.1, 0.2, 0.3, 0.4], dtype=np.float32), + } + + plan = plan_dimension_transfer( + source_dims, + target_names, + lambda name: target_arrays.get(name), + ) + + self.assertEqual(plan.add_new, {"OriginalOnly": np.dtype(np.uint16)}) + self.assertEqual(plan.overwrite, {"EmptyTarget": np.dtype(np.uint8)}) + self.assertEqual(plan.renamed, {"RealCollision": "RealCollision_1"}) + self.assertEqual( + plan.output_dtypes, + { + "OriginalOnly": np.dtype(np.uint16), + "EmptyTarget": np.dtype(np.uint8), + "RealCollision_1": np.dtype(np.float32), + }, + ) + self.assertEqual( + plan.output_to_source, + { + "OriginalOnly": "OriginalOnly", + "EmptyTarget": "EmptyTarget", + "RealCollision_1": "RealCollision", + }, + ) + + def test_constant_nonzero_target_is_overwritten(self): + plan = plan_dimension_transfer( + {"Classification": np.dtype(np.uint8)}, + {"Classification"}, + lambda _: np.array([2, 2, 2], dtype=np.uint8), + ) + + self.assertEqual(plan.overwrite, {"Classification": np.dtype(np.uint8)}) + self.assertFalse(plan.renamed) + + def test_empty_target_dimension_is_overwritten(self): + plan = plan_dimension_transfer( + {"PredInstance": np.dtype(np.uint16)}, + {"PredInstance"}, + lambda _: np.array([], dtype=np.uint16), + ) + + self.assertEqual(plan.overwrite, {"PredInstance": np.dtype(np.uint16)}) + self.assertFalse(plan.renamed) + + def test_renamed_collision_uses_next_available_suffix(self): + plan = plan_dimension_transfer( + {"Intensity": np.dtype(np.uint16)}, + {"Intensity", "Intensity_1"}, + lambda _: np.array([1, 2, 3], dtype=np.uint16), + ) + + self.assertEqual(plan.renamed, {"Intensity": "Intensity_2"}) + + def test_suffix_helpers_allocate_stable_names(self): + used = {"Dim_1"} + self.assertEqual(next_available_suffix("Dim", used), "Dim_2") + self.assertEqual(suffixes_for_collision("Dim", used), ("Dim_2", "Dim_3")) + self.assertEqual(used, {"Dim_1", "Dim_2", "Dim_3"}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_filter_buffer_instances.py b/tests/test_filter_buffer_instances.py new file mode 100644 index 0000000..ceb9c67 --- /dev/null +++ b/tests/test_filter_buffer_instances.py @@ -0,0 +1,139 @@ +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +import laspy +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +import filter_buffer_instances as filter_mod # noqa: E402 + + +class FilterBufferInstancesTests(unittest.TestCase): + def test_get_tile_neighbors_treats_non_grid_names_as_edge_tiles(self): + self.assertEqual( + filter_mod.get_tile_neighbors("single_tile", ["single_tile"]), + {"east": False, "west": False, "north": False, "south": False}, + ) + + def test_filter_directory_processes_laz_and_las_with_instance_dimension(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_dir = root / "input" + output_dir = root / "output" + input_dir.mkdir() + (input_dir / "c00_r00_segmented.laz").touch() + (input_dir / "c01_r00_segmented.las").touch() + + def fake_process(input_file, output_file, all_tile_names, buffer, instance_dimension): + output_file.parent.mkdir(parents=True, exist_ok=True) + output_file.touch() + return (100, 10, 1) + + with mock.patch.object(filter_mod, "process_tile", side_effect=fake_process) as process: + summary = filter_mod.filter_buffer_instances_dir( + input_dir=input_dir, + output_dir=output_dir, + buffer=20.0, + suffix="_clean", + instance_dimension="ModelInstance", + ) + + self.assertEqual(summary["input_files"], 2) + self.assertEqual(summary["total_original"], 200) + self.assertEqual(summary["total_removed"], 20) + self.assertEqual(summary["total_instances_removed"], 2) + self.assertEqual( + [call.args[0].name for call in process.call_args_list], + ["c00_r00_segmented.laz", "c01_r00_segmented.las"], + ) + self.assertEqual( + process.call_args_list[0].args[2], + ["c00_r00", "c01_r00"], + ) + self.assertEqual(process.call_args_list[0].args[3], 20.0) + self.assertEqual(process.call_args_list[0].kwargs["instance_dimension"], "ModelInstance") + self.assertTrue((output_dir / "c00_r00_segmented_clean.laz").exists()) + self.assertTrue((output_dir / "c01_r00_segmented_clean.las").exists()) + + def test_filter_directory_can_force_laz_output_extension(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_dir = root / "input" + output_dir = root / "output" + input_dir.mkdir() + (input_dir / "c00_r00_segmented.las").touch() + + def fake_process(input_file, output_file, all_tile_names, buffer, instance_dimension): + output_file.parent.mkdir(parents=True, exist_ok=True) + output_file.touch() + return (1, 0, 0) + + with mock.patch.object(filter_mod, "process_tile", side_effect=fake_process): + summary = filter_mod.filter_buffer_instances_dir( + input_dir=input_dir, + output_dir=output_dir, + output_extension=".laz", + ) + + self.assertEqual(summary["input_files"], 1) + self.assertTrue((output_dir / "c00_r00_segmented_filtered.laz").exists()) + self.assertFalse((output_dir / "c00_r00_segmented_filtered.las").exists()) + + def test_filter_directory_rejects_copc_output_extension(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_dir = root / "input" + input_dir.mkdir() + + with self.assertRaises(ValueError): + filter_mod.filter_buffer_instances_dir( + input_dir=input_dir, + output_dir=root / "output", + output_extension=".copc.laz", + ) + + def test_copy_path_strips_stale_copc_vlrs(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_file = root / "single.copc.laz" + output_file = root / "single_filtered.laz" + + header = laspy.LasHeader(point_format=3, version="1.2") + header.vlrs.append( + laspy.VLR( + user_id="copc", + record_id=1, + description="COPC info VLR", + record_data=b"stale", + ) + ) + las = laspy.LasData(header) + las.x = np.array([0.0, 1.0]) + las.y = np.array([0.0, 1.0]) + las.z = np.array([0.0, 1.0]) + las.write(str(input_file), do_compress=True, laz_backend=laspy.LazBackend.LazrsParallel) + + original, removed, removed_instances = filter_mod.process_tile( + input_file, + output_file, + all_tile_names=["single"], + instance_dimension="MissingInstance", + ) + + self.assertEqual((original, removed, removed_instances), (2, 0, 0)) + with laspy.open(str(output_file), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + stale_vlrs = [ + vlr for vlr in reader.header.vlrs + if getattr(vlr, "user_id", "") == "copc" + ] + self.assertEqual(stale_vlrs, []) + self.assertEqual(reader.header.point_count, 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_instance_label_contract.py b/tests/test_instance_label_contract.py new file mode 100644 index 0000000..da715af --- /dev/null +++ b/tests/test_instance_label_contract.py @@ -0,0 +1,209 @@ +import sys +import tempfile +import unittest +from datetime import date +from pathlib import Path + +import laspy +import numpy as np +from laspy.vlrs.vlr import VLR + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from instance_labels import ( # noqa: E402 + instance_output_dtype, + validate_prediction_instance_labels, +) +from merge_tiles import load_tile # noqa: E402 +from point_cloud_metadata import copc_files, point_cloud_files, raw_point_cloud_files # noqa: E402 +from point_cloud_outputs import merged_product_header, write_loaded_point_cloud # noqa: E402 + + +def _write_las_with_predinstance(path: Path, values: np.ndarray) -> None: + header = laspy.LasHeader(point_format=6, version="1.4") + header.scales = np.array([0.01, 0.01, 0.01]) + header.offsets = np.array([0.0, 0.0, 0.0]) + + las = laspy.LasData(header) + las.x = np.arange(len(values), dtype=np.float64) + las.y = np.zeros(len(values), dtype=np.float64) + las.z = np.zeros(len(values), dtype=np.float64) + las.add_extra_dim(laspy.ExtraBytesParams(name="PredInstance", type=values.dtype)) + las.PredInstance = values + las.write(path) + + +def _write_source_las(path: Path, system_identifier: str, projection_payload: bytes) -> None: + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([100.0, 200.0, 300.0]) + header.system_identifier = system_identifier + header.generating_software = "source-writer" + header.creation_date = date(2026, 6, 21) + header.vlrs.append( + VLR( + user_id="LASF_Projection", + record_id=34735, + description="synthetic projection", + record_data=projection_payload, + ) + ) + + las = laspy.LasData(header) + las.x = np.array([100.0]) + las.y = np.array([200.0]) + las.z = np.array([300.0]) + las.write(path) + + +class InstanceLabelContractTests(unittest.TestCase): + def test_cli_help_documents_uint32_threshold(self): + main_remap = (Path(__file__).resolve().parents[1] / "src" / "main_remap.py").read_text() + + self.assertIn("uint32 above 65,535", main_remap) + self.assertNotIn("63,535", main_remap) + + def test_point_cloud_files_prefers_copc_twins_but_accepts_raw_only_inputs(self): + with tempfile.TemporaryDirectory() as tmpdir: + directory = Path(tmpdir) + raw = directory / "source.laz" + copc = directory / "source.copc.laz" + raw.write_text("raw") + copc.write_text("copc") + + self.assertEqual(point_cloud_files(directory), [copc]) + + copc.unlink() + self.assertEqual(point_cloud_files(directory), [raw]) + + def test_point_cloud_file_discovery_is_case_insensitive(self): + with tempfile.TemporaryDirectory() as tmpdir: + directory = Path(tmpdir) + raw = directory / "UPPER.LAZ" + copc = directory / "UPPER.COPC.LAZ" + las = directory / "OTHER.LAS" + raw.write_text("raw") + copc.write_text("copc") + las.write_text("las") + + self.assertEqual(point_cloud_files(directory), [las, copc]) + self.assertEqual(raw_point_cloud_files(directory), [las, raw]) + self.assertEqual(copc_files(directory), [copc]) + + def test_point_cloud_file_discovery_missing_directory_is_empty(self): + missing = Path("/tmp/smarttile_missing_point_cloud_dir_for_test") + + self.assertEqual(point_cloud_files(missing), []) + self.assertEqual(raw_point_cloud_files(missing), []) + self.assertEqual(copc_files(missing), []) + + def test_accepts_background_and_positive_instances(self): + validate_prediction_instance_labels(np.array([0, 1, 65_535], dtype=np.uint16)) + + def test_rejects_negative_prediction_instance_labels(self): + with self.assertRaisesRegex(ValueError, "SmartTile expects PredInstance=0"): + validate_prediction_instance_labels( + np.array([0, -1, 7], dtype=np.int16), + "PredInstance", + "tile.laz", + ) + + def test_dtype_selection_still_uses_uint32_only_above_threshold(self): + self.assertEqual(instance_output_dtype(np.array([0, 65_535], dtype=np.uint32)), np.dtype(np.uint16)) + self.assertEqual(instance_output_dtype(np.array([0, 65_536], dtype=np.uint32)), np.dtype(np.uint32)) + + def test_load_tile_fails_fast_on_negative_predinstance(self): + with tempfile.TemporaryDirectory() as tmpdir: + path = Path(tmpdir) / "c00_r00.las" + _write_las_with_predinstance(path, np.array([0, -1, 1], dtype=np.int16)) + + with self.assertRaisesRegex(ValueError, "negative PredInstance values"): + load_tile( + path, + {"c00_r00": (0.0, 1.0, 0.0, 0.0)}, + buffer=10.0, + instance_dimension="PredInstance", + ) + + def test_single_source_merged_header_preserves_source_metadata(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) / "input" + tiles_dir = Path(tmpdir) / "tiles" + input_dir.mkdir() + tiles_dir.mkdir() + source = input_dir / "source.las" + payload = b"\x01\x00source-projection" + _write_source_las(source, "single-source", payload) + + header = merged_product_header( + np.array([[100.0, 200.0, 300.0], [101.0, 201.0, 301.0]]), + input_dir, + tiles_dir, + ) + + self.assertEqual(header.version, "1.2") + self.assertEqual(header.point_format.id, 3) + self.assertEqual(header.system_identifier, "single-source") + self.assertEqual(header.generating_software, "source-writer") + self.assertEqual(header.creation_date, date(2026, 6, 21)) + self.assertEqual(list(header.scales), [0.01, 0.01, 0.01]) + self.assertTrue(any(v.user_id == "LASF_Projection" and v.record_id == 34735 for v in header.vlrs)) + + def test_multi_source_merged_header_preserves_projection_without_source_identity(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) / "input" + tiles_dir = Path(tmpdir) / "tiles" + input_dir.mkdir() + tiles_dir.mkdir() + payload = b"\x01\x00shared-projection" + _write_source_las(input_dir / "a.las", "source-a", payload) + _write_source_las(input_dir / "b.las", "source-b", payload) + + header = merged_product_header( + np.array([[100.0, 200.0, 300.0], [101.0, 201.0, 301.0]]), + input_dir, + tiles_dir, + ) + + self.assertEqual(header.version, "1.4") + self.assertEqual(header.point_format.id, 6) + self.assertNotEqual(header.system_identifier, "source-a") + self.assertNotEqual(header.generating_software, "source-writer") + self.assertEqual(list(header.scales), [0.01, 0.01, 0.01]) + self.assertEqual( + [(v.user_id, v.record_id) for v in header.vlrs], + [("LASF_Projection", 34735)], + ) + + def test_write_loaded_point_cloud_preserves_selected_standard_dimensions(self): + with tempfile.TemporaryDirectory() as tmpdir: + source = Path(tmpdir) / "source.las" + output = Path(tmpdir) / "filtered.laz" + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([100.0, 200.0, 300.0]) + las = laspy.LasData(header) + las.x = np.array([100.0, 100.1, 100.2]) + las.y = np.array([200.0, 200.1, 200.2]) + las.z = np.array([300.0, 300.1, 300.2]) + las.intensity = np.array([11, 22, 33], dtype=np.uint16) + las.classification = np.array([2, 3, 4], dtype=np.uint8) + las.write(source) + + write_loaded_point_cloud( + source, + output, + np.array([[100.0, 200.0, 300.0], [100.2, 200.2, 300.2]]), + {"PredInstance": np.array([1, 2], dtype=np.uint16)}, + source_indices=np.array([0, 2]), + ) + + out = laspy.read(output) + np.testing.assert_array_equal(out.intensity, np.array([11, 33], dtype=np.uint16)) + np.testing.assert_array_equal(out.classification, np.array([2, 4], dtype=np.uint8)) + np.testing.assert_array_equal(out.PredInstance, np.array([1, 2], dtype=np.uint16)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_main_remap_file_discovery.py b/tests/test_main_remap_file_discovery.py new file mode 100644 index 0000000..a05e13a --- /dev/null +++ b/tests/test_main_remap_file_discovery.py @@ -0,0 +1,121 @@ +import sys +import tempfile +import unittest +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from main_remap import _match_files_via_json, _remap_point_cloud_files, find_matching_files # noqa: E402 + + +class MainRemapFileDiscoveryTests(unittest.TestCase): + def test_remap_file_discovery_includes_mixed_laz_and_las(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + (root / "tile_a.laz").write_text("placeholder") + (root / "tile_b.las").write_text("placeholder") + + files = [path.name for path in _remap_point_cloud_files(root)] + + self.assertEqual(files, ["tile_a.laz", "tile_b.las"]) + + def test_remap_file_discovery_prefers_copc_twin(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + (root / "tile_a.laz").write_text("placeholder") + (root / "tile_a.copc.laz").write_text("placeholder") + + files = [path.name for path in _remap_point_cloud_files(root)] + + self.assertEqual(files, ["tile_a.copc.laz"]) + + def test_spatial_matching_considers_mixed_laz_and_las(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source" + target = root / "target" + source.mkdir() + target.mkdir() + source_laz = source / "c00_r00_segmented.laz" + source_las = source / "c01_r00_segmented.las" + target_laz = target / "c00_r00.laz" + target_las = target / "c01_r00.las" + for path in (source_laz, source_las, target_laz, target_las): + path.write_text("placeholder") + + bounds = { + source_laz: (0.0, 10.0, 0.0, 10.0), + target_laz: (0.0, 10.0, 0.0, 10.0), + source_las: (20.0, 30.0, 0.0, 10.0), + target_las: (20.0, 30.0, 0.0, 10.0), + } + + import main_remap + + original_get_file_bounds = main_remap.get_file_bounds + try: + main_remap.get_file_bounds = bounds.__getitem__ + matches = find_matching_files(source, target) + finally: + main_remap.get_file_bounds = original_get_file_bounds + + self.assertEqual( + [(src.name, tgt.name) for src, tgt, _ in matches], + [ + ("c00_r00_segmented.laz", "c00_r00.laz"), + ("c01_r00_segmented.las", "c01_r00.las"), + ], + ) + + def test_json_matching_considers_mixed_laz_and_las(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source" + target = root / "target" + source.mkdir() + target.mkdir() + source_laz = source / "c00_r00_segmented.laz" + source_las = source / "c01_r00_segmented.las" + target_laz = target / "c00_r00.laz" + target_las = target / "c01_r00.las" + for path in (source_laz, source_las, target_laz, target_las): + path.write_text("placeholder") + + bounds = { + source_laz: (0.0, 10.0, 0.0, 10.0), + target_laz: (0.0, 10.0, 0.0, 10.0), + source_las: (20.0, 30.0, 0.0, 10.0), + target_las: (20.0, 30.0, 0.0, 10.0), + } + + import main_remap + + original_get_file_bounds = main_remap.get_file_bounds + original_build = main_remap.build_neighbor_graph_from_bounds_json + try: + main_remap.get_file_bounds = bounds.__getitem__ + main_remap.build_neighbor_graph_from_bounds_json = lambda *_: ( + [ + (0.0, 10.0, 0.0, 10.0), + (20.0, 30.0, 0.0, 10.0), + ], + [(5.0, 5.0), (25.0, 5.0)], + {}, + ) + matches = _match_files_via_json(root / "tile_bounds_tindex.json", source, target) + finally: + main_remap.get_file_bounds = original_get_file_bounds + main_remap.build_neighbor_graph_from_bounds_json = original_build + + self.assertEqual( + [(src.name, tgt.name) for src, tgt, _ in matches], + [ + ("c00_r00_segmented.laz", "c00_r00.laz"), + ("c01_r00_segmented.las", "c01_r00.las"), + ], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_main_remap_scale_preservation.py b/tests/test_main_remap_scale_preservation.py new file mode 100644 index 0000000..900d082 --- /dev/null +++ b/tests/test_main_remap_scale_preservation.py @@ -0,0 +1,77 @@ +import sys +import tempfile +import unittest +from pathlib import Path + +import laspy +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from main_remap import remap_single_tile # noqa: E402 + + +def _write_source(path: Path, offset: np.ndarray, scale: np.ndarray) -> None: + header = laspy.LasHeader(point_format=3, version="1.2") + header.offsets = offset + header.scales = scale + las = laspy.LasData(header) + las.x = np.array([1000.001, 1000.011, 1000.021]) + las.y = np.array([2000.001, 2000.011, 2000.021]) + las.z = np.array([50.001, 50.011, 50.021]) + las.add_extra_dim(laspy.ExtraBytesParams(name="PredInstance", type=np.uint16)) + las.PredInstance = np.array([1, 2, 3], dtype=np.uint16) + las.write(path) + + +def _write_target(path: Path, offset: np.ndarray, scale: np.ndarray) -> np.ndarray: + header = laspy.LasHeader(point_format=3, version="1.2") + header.offsets = offset + header.scales = scale + las = laspy.LasData(header) + expected_xyz = np.array( + [ + [1000.001, 2000.001, 50.001], + [1000.011, 2000.011, 50.011], + [1000.021, 2000.021, 50.021], + ] + ) + las.x = expected_xyz[:, 0] + las.y = expected_xyz[:, 1] + las.z = expected_xyz[:, 2] + las.write(path) + return expected_xyz + + +class RemapScalePreservationTests(unittest.TestCase): + def test_output_scales_do_not_move_target_coordinates(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + source = root / "source.las" + target = root / "target.las" + output = root / "out.laz" + offset = np.array([1000.0, 2000.0, 50.0]) + + _write_source(source, offset, np.array([0.001, 0.001, 0.001])) + expected_xyz = _write_target(target, offset, np.array([0.001, 0.001, 0.001])) + + _, success, message, point_count = remap_single_tile( + source, + target, + output, + instance_dimension="PredInstance", + output_scales=(0.01, 0.01, 0.01), + ) + + self.assertTrue(success, message) + self.assertEqual(point_count, len(expected_xyz)) + remapped = laspy.read(output) + actual_xyz = np.vstack([remapped.x, remapped.y, remapped.z]).T + np.testing.assert_allclose(actual_xyz, expected_xyz, atol=0.005) + np.testing.assert_allclose(remapped.header.scales, np.array([0.01, 0.01, 0.01])) + np.testing.assert_array_equal(remapped.PredInstance, np.array([1, 2, 3], dtype=np.uint16)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_main_subsample_bounds.py b/tests/test_main_subsample_bounds.py new file mode 100644 index 0000000..c9ae528 --- /dev/null +++ b/tests/test_main_subsample_bounds.py @@ -0,0 +1,90 @@ +import json +import subprocess +import sys +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from main_subsample import get_file_bounds # noqa: E402 + + +class PdalBoundsParsingTests(unittest.TestCase): + def test_get_file_bounds_preserves_scientific_notation_values(self): + stdout = json.dumps( + { + "metadata": { + "readers.copc": { + "minx": 4.000000047e-07, + "maxx": 30.0, + "miny": "5.000000058e-07", + "maxy": "31.5", + } + } + } + ) + + with mock.patch( + "main_subsample.subprocess.run", + return_value=subprocess.CompletedProcess(["pdal"], 0, stdout=stdout, stderr=""), + ): + bounds = get_file_bounds(Path("tile.copc.laz")) + + self.assertEqual(bounds, (4.000000047e-07, 30.0, 5.000000058e-07, 31.5)) + + def test_get_file_bounds_accepts_direct_metadata_bounds(self): + stdout = json.dumps( + { + "metadata": { + "minx": 0, + "maxx": 30, + "miny": 0, + "maxy": 30, + } + } + ) + + with mock.patch( + "main_subsample.subprocess.run", + return_value=subprocess.CompletedProcess(["pdal"], 0, stdout=stdout, stderr=""), + ): + bounds = get_file_bounds(Path("tile.laz")) + + self.assertEqual(bounds, (0.0, 30.0, 0.0, 30.0)) + + def test_get_file_bounds_returns_none_for_invalid_ranges(self): + stdout = json.dumps( + { + "metadata": { + "readers.las": { + "minx": 30, + "maxx": 0, + "miny": 0, + "maxy": 30, + } + } + } + ) + + with mock.patch( + "main_subsample.subprocess.run", + return_value=subprocess.CompletedProcess(["pdal"], 0, stdout=stdout, stderr=""), + ): + bounds = get_file_bounds(Path("tile.laz")) + + self.assertIsNone(bounds) + + def test_get_file_bounds_returns_none_for_invalid_json(self): + with mock.patch( + "main_subsample.subprocess.run", + return_value=subprocess.CompletedProcess(["pdal"], 0, stdout="not-json", stderr=""), + ): + bounds = get_file_bounds(Path("tile.laz")) + + self.assertIsNone(bounds) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_main_subsample_methods.py b/tests/test_main_subsample_methods.py new file mode 100644 index 0000000..bd33163 --- /dev/null +++ b/tests/test_main_subsample_methods.py @@ -0,0 +1,338 @@ +import sys +import tempfile +import types +import unittest +from unittest import mock +from pathlib import Path + +import laspy +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from main_subsample import ( # noqa: E402 + SUBSAMPLING_METHOD_CENTER_OF_MASS, + SUBSAMPLING_METHOD_NEAREST_TO_CENTROID, + _aggregate_center_of_mass_xyz, + _iter_copc_center_of_mass_windows, + _subsample_input_files, + _voxel_subsampling_filter, + _write_center_of_mass_points, + center_of_mass_subsample_las, + normalize_subsampling_method, +) +import main_subsample # noqa: E402 +import subsample_com # noqa: E402 + + +class FakePoints: + def __init__(self, x, y, z): + self.x = x + self.y = y + self.z = z + + def __len__(self): + return len(self.x) + + +def _write_test_las(path: Path) -> None: + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([0.0, 0.0, 0.0]) + + las = laspy.LasData(header) + las.x = np.array([0.0, 0.02, 0.08, 0.12]) + las.y = np.array([0.0, 0.0, 0.0, 0.0]) + las.z = np.array([0.0, 0.0, 0.0, 0.0]) + las.intensity = np.array([10, 20, 80, 120], dtype=np.uint16) + las.classification = np.array([1, 2, 8, 12], dtype=np.uint8) + las.add_extra_dim(laspy.ExtraBytesParams(name="PredInstance", type=np.uint16)) + las.PredInstance = np.array([10, 20, 80, 120], dtype=np.uint16) + las.write(path) + + +class SubsamplingMethodTests(unittest.TestCase): + def test_normalize_subsampling_method_defaults_to_center_of_mass(self): + self.assertEqual(normalize_subsampling_method(None), SUBSAMPLING_METHOD_CENTER_OF_MASS) + self.assertEqual(normalize_subsampling_method("com"), SUBSAMPLING_METHOD_CENTER_OF_MASS) + self.assertEqual(normalize_subsampling_method("centroid"), SUBSAMPLING_METHOD_NEAREST_TO_CENTROID) + + def test_nearest_to_centroid_uses_existing_pdal_filter(self): + self.assertEqual( + _voxel_subsampling_filter(0.1, "nearest-to-centroid"), + {"type": "filters.voxelcentroidnearestneighbor", "cell": 0.1}, + ) + + def test_center_of_mass_averages_xyz_only_and_preserves_selected_attributes(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + src = tmp_path / "source.las" + out = tmp_path / "out.las" + _write_test_las(src) + + count = center_of_mass_subsample_las(src, out, 0.1, dimension_reduction=False) + + self.assertEqual(count, 2) + result = laspy.read(out) + self.assertTrue(np.allclose(result.x, [0.033, 0.12], atol=0.001)) + self.assertTrue(np.allclose(result.y, [0.0, 0.0], atol=0.001)) + self.assertTrue(np.allclose(result.z, [0.0, 0.0], atol=0.001)) + # Non-XYZ dimensions come from the real point nearest to the averaged XYZ. + self.assertEqual(list(result.intensity), [20, 120]) + self.assertEqual(list(result.classification), [2, 12]) + self.assertEqual(list(result.PredInstance), [20, 120]) + + def test_center_of_mass_dimension_reduction_drops_extra_dimensions(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + src = tmp_path / "source.las" + out = tmp_path / "out.las" + _write_test_las(src) + + center_of_mass_subsample_las(src, out, 0.1, dimension_reduction=True) + + result = laspy.read(out) + self.assertEqual(result.header.point_format.id, 0) + self.assertNotIn("PredInstance", set(result.point_format.dimension_names)) + + def test_center_of_mass_xyz_aggregates_without_non_coordinate_attributes(self): + points = FakePoints( + x=np.array([0.0, 0.02, 0.11]), + y=np.array([0.0, 0.02, 0.0]), + z=np.array([0.0, 0.02, 0.0]), + ) + + centers = _aggregate_center_of_mass_xyz(points, 0.1) + + self.assertEqual(len(centers), 2) + self.assertTrue(any(np.allclose(center, [0.01, 0.01, 0.01]) for center in centers)) + self.assertTrue(any(np.allclose(center, [0.11, 0.0, 0.0]) for center in centers)) + + def test_copc_center_of_mass_windows_are_voxel_aligned_and_half_open(self): + header = types.SimpleNamespace( + x_min=0.0, + x_max=0.4, + y_min=0.0, + y_max=0.2, + z_min=0.0, + z_max=1.0, + scales=np.array([0.01, 0.01, 0.01]), + ) + + with mock.patch.object(subsample_com, "COPC_COM_TARGET_WINDOW_CELLS", 1): + windows = list(_iter_copc_center_of_mass_windows(header, 0.2)) + + self.assertEqual(len(windows), 2) + self.assertTrue(np.allclose(windows[0].mins, [0.0, 0.0, 0.0])) + self.assertTrue(np.allclose(windows[0].maxs, [0.195, 0.2, 1.0])) + self.assertTrue(np.allclose(windows[1].mins, [0.2, 0.0, 0.0])) + self.assertTrue(np.allclose(windows[1].maxs, [0.4, 0.2, 1.0])) + + def test_copc_xyz_center_of_mass_bypasses_stripe_chunking(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_file = tmp_path / "tile.copc.laz" + output_file = tmp_path / "tile_subsampled_20cm.laz" + + with mock.patch.object( + main_subsample, + "center_of_mass_subsample_copc", + return_value=42, + ) as optimized: + result = main_subsample.subsample_single_file( + ( + input_file, + output_file, + 0.2, + tmp_path, + 8, + True, + "center-of-mass", + ) + ) + + self.assertEqual(result, (input_file.name, True, "Success", 42)) + optimized.assert_called_once_with(input_file, output_file, 0.2, num_workers=8) + + def test_subsample_parallel_uses_copc_extension_when_requested(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "tiles" + output_dir = tmp_path / "subsampled" + input_dir.mkdir() + (input_dir / "tile.copc.laz").write_bytes(b"placeholder") + + def fake_subsample(args): + output_file = args[1] + output_copc = args[-1] + output_file.write_bytes(b"placeholder") + return (args[0].name, True, "Success", 1 if output_copc else 0) + + with mock.patch.object(main_subsample, "subsample_single_file", side_effect=fake_subsample) as worker: + outputs = main_subsample.subsample_parallel( + input_dir=input_dir, + output_dir=output_dir, + resolution=0.01, + num_cores=1, + num_threads=1, + output_copc=True, + ) + + self.assertEqual([path.name for path in outputs], ["tile_subsampled_1cm.copc.laz"]) + self.assertTrue(worker.call_args.args[0][-1]) + + def test_subsample_parallel_strips_previous_subsampled_cm_suffix(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "subsampled_res1" + output_dir = tmp_path / "subsampled_res2" + input_dir.mkdir() + (input_dir / "tile_subsampled_1cm.copc.laz").write_bytes(b"placeholder") + + def fake_subsample(args): + output_file = args[1] + output_file.write_bytes(b"placeholder") + return (args[0].name, True, "Success", 1) + + with mock.patch.object(main_subsample, "subsample_single_file", side_effect=fake_subsample): + outputs = main_subsample.subsample_parallel( + input_dir=input_dir, + output_dir=output_dir, + resolution=0.1, + num_cores=1, + num_threads=1, + output_copc=False, + ) + + self.assertEqual([path.name for path in outputs], ["tile_subsampled_10cm.laz"]) + self.assertFalse((output_dir / "tile_subsampled_subsampled_10cm.laz").exists()) + + def test_subsample_parallel_manifest_skips_matching_rerun(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "tiles" + output_dir = tmp_path / "subsampled" + input_dir.mkdir() + (input_dir / "tile.laz").write_bytes(b"placeholder") + + def fake_subsample(args): + output_file = args[1] + output_file.write_bytes(b"subsampled") + return (args[0].name, True, "Success", 1) + + with mock.patch.object(main_subsample, "subsample_single_file", side_effect=fake_subsample) as worker: + first = main_subsample.subsample_parallel( + input_dir=input_dir, + output_dir=output_dir, + resolution=0.01, + num_cores=1, + num_threads=1, + output_copc=False, + subsampling_method="center-of-mass", + ) + second = main_subsample.subsample_parallel( + input_dir=input_dir, + output_dir=output_dir, + resolution=0.01, + num_cores=1, + num_threads=1, + output_copc=False, + subsampling_method="center-of-mass", + ) + + self.assertEqual([path.name for path in first], ["tile_subsampled_1cm.laz"]) + self.assertEqual([path.name for path in second], ["tile_subsampled_1cm.laz"]) + self.assertEqual(worker.call_count, 1) + + def test_subsample_parallel_rebuilds_when_method_changes(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_dir = tmp_path / "tiles" + output_dir = tmp_path / "subsampled" + input_dir.mkdir() + (input_dir / "tile.laz").write_bytes(b"placeholder") + + writes = [] + + def fake_subsample(args): + output_file = args[1] + method = args[6] + writes.append(method) + output_file.write_bytes(method.encode("ascii")) + return (args[0].name, True, "Success", 1) + + with mock.patch.object(main_subsample, "subsample_single_file", side_effect=fake_subsample): + main_subsample.subsample_parallel( + input_dir=input_dir, + output_dir=output_dir, + resolution=0.01, + num_cores=1, + num_threads=1, + output_copc=False, + subsampling_method="center-of-mass", + ) + main_subsample.subsample_parallel( + input_dir=input_dir, + output_dir=output_dir, + resolution=0.01, + num_cores=1, + num_threads=1, + output_copc=False, + subsampling_method="nearest-to-centroid", + ) + + self.assertEqual(writes, ["center-of-mass", "nearest-to-centroid"]) + self.assertEqual((output_dir / "tile_subsampled_1cm.laz").read_bytes(), b"nearest-to-centroid") + + def test_subsample_input_files_include_mixed_laz_and_las(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) + (input_dir / "first.laz").write_bytes(b"placeholder") + (input_dir / "second.las").write_bytes(b"placeholder") + + files = [path.name for path in _subsample_input_files(input_dir)] + + self.assertEqual(files, ["first.laz", "second.las"]) + + def test_subsample_input_files_prefer_copc_twin(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) + (input_dir / "tile.laz").write_bytes(b"placeholder") + (input_dir / "tile.copc.laz").write_bytes(b"placeholder") + + files = [path.name for path in _subsample_input_files(input_dir)] + + self.assertEqual(files, ["tile.copc.laz"]) + + def test_center_of_mass_points_can_be_written_in_streamed_batches(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + out = tmp_path / "streamed.laz" + header = laspy.LasHeader(point_format=0, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([0.0, 0.0, 0.0]) + + with laspy.open(str(out), mode="w", header=header, do_compress=True) as writer: + first = _write_center_of_mass_points( + writer, + header, + np.array([[0.01, 0.02, 0.03], [0.04, 0.05, 0.06]]), + ) + second = _write_center_of_mass_points( + writer, + header, + np.array([[0.07, 0.08, 0.09]]), + ) + + result = laspy.read(out) + self.assertEqual(first, 2) + self.assertEqual(second, 1) + self.assertEqual(len(result.points), 3) + self.assertTrue(np.allclose(result.x, [0.01, 0.04, 0.07])) + self.assertTrue(np.allclose(result.y, [0.02, 0.05, 0.08])) + self.assertTrue(np.allclose(result.z, [0.03, 0.06, 0.09])) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_main_tile_crs_validation.py b/tests/test_main_tile_crs_validation.py new file mode 100644 index 0000000..9dbc1e5 --- /dev/null +++ b/tests/test_main_tile_crs_validation.py @@ -0,0 +1,194 @@ +import sys +import tempfile +import unittest +from pathlib import Path + +import laspy +import numpy as np +from laspy.vlrs.vlr import VLR + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from copc_metadata import ( # noqa: E402 + append_source_geotiff_projection_evlrs, + copc_preserves_source_crs, + crs_authority_string, + crs_equivalent, +) + + +class _FakeCrs: + def __init__(self, authority=None, epsg=None, wkt="FAKE_WKT", equals_result=None): + self.authority = authority + self.epsg = epsg + self.wkt = wkt + self.equals_result = equals_result + + def to_authority(self): + return self.authority + + def to_epsg(self): + return self.epsg + + def equals(self, other, ignore_axis_order=False): + return self.equals_result if self.equals_result is not None else False + + def to_wkt(self): + return self.wkt + + +class _FakeHeader: + def __init__(self, crs): + self.crs = crs + + def parse_crs(self): + return self.crs + + +def _write_las(path: Path, projection_payload: bytes | None, record_id: int = 2112) -> None: + header = laspy.LasHeader(point_format=6, version="1.4") + header.scales = np.array([0.01, 0.01, 0.01]) + header.offsets = np.array([0.0, 0.0, 0.0]) + if projection_payload is not None: + header.vlrs.append( + VLR( + user_id="LASF_Projection", + record_id=record_id, + description="synthetic CRS metadata", + record_data=projection_payload, + ) + ) + + las = laspy.LasData(header) + las.x = np.array([0.0]) + las.y = np.array([0.0]) + las.z = np.array([0.0]) + las.write(path) + + +class CopcCrsValidationTests(unittest.TestCase): + def test_crs_authority_string_prefers_authority_code(self): + header = _FakeHeader(_FakeCrs(authority=("EPSG", "32632"), epsg=32633)) + + self.assertEqual(crs_authority_string(header), "EPSG:32632") + + def test_crs_authority_string_falls_back_to_epsg_code(self): + header = _FakeHeader(_FakeCrs(authority=None, epsg=32632)) + + self.assertEqual(crs_authority_string(header), "EPSG:32632") + + def test_crs_equivalent_accepts_same_authority_with_different_wkt(self): + source = _FakeCrs(authority=("EPSG", "4978"), wkt="LONG_WKT") + output = _FakeCrs(authority=("EPSG", "4978"), wkt="SHORT_WKT") + + self.assertTrue(crs_equivalent(source, output)) + + def test_crs_equivalent_accepts_pyproj_equals_match(self): + source = _FakeCrs(wkt="LONG_WKT", equals_result=True) + output = _FakeCrs(wkt="SHORT_WKT") + + self.assertTrue(crs_equivalent(source, output)) + + def test_accepts_matching_projection_vlr(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + payload = b'LOCAL_CS["3dtrees-test"]' + _write_las(source, payload) + _write_las(output, payload) + + ok, message = copc_preserves_source_crs(source, output) + + self.assertTrue(ok, message) + + def test_rejects_missing_projection_vlr(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + _write_las(source, b"\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x04\x00\x01\x00r\x13", record_id=34735) + _write_las(output, None) + + ok, message = copc_preserves_source_crs(source, output) + + self.assertFalse(ok) + self.assertIn("missing or changed", message) + + def test_appends_source_geokey_vlr_as_output_evlr(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + _write_las( + source, + b"\x01\x00\x01\x00\x00\x00\x01\x00\x00\x00\x04\x00\x01\x00r\x13", + record_id=34735, + ) + _write_las(output, None) + + ok, message = append_source_geotiff_projection_evlrs(source, output) + self.assertTrue(ok, message) + ok, message = copc_preserves_source_crs(source, output) + + self.assertTrue(ok, message) + + def test_appends_source_wkt_projection_vlr_as_output_evlr(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + _write_las(source, b'PROJCS["source-original"]', record_id=2112) + _write_las(output, b'PROJCS["writer-normalized"]', record_id=2112) + + ok, message = append_source_geotiff_projection_evlrs(source, output) + self.assertTrue(ok, message) + ok, message = copc_preserves_source_crs(source, output) + + self.assertTrue(ok, message) + + def test_does_not_append_duplicate_matching_wkt_projection_vlr(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + payload = b'PROJCS["source-original"]' + _write_las(source, payload, record_id=2112) + _write_las(output, payload, record_id=2112) + + ok, message = append_source_geotiff_projection_evlrs(source, output) + + self.assertTrue(ok, message) + self.assertEqual(message, "projection VLRs already preserved") + with laspy.open(output) as reader: + self.assertEqual(int(reader.header.number_of_evlrs or 0), 0) + + def test_rejects_changed_projection_vlr(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + _write_las(source, b'LOCAL_CS["3dtrees-test"]') + _write_las(output, b'LOCAL_CS["different"]') + + ok, message = copc_preserves_source_crs(source, output) + + self.assertFalse(ok) + self.assertIn("missing or changed", message) + + def test_accepts_source_without_crs(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + source = tmp_path / "source.las" + output = tmp_path / "output.las" + _write_las(source, None) + _write_las(output, None) + + ok, message = copc_preserves_source_crs(source, output) + + self.assertTrue(ok, message) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_main_tile_input_discovery.py b/tests/test_main_tile_input_discovery.py new file mode 100644 index 0000000..9fc6a31 --- /dev/null +++ b/tests/test_main_tile_input_discovery.py @@ -0,0 +1,83 @@ +import sys +import tempfile +import types +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +sys.modules.setdefault( + "plot_tiles_and_copc", + types.SimpleNamespace(plot_extents=lambda *_args, **_kwargs: None), +) +sys.modules.setdefault( + "tile_tindex", + types.SimpleNamespace( + bounds_overlap=lambda *_args, **_kwargs: False, + build_tindex=lambda *_args, **_kwargs: None, + calculate_tile_bounds=lambda *_args, **_kwargs: None, + filter_source_files_for_tile=lambda *_args, **_kwargs: [], + get_bounds=lambda *_args, **_kwargs: None, + get_pdal_path=lambda: "pdal", + get_pdal_wrench_path=lambda: "pdal_wrench", + get_source_bounds_from_tindex=lambda *_args, **_kwargs: {}, + get_source_files_from_tindex=lambda *_args, **_kwargs: [], + parse_proj_bounds=lambda *_args, **_kwargs: None, + update_tile_bounds_json_from_files=lambda *_args, **_kwargs: 0, + ), +) + +import main_tile # noqa: E402 + + +class MainTileInputDiscoveryTests(unittest.TestCase): + def test_tiling_input_files_prefer_copc_twin(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) + (input_dir / "source.laz").write_text("raw") + (input_dir / "source.copc.laz").write_text("copc") + (input_dir / "other.las").write_text("raw") + + files = [path.name for path in main_tile._tiling_input_files(input_dir)] + + self.assertEqual(files, ["other.las", "source.copc.laz"]) + + def test_small_single_copc_input_is_reused_for_skip_tiling(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_dir = root / "input" + output_dir = root / "output" + input_dir.mkdir() + output_dir.mkdir() + source = input_dir / "source.copc.laz" + source.write_bytes(b"copc") + + def fake_build_tindex(_input_dir, output_gpkg): + output_gpkg.parent.mkdir(parents=True, exist_ok=True) + output_gpkg.write_bytes(b"gpkg") + return output_gpkg + + with mock.patch.object(main_tile, "build_tindex", side_effect=fake_build_tindex): + with mock.patch.object( + main_tile, + "calculate_tile_bounds", + return_value=(output_dir / "jobs.txt", output_dir / "bounds.json", {}), + ): + with mock.patch.object(main_tile.plot_tiles_and_copc, "plot_extents"): + with mock.patch.object(main_tile, "_convert_laz_to_copc") as convert: + result = main_tile.run_tiling_pipeline( + input_dir=input_dir, + output_dir=output_dir, + tile_length=100, + tile_buffer=10, + tiling_threshold=10000, + ) + + self.assertEqual(result, output_dir / "copc_single") + self.assertEqual((result / "source.copc.laz").read_bytes(), b"copc") + convert.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_merge_tile_discovery.py b/tests/test_merge_tile_discovery.py new file mode 100644 index 0000000..4752067 --- /dev/null +++ b/tests/test_merge_tile_discovery.py @@ -0,0 +1,49 @@ +import sys +import tempfile +import unittest +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from merge_tile_loading import merge_tile_name # noqa: E402 +from merge_tiles import _merge_input_files # noqa: E402 + + +class MergeTileDiscoveryTests(unittest.TestCase): + def test_merge_input_files_include_mixed_laz_and_las(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + (root / "c00_r00_segmented_remapped.laz").write_text("placeholder") + (root / "c01_r00_segmented_remapped.las").write_text("placeholder") + + files = [path.name for path in _merge_input_files(root)] + + self.assertEqual( + files, + ["c00_r00_segmented_remapped.laz", "c01_r00_segmented_remapped.las"], + ) + + def test_merge_input_files_prefer_copc_twin(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + (root / "c00_r00_segmented_remapped.laz").write_text("placeholder") + (root / "c00_r00_segmented_remapped.copc.laz").write_text("placeholder") + + files = [path.name for path in _merge_input_files(root)] + + self.assertEqual(files, ["c00_r00_segmented_remapped.copc.laz"]) + + def test_merge_tile_name_strips_copc_and_processing_suffixes(self): + self.assertEqual( + merge_tile_name(Path("c00_r00_segmented_remapped.copc.laz")), + "c00_r00", + ) + self.assertEqual( + merge_tile_name(Path("c01_r00_segmented.las")), + "c01_r00", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_multi_collection_remap.py b/tests/test_multi_collection_remap.py new file mode 100644 index 0000000..5360939 --- /dev/null +++ b/tests/test_multi_collection_remap.py @@ -0,0 +1,418 @@ +import sys +import tempfile +import unittest +from unittest import mock +from datetime import date +from pathlib import Path + +import laspy +import numpy as np +from laspy.vlrs.vlr import VLR + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from prediction_collection_remap import ( # noqa: E402 + _copc_spatial_windows, + load_collection_subset_for_bounds, + prediction_collection_files, + remap_prediction_collections_to_original_files, + stream_add_collections_to_file, +) + + +def _base_header() -> laspy.LasHeader: + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([500.0, 600.0, 50.0]) + header.system_identifier = "synthetic-source" + header.generating_software = "unit-test" + header.creation_date = date(2026, 6, 24) + header.vlrs.append( + VLR( + user_id="LASF_Projection", + record_id=34735, + description="synthetic projection", + record_data=b"projection-payload", + ) + ) + return header + + +def _write_las(path: Path, extra_dims: dict[str, np.ndarray] | None = None) -> None: + extra_dims = extra_dims or {} + header = _base_header() + las = laspy.LasData(header) + n_points = 4 + las.x = np.array([500.0, 500.1, 500.2, 500.3]) + las.y = np.array([600.0, 600.0, 600.1, 600.1]) + las.z = np.array([50.0, 50.1, 50.2, 50.3]) + las.intensity = np.array([10, 11, 12, 13], dtype=np.uint16) + for name, values in extra_dims.items(): + values = np.asarray(values) + assert len(values) == n_points + las.add_extra_dim(laspy.ExtraBytesParams(name=name, type=values.dtype)) + setattr(las, name, values) + las.write(path) + + +def _write_shifted_prediction_las(path: Path) -> None: + header = _base_header() + las = laspy.LasData(header) + las.x = np.array([500.01, 500.11, 500.21, 500.31]) + las.y = np.array([600.01, 600.01, 600.11, 600.11]) + las.z = np.array([50.0, 50.1, 50.2, 50.3]) + las.add_extra_dim(laspy.ExtraBytesParams(name="PredInstance_Shifted", type=np.uint16)) + las.PredInstance_Shifted = np.array([1, 1, 0, 2], dtype=np.uint16) + las.write(path) + + +class MultiCollectionRemapTests(unittest.TestCase): + def test_prediction_collection_files_prefer_copc_twins(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + collection = root / "collection" + collection.mkdir() + (collection / "source.laz").write_text("placeholder") + (collection / "source.copc.laz").write_text("placeholder") + (collection / "other.las").write_text("placeholder") + + files = [path.name for path in prediction_collection_files(collection)] + + self.assertEqual(files, ["other.las", "source.copc.laz"]) + + def test_copc_spatial_windows_cover_header_extent(self): + header = mock.Mock(x_min=0.0, x_max=10.0) + self.assertEqual( + _copc_spatial_windows(header, 4), + [ + (0.0, 2.5, False), + (2.5, 5.0, False), + (5.0, 7.5, False), + (7.5, 10.0, True), + ], + ) + + def test_remaps_distinct_model_named_dims_and_preserves_source_metadata(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + sat_dir = root / "sat" + foma_dir = root / "foma" + output_dir = root / "original_with_predictions" + original_dir.mkdir() + sat_dir.mkdir() + foma_dir.mkdir() + + _write_las(original_dir / "source.las", {"OriginalExtra": np.array([1, 2, 3, 4], dtype=np.uint16)}) + _write_las( + sat_dir / "source_sat.las", + { + "PredInstance_SAT": np.array([1, 1, 0, 2], dtype=np.uint16), + "PredSemantic_SAT": np.array([1, 1, 0, 1], dtype=np.uint8), + "species_id_sat": np.array([10, 10, 0, 11], dtype=np.uint16), + "species_prob_sat": np.array([0.9, 0.8, 0.0, 0.7], dtype=np.float32), + }, + ) + _write_las( + foma_dir / "source_foma.las", + { + "PredInstance_ForestMamba": np.array([5, 5, 0, 6], dtype=np.uint32), + "PredSemantic_ForestMamba": np.array([1, 1, 0, 1], dtype=np.uint8), + "ForestMambaConfidence": np.array([0.5, 0.6, 0.0, 0.7], dtype=np.float32), + "species_id_foma": np.array([20, 20, 0, 21], dtype=np.uint16), + "species_prob_foma": np.array([0.4, 0.5, 0.0, 0.6], dtype=np.float32), + }, + ) + + remap_prediction_collections_to_original_files( + [sat_dir, foma_dir], + original_dir, + output_dir, + tolerance=0.001, + num_threads=1, + ) + + out = laspy.read(output_dir / "source.las") + self.assertEqual(len(out), 4) + self.assertFalse(any(output_dir.glob("*.collection_*.laz"))) + out_dims = set(out.point_format.dimension_names) | {dim.name for dim in out.point_format.extra_dimensions} + for name in ( + "OriginalExtra", + "PredInstance_SAT", + "PredSemantic_SAT", + "species_id_sat", + "species_prob_sat", + "PredInstance_ForestMamba", + "PredSemantic_ForestMamba", + "ForestMambaConfidence", + "species_id_foma", + "species_prob_foma", + ): + self.assertIn(name, out_dims) + + np.testing.assert_array_equal(out.PredInstance_SAT, np.array([1, 1, 0, 2], dtype=np.uint16)) + np.testing.assert_array_equal(out.PredInstance_ForestMamba, np.array([5, 5, 0, 6], dtype=np.uint32)) + np.testing.assert_array_equal(out.OriginalExtra, np.array([1, 2, 3, 4], dtype=np.uint16)) + self.assertEqual(out.header.system_identifier, "synthetic-source") + self.assertEqual(out.header.generating_software, "unit-test") + self.assertEqual(out.header.creation_date, date(2026, 6, 24)) + self.assertTrue(any(v.user_id == "LASF_Projection" and v.record_id == 34735 for v in out.header.vlrs)) + + def test_duplicate_prediction_dim_names_fail_before_output(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + a_dir = root / "a" + b_dir = root / "b" + output_dir = root / "out" + original_dir.mkdir() + a_dir.mkdir() + b_dir.mkdir() + + _write_las(original_dir / "source.las") + _write_las(a_dir / "a.las", {"PredInstance_SAT": np.array([1, 1, 0, 2], dtype=np.uint16)}) + _write_las(b_dir / "b.las", {"PredInstance_SAT": np.array([5, 5, 0, 6], dtype=np.uint16)}) + + with self.assertRaisesRegex(ValueError, "Duplicate prediction dimension name"): + remap_prediction_collections_to_original_files( + [a_dir, b_dir], + original_dir, + output_dir, + tolerance=0.001, + num_threads=1, + ) + self.assertFalse(output_dir.exists()) + + def test_rerun_reprocesses_stale_existing_output_missing_selected_dims(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + sat_dir = root / "sat" + output_dir = root / "out" + original_dir.mkdir() + sat_dir.mkdir() + output_dir.mkdir() + + _write_las(original_dir / "source.las") + _write_las(output_dir / "source.las") + _write_las( + sat_dir / "source_sat.las", + {"PredInstance_SAT": np.array([1, 1, 0, 2], dtype=np.uint16)}, + ) + + remap_prediction_collections_to_original_files( + [sat_dir], + original_dir, + output_dir, + tolerance=0.001, + num_threads=1, + ) + + out = laspy.read(output_dir / "source.las") + dims = set(out.point_format.dimension_names) | {dim.name for dim in out.point_format.extra_dimensions} + self.assertIn("PredInstance_SAT", dims) + np.testing.assert_array_equal(out.PredInstance_SAT, np.array([1, 1, 0, 2], dtype=np.uint16)) + + def test_incomplete_spatial_match_fails(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + shifted_dir = root / "shifted" + output_dir = root / "out" + original_dir.mkdir() + shifted_dir.mkdir() + + _write_las(original_dir / "source.las") + _write_shifted_prediction_las(shifted_dir / "shifted.las") + + with self.assertRaisesRegex(RuntimeError, "matched 0/4 points"): + remap_prediction_collections_to_original_files( + [shifted_dir], + original_dir, + output_dir, + tolerance=0.001, + num_threads=1, + ) + + def test_stream_remap_honors_caller_spatial_buffer(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_file = root / "source.las" + output_file = root / "out.las" + _write_las(input_file) + + collection_meta = [{ + "path": root / "collection", + "dims": ["PredInstance_Buffered"], + "extra_params": { + "PredInstance_Buffered": laspy.ExtraBytesParams( + name="PredInstance_Buffered", + type=np.uint16, + ) + }, + }] + source_points = np.column_stack([ + np.array([500.0, 500.1, 500.2, 500.3]), + np.array([600.0, 600.0, 600.1, 600.1]), + np.array([50.0, 50.1, 50.2, 50.3]), + ]) + captured_buffers = [] + + def load_subset(_, __, spatial_buffer): + captured_buffers.append(spatial_buffer) + return ( + source_points, + {"PredInstance_Buffered": np.array([1, 1, 0, 2], dtype=np.uint16)}, + ) + + with mock.patch( + "prediction_collection_remap.load_collection_subset_for_bounds", + side_effect=load_subset, + ): + stream_add_collections_to_file( + input_file, + output_file, + collection_meta, + spatial_buffer=3.0, + tolerance=0.001, + chunk_size=10, + kdtree_workers=1, + ) + + self.assertEqual(captured_buffers, [3.0]) + + def test_copc_original_routes_through_spatial_query_fast_path(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + pred_dir = root / "pred" + output_dir = root / "out" + original_dir.mkdir() + pred_dir.mkdir() + + copc_original = original_dir / "source.copc.laz" + raw_twin = original_dir / "source.laz" + copc_original.write_text("placeholder") + raw_twin.write_text("placeholder") + _write_las( + pred_dir / "source_pred.las", + {"PredInstance_Model": np.array([1, 1, 0, 2], dtype=np.uint16)}, + ) + + fake_header = mock.Mock(point_count=4) + fake_reader = mock.Mock() + fake_reader.__enter__ = mock.Mock(return_value=fake_reader) + fake_reader.__exit__ = mock.Mock(return_value=False) + fake_reader.header = fake_header + + real_laspy_open = laspy.open + + def open_side_effect(path, *args, **kwargs): + if Path(path).name == "source.copc.laz": + return fake_reader + return real_laspy_open(path, *args, **kwargs) + + with mock.patch("prediction_collection_remap.laspy.open", side_effect=open_side_effect): + with mock.patch( + "prediction_collection_remap.stream_add_collections_to_copc_file_spatial", + return_value=(4, 4), + ) as fast_path: + remap_prediction_collections_to_original_files( + [pred_dir], + original_dir, + output_dir, + tolerance=0.001, + num_threads=1, + num_spatial_chunks=7, + ) + + fast_path.assert_called_once() + args, kwargs = fast_path.call_args + self.assertEqual(args[0], copc_original) + self.assertEqual(args[1], output_dir / "source.laz") + self.assertEqual(kwargs["num_spatial_chunks"], 7) + + def test_raw_original_mode_ignores_matching_copc_twin(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + pred_dir = root / "pred" + output_dir = root / "out" + original_dir.mkdir() + pred_dir.mkdir() + + raw_original = original_dir / "source.las" + copc_twin = original_dir / "source.copc.laz" + _write_las(raw_original, {"OriginalExtra": np.array([1, 2, 3, 4], dtype=np.uint16)}) + copc_twin.write_text("placeholder") + _write_las( + pred_dir / "source_pred.las", + {"PredInstance_Model": np.array([1, 1, 0, 2], dtype=np.uint16)}, + ) + + with mock.patch( + "prediction_collection_remap.stream_add_collections_to_file", + return_value=(4, 4), + ) as raw_path: + with mock.patch( + "prediction_collection_remap.stream_add_collections_to_copc_file_spatial", + ) as copc_path: + remap_prediction_collections_to_original_files( + [pred_dir], + original_dir, + output_dir, + tolerance=0.001, + num_threads=1, + prefer_copc_sources=False, + ) + + raw_path.assert_called_once() + copc_path.assert_not_called() + args, _ = raw_path.call_args + self.assertEqual(args[0], raw_original) + self.assertEqual(args[1], output_dir / "source.las") + + def test_copc_prediction_subset_uses_spatial_query(self): + root = Path("/tmp") + fake_points = mock.MagicMock() + fake_points.__len__.return_value = 4 + fake_points.x = np.array([500.0, 500.1, 500.2, 500.3]) + fake_points.y = np.array([600.0, 600.0, 600.1, 600.1]) + fake_points.z = np.array([50.0, 50.1, 50.2, 50.3]) + fake_points.__getitem__.side_effect = lambda key: { + "PredInstance_Model": np.array([1, 1, 0, 2], dtype=np.uint16), + }[key] + + fake_reader = mock.Mock() + fake_reader.__enter__ = mock.Mock(return_value=fake_reader) + fake_reader.__exit__ = mock.Mock(return_value=False) + fake_reader.spatial_query.return_value = fake_points + + coll_meta = { + "path": root / "collection", + "files": [{ + "path": root / "source.copc.laz", + "bounds": (499.0, 501.0, 599.0, 601.0), + "z_bounds": (49.0, 51.0), + }], + "dims": ["PredInstance_Model"], + } + + with mock.patch("prediction_collection_remap.laspy.CopcReader.open", return_value=fake_reader): + points, dims = load_collection_subset_for_bounds( + coll_meta, + (500.0, 500.3, 600.0, 600.1), + spatial_buffer=0.01, + ) + + fake_reader.spatial_query.assert_called_once() + self.assertEqual(points.shape, (4, 3)) + np.testing.assert_array_equal( + dims["PredInstance_Model"], + np.array([1, 1, 0, 2], dtype=np.uint16), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_no_private_defaults.py b/tests/test_no_private_defaults.py new file mode 100644 index 0000000..f7e7aa4 --- /dev/null +++ b/tests/test_no_private_defaults.py @@ -0,0 +1,43 @@ +import sys +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +import prepare_tile_jobs # noqa: E402 +from prepare_tile_jobs import DEFAULT_BOUNDS_JSON # noqa: E402 + + +class NoPrivateDefaultsTests(unittest.TestCase): + def test_helper_cli_defaults_are_workspace_relative(self): + self.assertEqual(DEFAULT_BOUNDS_JSON, Path("tile_bounds_tindex.json")) + self.assertFalse(DEFAULT_BOUNDS_JSON.is_absolute()) + + def test_tool_source_has_no_machine_specific_default_paths(self): + src_dir = Path(__file__).resolve().parents[1] / "src" + source_text = "\n".join(path.read_text(encoding="utf-8") for path in src_dir.glob("*.py")) + self.assertNotIn("/home/kg281", source_text) + self.assertNotIn("pdal_experiments", source_text) + + def test_prepare_tile_jobs_accepts_grid_offset_from_tindex_caller(self): + argv = [ + "prepare_tile_jobs.py", + "input.gpkg", + "--tile-length=300", + "--tile-buffer=20", + "--jobs-out=jobs.txt", + "--bounds-out=bounds.json", + "--grid-offset=1.0", + ] + with mock.patch.object(sys, "argv", argv): + with mock.patch.object(prepare_tile_jobs, "run_get_bounds", return_value={"tile_count": "1"}): + with mock.patch.object(prepare_tile_jobs, "write_job_list") as write_jobs: + prepare_tile_jobs.main() + + write_jobs.assert_called_once_with(Path("bounds.json"), Path("jobs.txt")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_output_remap.py b/tests/test_output_remap.py new file mode 100644 index 0000000..ee6a4f9 --- /dev/null +++ b/tests/test_output_remap.py @@ -0,0 +1,461 @@ +import sys +import tempfile +import unittest +from unittest import mock +from pathlib import Path + +import laspy +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from output_remap import ( # noqa: E402 + remap_merged_file_to_original_input_files, + remap_to_original_input_files, + retile_to_original_files, +) + + +def _write_original(path: Path) -> np.ndarray: + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([100.0, 200.0, 50.0]) + las = laspy.LasData(header) + points = np.array( + [ + [100.0, 200.0, 50.0], + [100.1, 200.1, 50.1], + [100.2, 200.2, 50.2], + ], + dtype=np.float64, + ) + las.x = points[:, 0] + las.y = points[:, 1] + las.z = points[:, 2] + las.intensity = np.array([10, 20, 30], dtype=np.uint16) + las.add_extra_dim(laspy.ExtraBytesParams(name="OriginalExtra", type=np.uint16)) + las.OriginalExtra = np.array([1, 2, 3], dtype=np.uint16) + las.write(path) + return points + + +def _write_merged_with_predictions(path: Path, points: np.ndarray) -> None: + header = laspy.LasHeader(point_format=3, version="1.2") + header.scales = np.array([0.001, 0.001, 0.001]) + header.offsets = np.array([100.0, 200.0, 50.0]) + las = laspy.LasData(header) + las.x = points[:, 0] + las.y = points[:, 1] + las.z = points[:, 2] + las.add_extra_dim(laspy.ExtraBytesParams(name="PredInstance", type=np.uint16)) + las.PredInstance = np.array([1, 0, 2], dtype=np.uint16) + las.write(path) + + +class _FakeCopcRecord: + def __init__(self, points: np.ndarray, pred_instance: np.ndarray): + self.x = points[:, 0] + self.y = points[:, 1] + self.z = points[:, 2] + self._dims = {"PredInstance": pred_instance} + + def __len__(self): + return len(self.x) + + def __getitem__(self, name): + return self._dims[name] + + +class _FakeCopcReader: + def __init__(self, header, points: np.ndarray, pred_instance: np.ndarray): + self.header = header + self.points = points + self.pred_instance = pred_instance + self.query_count = 0 + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def spatial_query(self, bounds): + self.query_count += 1 + return _FakeCopcRecord(self.points, self.pred_instance) + + +class OutputRemapTests(unittest.TestCase): + def test_remap_to_original_preserves_original_dims_and_adds_branded_predictions(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + points = _write_original(original_dir / "source.las") + + remap_to_original_input_files( + merged_points=points, + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + "PredSemantic": np.array([1, 0, 1], dtype=np.uint8), + "IgnoredDim": np.array([9, 9, 9], dtype=np.uint16), + }, + merged_extra_dim_params=None, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance", "PredSemantic"], + threedtrees_suffix="SAT", + ) + + out = laspy.read(output_dir / "source.las") + dims = set(out.point_format.dimension_names) | {dim.name for dim in out.point_format.extra_dimensions} + self.assertIn("OriginalExtra", dims) + self.assertIn("PredInstance_SAT", dims) + self.assertIn("PredSemantic_SAT", dims) + self.assertNotIn("IgnoredDim_SAT", dims) + np.testing.assert_array_equal(out.intensity, np.array([10, 20, 30], dtype=np.uint16)) + np.testing.assert_array_equal(out.OriginalExtra, np.array([1, 2, 3], dtype=np.uint16)) + np.testing.assert_array_equal(out.PredInstance_SAT, np.array([1, 0, 2], dtype=np.uint16)) + np.testing.assert_array_equal(out.PredSemantic_SAT, np.array([1, 0, 1], dtype=np.uint8)) + + def test_remap_to_original_preserves_prediction_extra_byte_metadata(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + points = _write_original(original_dir / "source.las") + + remap_to_original_input_files( + merged_points=points, + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + }, + merged_extra_dim_params={ + "PredInstance": laspy.ExtraBytesParams( + name="PredInstance", + type=np.uint16, + description="model instance id", + ), + }, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + ) + + with laspy.open(output_dir / "source.las") as reader: + descriptions = { + dim.name: dim.description + for dim in reader.header.point_format.extra_dimensions + } + self.assertEqual(descriptions["PredInstance_SAT"], "model instance id") + + def test_remap_to_original_reprocesses_stale_existing_output(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + output_dir.mkdir() + points = _write_original(original_dir / "source.las") + _write_original(output_dir / "source.las") + + remap_to_original_input_files( + merged_points=points, + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + "PredSemantic": np.array([1, 0, 1], dtype=np.uint8), + }, + merged_extra_dim_params=None, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance", "PredSemantic"], + threedtrees_suffix="SAT", + ) + + out = laspy.read(output_dir / "source.las") + dims = set(out.point_format.dimension_names) | {dim.name for dim in out.point_format.extra_dimensions} + self.assertIn("PredInstance_SAT", dims) + self.assertIn("PredSemantic_SAT", dims) + + def test_existing_output_reuse_respects_prediction_name_collisions(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + output_dir.mkdir() + source_file = original_dir / "source.las" + points = _write_original(source_file) + + source = laspy.read(source_file) + source.add_extra_dim(laspy.ExtraBytesParams(name="PredInstance_SAT", type=np.uint16)) + source.PredInstance_SAT = np.array([99, 99, 99], dtype=np.uint16) + source.write(source_file) + source.write(output_dir / "source.las") + + remap_to_original_input_files( + merged_points=points, + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + }, + merged_extra_dim_params=None, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + ) + + out = laspy.read(output_dir / "source.las") + dims = set(out.point_format.dimension_names) | {dim.name for dim in out.point_format.extra_dimensions} + self.assertIn("PredInstance_SAT", dims) + self.assertIn("PredInstance_SAT_1", dims) + np.testing.assert_array_equal(out.PredInstance_SAT, np.array([99, 99, 99], dtype=np.uint16)) + np.testing.assert_array_equal(out.PredInstance_SAT_1, np.array([1, 0, 2], dtype=np.uint16)) + + def test_remap_to_original_fails_when_nearest_points_exceed_tolerance(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + points = _write_original(original_dir / "source.las") + shifted_points = points + np.array([0.01, 0.0, 0.0]) + + with self.assertRaisesRegex(RuntimeError, "matched 0/3"): + remap_to_original_input_files( + merged_points=shifted_points, + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + }, + merged_extra_dim_params=None, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + ) + + self.assertFalse((output_dir / "source.las").exists()) + + def test_retile_to_original_fails_when_nearest_points_exceed_tolerance(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "original_tiles" + output_dir = root / "out" + original_dir.mkdir() + points = _write_original(original_dir / "tile.las") + shifted_points = points + np.array([0.01, 0.0, 0.0]) + + with self.assertRaisesRegex(RuntimeError, "matched 0/3"): + retile_to_original_files( + merged_points=shifted_points, + merged_instances=np.array([1, 0, 2], dtype=np.uint16), + merged_extra_dims={}, + merged_extra_dim_params=None, + original_tiles_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + parallel_tiles=1, + instance_dimension="PredInstance", + ) + + self.assertFalse((output_dir / "tile.las").exists()) + + def test_retile_to_original_includes_mixed_laz_and_las_inputs(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "original_tiles" + output_dir = root / "out" + original_dir.mkdir() + (original_dir / "tile_a.laz").write_text("placeholder") + (original_dir / "tile_b.las").write_text("placeholder") + + import output_remap + + calls = [] + original_process = output_remap._process_single_tile + + def fake_process(args): + calls.append(args) + return (args[0].name, 1, 1, 1, True, "OK") + + try: + output_remap._process_single_tile = fake_process + retile_to_original_files( + merged_points=np.array([[0.0, 0.0, 0.0]], dtype=np.float64), + merged_instances=np.array([1], dtype=np.uint16), + merged_extra_dims={}, + merged_extra_dim_params=None, + original_tiles_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + parallel_tiles=1, + instance_dimension="PredInstance", + ) + finally: + output_remap._process_single_tile = original_process + + self.assertEqual( + [args[0].name for args in calls], + ["tile_a.laz", "tile_b.las"], + ) + + def test_legacy_remap_uses_copc_spatial_fast_path_for_copc_original(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + copc_original = original_dir / "source.copc.laz" + raw_twin = original_dir / "source.laz" + copc_original.write_text("placeholder") + raw_twin.write_text("placeholder") + + with mock.patch( + "output_remap._process_single_original_copc_file", + return_value=("source.copc.laz", 3, 3, 2, True, "Success"), + ) as fast_path: + remap_to_original_input_files( + merged_points=np.array( + [ + [100.0, 200.0, 50.0], + [100.1, 200.1, 50.1], + [100.2, 200.2, 50.2], + ], + dtype=np.float64, + ), + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + }, + merged_extra_dim_params=None, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + num_spatial_chunks=6, + ) + + fast_path.assert_called_once() + self.assertEqual(fast_path.call_args.args[0], copc_original) + self.assertEqual(fast_path.call_args.kwargs["num_spatial_chunks"], 6) + + def test_legacy_raw_original_mode_ignores_matching_copc_twin(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + raw_original = original_dir / "source.las" + copc_original = original_dir / "source.copc.laz" + points = _write_original(raw_original) + copc_original.write_text("placeholder") + + with mock.patch( + "output_remap._process_single_original_input_file", + return_value=("source.las", 3, 3, 2, True, "Success"), + ) as raw_path: + with mock.patch("output_remap._process_single_original_copc_file") as copc_path: + remap_to_original_input_files( + merged_points=points, + merged_extra_dims={ + "PredInstance": np.array([1, 0, 2], dtype=np.uint16), + }, + merged_extra_dim_params=None, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + prefer_copc_sources=False, + ) + + raw_path.assert_called_once() + copc_path.assert_not_called() + process_args = raw_path.call_args.args[0] + self.assertEqual(process_args[0], raw_original) + self.assertEqual(process_args[1], output_dir / "source.las") + + def test_merged_copc_remap_streams_original_chunks_and_adds_predictions(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + points = _write_original(original_dir / "source.las") + merged_copc = root / "merged.copc.laz" + _write_merged_with_predictions(merged_copc, points) + pred_instance = np.array([1, 0, 2], dtype=np.uint16) + with laspy.open(str(merged_copc), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + fake_reader = _FakeCopcReader(reader.header, points, pred_instance) + + with mock.patch("laspy.CopcReader.open", return_value=fake_reader): + remap_merged_file_to_original_input_files( + merged_file=merged_copc, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + chunk_size=2, + prefer_copc_sources=False, + ) + + out = laspy.read(output_dir / "source.las") + dims = set(out.point_format.dimension_names) | {dim.name for dim in out.point_format.extra_dimensions} + self.assertIn("OriginalExtra", dims) + self.assertIn("PredInstance_SAT", dims) + np.testing.assert_array_equal(out.OriginalExtra, np.array([1, 2, 3], dtype=np.uint16)) + np.testing.assert_array_equal(out.PredInstance_SAT, pred_instance) + self.assertEqual(fake_reader.query_count, 2) + + def test_merged_copc_remap_fails_when_nearest_points_exceed_tolerance(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + original_dir = root / "originals" + output_dir = root / "out" + original_dir.mkdir() + points = _write_original(original_dir / "source.las") + merged_copc = root / "merged.copc.laz" + _write_merged_with_predictions(merged_copc, points) + shifted_points = points + np.array([0.01, 0.0, 0.0]) + pred_instance = np.array([1, 0, 2], dtype=np.uint16) + with laspy.open(str(merged_copc), laz_backend=laspy.LazBackend.LazrsParallel) as reader: + fake_reader = _FakeCopcReader(reader.header, shifted_points, pred_instance) + + with mock.patch("laspy.CopcReader.open", return_value=fake_reader): + with self.assertRaisesRegex(RuntimeError, "matched 0/3"): + remap_merged_file_to_original_input_files( + merged_file=merged_copc, + original_input_dir=original_dir, + output_dir=output_dir, + tolerance=0.001, + num_threads=1, + threedtrees_dims=["PredInstance"], + threedtrees_suffix="SAT", + chunk_size=2, + prefer_copc_sources=False, + ) + + self.assertFalse((output_dir / "source.las").exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_parameters_subsampling_method.py b/tests/test_parameters_subsampling_method.py new file mode 100644 index 0000000..33392fb --- /dev/null +++ b/tests/test_parameters_subsampling_method.py @@ -0,0 +1,129 @@ +import sys +import unittest +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +try: + from parameters import Parameters, get_tile_params # noqa: E402 + import run # noqa: E402 +except ModuleNotFoundError as exc: # pragma: no cover - environment-dependent + if exc.name != "pydantic_settings": + raise + Parameters = None + get_tile_params = None + run = None + + +class ParameterSubsamplingMethodTests(unittest.TestCase): + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_default_subsampling_method_is_center_of_mass(self): + params = Parameters(_cli_parse_args=False) + + self.assertEqual(params.subsampling_method, "center-of-mass") + self.assertEqual(get_tile_params(params)["subsampling_method"], "center-of-mass") + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_subsampling_method_alias_normalizes_to_nearest_to_centroid(self): + params = Parameters(subsampling_method="centroid", _cli_parse_args=False) + + self.assertEqual(params.subsampling_method, "nearest-to-centroid") + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_invalid_subsampling_method_is_rejected(self): + with self.assertRaises(ValueError): + Parameters(subsampling_method="voxel-center", _cli_parse_args=False) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_default_merged_output_format_is_copc_laz(self): + params = Parameters(_cli_parse_args=False) + + self.assertEqual(params.merged_output_formats, "copc.laz") + self.assertIsNone(params.staged_copc_dir) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_staged_copc_dir_alias_is_available(self): + params = Parameters(staged_copc_dir="/tmp/staged-copc", _cli_parse_args=False) + + self.assertEqual(params.staged_copc_dir, Path("/tmp/staged-copc")) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_raw_original_lane_aliases_are_available(self): + params = Parameters( + original_copc_input_dir="/tmp/copc", + original_laz_input_dir="/tmp/raw", + original_laz_output_dir="/tmp/raw-out", + _cli_parse_args=False, + ) + + self.assertEqual(params.original_copc_input_dir, Path("/tmp/copc")) + self.assertEqual(params.original_raw_input_dir, Path("/tmp/raw")) + self.assertEqual(params.original_raw_output_dir, Path("/tmp/raw-out")) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_filter_output_extension_alias_is_available(self): + params = Parameters(filter_output_extension=".laz", _cli_parse_args=False) + + self.assertEqual(params.filter_output_extension, ".laz") + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_merged_output_format_aliases_are_normalized_and_deduped(self): + params = Parameters(merged_output_formats="copc,laz,.ply,copc.laz", _cli_parse_args=False) + + self.assertEqual(params.merged_output_formats, "copc.laz,laz,ply") + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_merged_output_format_list_like_values_are_normalized(self): + params = Parameters(merged_output_formats=["copc.laz", "laz"], _cli_parse_args=False) + self.assertEqual(params.merged_output_formats, "copc.laz,laz") + + params = Parameters(merged_output_formats="['copc.laz', 'ply']", _cli_parse_args=False) + self.assertEqual(params.merged_output_formats, "copc.laz,ply") + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_invalid_merged_output_format_is_rejected(self): + with self.assertRaises(ValueError): + Parameters(merged_output_formats="copc.laz,txt", _cli_parse_args=False) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_zero_tile_buffer_is_allowed(self): + params = Parameters(tile_buffer=0, _cli_parse_args=False) + + self.assertEqual(params.tile_buffer, 0) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_negative_tile_buffer_is_rejected(self): + with self.assertRaises(ValueError): + Parameters(tile_buffer=-1, _cli_parse_args=False) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_num_spatial_chunks_must_be_positive(self): + with self.assertRaises(ValueError): + Parameters(num_spatial_chunks=0, _cli_parse_args=False) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_cli_unknown_long_flags_fail_fast(self): + self.assertEqual(run._unknown_cli_flags(["--task", "tile", "--tilng-threshold", "1"]), ["tilng-threshold"]) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_cli_known_alias_and_preprocessor_flags_are_accepted(self): + self.assertEqual( + run._unknown_cli_flags( + [ + "--task", + "remap", + "--subsampled-segmented-folder", + "segmented", + "--no-transfer-original-dims-to-merged", + "--show-params", + "--output-copc-res1", + "True", + ] + ), + [], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_run_filter_task.py b/tests/test_run_filter_task.py new file mode 100644 index 0000000..7f0a704 --- /dev/null +++ b/tests/test_run_filter_task.py @@ -0,0 +1,93 @@ +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +try: + from parameters import Parameters # noqa: E402 + import run # noqa: E402 +except ModuleNotFoundError as exc: # pragma: no cover - environment-dependent + if exc.name not in {"pydantic_settings", "pydantic"}: + raise + Parameters = None + run = None + + +class RunFilterTaskTests(unittest.TestCase): + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_filter_task_forwards_directory_parameters(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + input_dir = root / "input" + output_dir = root / "output" + input_dir.mkdir() + params = Parameters( + task="filter", + input_dir=input_dir, + output_dir=output_dir, + buffer=12.5, + filter_suffix="_kept", + filter_output_extension=".laz", + instance_dimension="FomaInstance", + _cli_parse_args=False, + ) + + with mock.patch( + "filter_buffer_instances.filter_buffer_instances_dir", + return_value={"input_files": 2, "output_files": [output_dir / "a.laz"]}, + ) as filter_dir: + run.run_filter_task(params) + + filter_dir.assert_called_once_with( + input_dir=input_dir, + output_dir=output_dir, + buffer=12.5, + suffix="_kept", + instance_dimension="FomaInstance", + output_extension=".laz", + ) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_filter_task_requires_input_and_output_dirs(self): + params = Parameters(task="filter", input_dir=None, output_dir=None, _cli_parse_args=False) + + with self.assertRaises(SystemExit): + run.run_filter_task(params) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_filter_task_rejects_in_place_overwrite_without_suffix(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) + params = Parameters( + task="filter", + input_dir=input_dir, + output_dir=input_dir, + filter_suffix="", + _cli_parse_args=False, + ) + + with self.assertRaises(SystemExit): + run.run_filter_task(params) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_filter_task_reports_invalid_output_extension(self): + with tempfile.TemporaryDirectory() as tmpdir: + input_dir = Path(tmpdir) + params = Parameters( + task="filter", + input_dir=input_dir, + output_dir=input_dir / "out", + filter_output_extension=".copc.laz", + _cli_parse_args=False, + ) + + with self.assertRaises(SystemExit): + run.run_filter_task(params) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_run_merge_direct_laz.py b/tests/test_run_merge_direct_laz.py new file mode 100644 index 0000000..5445f66 --- /dev/null +++ b/tests/test_run_merge_direct_laz.py @@ -0,0 +1,90 @@ +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +try: + from parameters import Parameters # noqa: E402 + import run # noqa: E402 +except ModuleNotFoundError as exc: # pragma: no cover - environment-dependent + if exc.name not in {"pydantic_settings", "pydantic"}: + raise + Parameters = None + run = None + + +class RunMergeDirectLazTests(unittest.TestCase): + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_merge_stage7_uses_laz_lane_when_copc_lane_is_present(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + segmented = root / "segmented_remapped" + copc_dir = root / "copc" + laz_dir = root / "raw_laz" + out_tiles = root / "output_tiles" + tile_bounds = root / "tile_bounds_tindex.json" + for directory in (segmented, copc_dir, laz_dir): + directory.mkdir() + tile_bounds.write_text("{}", encoding="utf-8") + + params = Parameters( + task="merge", + segmented_remapped_folder=segmented, + tile_bounds_json=tile_bounds, + output_tiles_folder=out_tiles, + output_merged_laz=root / "merged.laz", + original_copc_input_dir=copc_dir, + original_laz_input_dir=laz_dir, + transfer_original_dims_to_merged=False, + workers=1, + _cli_parse_args=False, + ) + + with mock.patch.object(run, "_validate_raw_original_lane") as validate_raw: + with mock.patch.object(run, "_validate_copc_original_lane") as validate_copc: + with mock.patch.object(run, "_validate_copc_laz_source_pairs") as validate_pairs: + with mock.patch("main_merge.run_merge", return_value=root / "merged.laz") as run_merge: + run.run_merge_task(params) + + validate_copc.assert_called_once_with(copc_dir) + validate_pairs.assert_called_once_with(copc_dir, laz_dir) + validate_raw.assert_called_once_with(laz_dir, out_tiles.parent / "original_with_predictions") + run_merge.assert_called_once() + _, kwargs = run_merge.call_args + self.assertEqual(kwargs["original_input_dir"], laz_dir) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_merge_copc_only_original_lane_is_rejected(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + segmented = root / "segmented_remapped" + copc_dir = root / "copc" + tile_bounds = root / "tile_bounds_tindex.json" + for directory in (segmented, copc_dir): + directory.mkdir() + tile_bounds.write_text("{}", encoding="utf-8") + + params = Parameters( + task="merge", + segmented_remapped_folder=segmented, + tile_bounds_json=tile_bounds, + output_tiles_folder=root / "output_tiles", + original_copc_input_dir=copc_dir, + transfer_original_dims_to_merged=False, + workers=1, + _cli_parse_args=False, + ) + + with mock.patch("main_merge.run_merge") as run_merge: + with self.assertRaises(SystemExit): + run.run_merge_task(params) + + run_merge.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_run_remap_direct_laz.py b/tests/test_run_remap_direct_laz.py new file mode 100644 index 0000000..0d351a0 --- /dev/null +++ b/tests/test_run_remap_direct_laz.py @@ -0,0 +1,223 @@ +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +try: + from parameters import Parameters # noqa: E402 + import run # noqa: E402 +except ModuleNotFoundError as exc: # pragma: no cover - environment-dependent + if exc.name not in {"pydantic_settings", "pydantic"}: + raise + Parameters = None + run = None + + +class RunRemapDirectLazTests(unittest.TestCase): + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_copc_original_lane_accepts_uppercase_copc_extension(self): + with tempfile.TemporaryDirectory() as tmpdir: + copc_dir = Path(tmpdir) + (copc_dir / "SOURCE.COPC.LAZ").write_text("placeholder") + + run._validate_copc_original_lane(copc_dir) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_segmented_collections_enrich_laz_directly_when_laz_lane_is_set(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + copc_dir = root / "copc" + laz_dir = root / "raw_laz" + pred_dir = root / "pred" + processing_out = root / "processing_unused" + laz_out = root / "raw_enriched" + for directory in (copc_dir, laz_dir, pred_dir): + directory.mkdir() + (copc_dir / "source.copc.laz").write_text("placeholder") + (laz_dir / "source.laz").write_text("placeholder") + (pred_dir / "pred.laz").write_text("placeholder") + + params = Parameters( + task="remap", + segmented_folders=str(pred_dir), + original_copc_input_dir=copc_dir, + output_dir=processing_out, + original_laz_input_dir=laz_dir, + original_laz_output_dir=laz_out, + remap_dims="PredInstance_SAT,PredSemantic_SAT", + transfer_original_dims_to_merged=False, + workers=1, + _cli_parse_args=False, + ) + + with mock.patch.object(run, "_validate_copc_original_lane") as validate_copc: + with mock.patch.object(run, "_validate_copc_laz_source_pairs") as validate_pairs: + with mock.patch( + "prediction_collection_remap.remap_prediction_collections_to_original_files" + ) as remap: + run.run_remap_task(params) + + validate_copc.assert_called_once_with(copc_dir) + validate_pairs.assert_called_once_with(copc_dir, laz_dir) + remap.assert_called_once() + args, kwargs = remap.call_args + self.assertEqual(args[0], [pred_dir]) + self.assertEqual(args[1], laz_dir) + self.assertEqual(args[2], laz_out) + self.assertEqual(kwargs["target_dims"], {"PredInstance_SAT", "PredSemantic_SAT"}) + self.assertFalse(kwargs["prefer_copc_sources"]) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_prod_merged_uses_enriched_laz_dir_when_laz_lane_is_set(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + copc_dir = root / "copc" + laz_dir = root / "raw_laz" + pred_dir = root / "pred" + processing_out = root / "processing_unused" + laz_out = root / "raw_enriched" + for directory in (copc_dir, laz_dir, pred_dir): + directory.mkdir() + (copc_dir / "source.copc.laz").write_text("placeholder") + (laz_dir / "source.laz").write_text("placeholder") + (pred_dir / "pred.laz").write_text("placeholder") + + params = Parameters( + task="remap", + segmented_folders=str(pred_dir), + original_copc_input_dir=copc_dir, + output_dir=processing_out, + original_laz_input_dir=laz_dir, + original_laz_output_dir=laz_out, + remap_dims="PredInstance_SAT", + transfer_original_dims_to_merged=True, + workers=1, + _cli_parse_args=False, + ) + + def remap_side_effect(*args, **kwargs): + args[2].mkdir(parents=True, exist_ok=True) + + with mock.patch.object(run, "_validate_copc_original_lane"): + with mock.patch.object(run, "_validate_copc_laz_source_pairs"): + with mock.patch( + "prediction_collection_remap.remap_prediction_collections_to_original_files", + side_effect=remap_side_effect, + ): + with mock.patch( + "main_create_merged_file.create_prod_merged_files", + return_value=[root / "prod_merged_1cm.copc.laz"], + ) as create_prod: + run.run_remap_task(params) + + create_prod.assert_called_once() + _, kwargs = create_prod.call_args + self.assertEqual(kwargs["original_with_predictions_dir"], laz_out) + self.assertEqual(kwargs["output_dir"], laz_out.parent) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_legacy_original_input_dir_is_treated_as_laz_source(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + laz_dir = root / "legacy_originals" + pred_dir = root / "pred" + laz_out = root / "raw_enriched" + for directory in (laz_dir, pred_dir): + directory.mkdir() + (laz_dir / "source.laz").write_text("placeholder") + (pred_dir / "pred.laz").write_text("placeholder") + + params = Parameters( + task="remap", + segmented_folders=str(pred_dir), + original_input_dir=laz_dir, + output_dir=laz_out, + remap_dims="PredInstance_SAT", + transfer_original_dims_to_merged=False, + workers=1, + _cli_parse_args=False, + ) + + with mock.patch( + "prediction_collection_remap.remap_prediction_collections_to_original_files" + ) as remap: + run.run_remap_task(params) + + remap.assert_called_once() + args, kwargs = remap.call_args + self.assertEqual(args[0], [pred_dir]) + self.assertEqual(args[1], laz_dir) + self.assertEqual(args[2], laz_out) + self.assertFalse(kwargs["prefer_copc_sources"]) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_copc_only_remap_is_rejected(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + copc_dir = root / "copc" + pred_dir = root / "pred" + for directory in (copc_dir, pred_dir): + directory.mkdir() + (copc_dir / "source.copc.laz").write_text("placeholder") + (pred_dir / "pred.laz").write_text("placeholder") + + params = Parameters( + task="remap", + segmented_folders=str(pred_dir), + original_copc_input_dir=copc_dir, + output_dir=root / "unused", + remap_dims="PredInstance_SAT", + transfer_original_dims_to_merged=False, + workers=1, + _cli_parse_args=False, + ) + + with self.assertRaises(SystemExit): + run.run_remap_task(params) + + @unittest.skipIf(Parameters is None, "pydantic_settings is not installed") + def test_merged_copc_remap_uses_streaming_entrypoint(self): + with tempfile.TemporaryDirectory() as tmpdir: + root = Path(tmpdir) + laz_dir = root / "raw_laz" + laz_out = root / "raw_enriched" + laz_dir.mkdir() + (laz_dir / "source.laz").write_text("placeholder") + merged_copc = root / "merged.copc.laz" + merged_copc.write_text("placeholder") + + params = Parameters( + task="remap", + merged_laz=merged_copc, + original_laz_input_dir=laz_dir, + original_laz_output_dir=laz_out, + threedtrees_dims="PredInstance", + threedtrees_suffix="SAT", + transfer_original_dims_to_merged=False, + workers=2, + chunk_size=123, + num_spatial_chunks=4, + _cli_parse_args=False, + ) + + with mock.patch.object(run, "_validate_raw_original_lane"): + with mock.patch("output_remap.remap_merged_file_to_original_input_files") as remap: + run.run_remap_task(params) + + remap.assert_called_once() + args, kwargs = remap.call_args + self.assertEqual(args[0], merged_copc) + self.assertEqual(args[1], laz_dir) + self.assertEqual(args[2], laz_out) + self.assertEqual(kwargs["chunk_size"], 123) + self.assertEqual(kwargs["num_threads"], 2) + self.assertEqual(kwargs["num_spatial_chunks"], 4) + self.assertFalse(kwargs["prefer_copc_sources"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_tile_bounds_graph.py b/tests/test_tile_bounds_graph.py new file mode 100644 index 0000000..a4fecd3 --- /dev/null +++ b/tests/test_tile_bounds_graph.py @@ -0,0 +1,63 @@ +import json +import sys +import tempfile +import unittest +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from tile_bounds_graph import ( # noqa: E402 + build_neighbor_graph_from_bounds_json, + match_tiles_to_json_bounds, +) + + +class TileBoundsGraphTests(unittest.TestCase): + def test_grid_neighbors_use_col_row_when_available(self): + with tempfile.TemporaryDirectory() as tmpdir: + path = Path(tmpdir) / "tile_bounds_tindex.json" + path.write_text(json.dumps({ + "tiles": [ + {"col": 0, "row": 0, "bounds": [[0, 10], [0, 10]]}, + {"col": 1, "row": 0, "bounds": [[10, 20], [0, 10]]}, + {"col": 0, "row": 1, "bounds": [[0, 10], [10, 20]]}, + ] + })) + + bounds, centers, neighbors = build_neighbor_graph_from_bounds_json(path) + + self.assertEqual(bounds[0], (0.0, 10.0, 0.0, 10.0)) + self.assertEqual(centers[0], (5.0, 5.0)) + self.assertEqual(neighbors[0]["east"], 1) + self.assertEqual(neighbors[0]["north"], 2) + self.assertIsNone(neighbors[0]["west"]) + self.assertIsNone(neighbors[0]["south"]) + + def test_match_tiles_to_json_bounds_supports_tolerance(self): + json_bounds = [(0.0, 10.0, 0.0, 10.0), (10.0, 20.0, 0.0, 10.0)] + centers = [(5.0, 5.0), (15.0, 5.0)] + tile_boundaries = { + "c00_r00": (0.05, 10.05, 0.0, 10.0), + "c01_r00": (10.04, 20.04, 0.0, 10.0), + } + + tile_to_json, json_to_tile = match_tiles_to_json_bounds(tile_boundaries, json_bounds, centers) + + self.assertEqual(tile_to_json, {"c00_r00": 0, "c01_r00": 1}) + self.assertEqual(json_to_tile, {0: "c00_r00", 1: "c01_r00"}) + + def test_match_tiles_to_json_bounds_fails_for_unmatched_tile(self): + with self.assertRaisesRegex(ValueError, "Unmatched tiles: far_away"): + match_tiles_to_json_bounds( + { + "c00_r00": (0.0, 10.0, 0.0, 10.0), + "far_away": (100.0, 110.0, 100.0, 110.0), + }, + [(0.0, 10.0, 0.0, 10.0), (10.0, 20.0, 0.0, 10.0)], + [(5.0, 5.0), (15.0, 5.0)], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_tile_copc_dimension_policy.py b/tests/test_tile_copc_dimension_policy.py new file mode 100644 index 0000000..d8dcc99 --- /dev/null +++ b/tests/test_tile_copc_dimension_policy.py @@ -0,0 +1,282 @@ +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +import tile_copc # noqa: E402 + + +class TileCopcDimensionPolicyTests(unittest.TestCase): + def setUp(self): + self._previous_empty_dims_supported = tile_copc._UNTWINE_EMPTY_DIMS_SUPPORTED + tile_copc._UNTWINE_EMPTY_DIMS_SUPPORTED = None + + def tearDown(self): + tile_copc._UNTWINE_EMPTY_DIMS_SUPPORTED = self._previous_empty_dims_supported + + def test_pdal_copc_conversion_strips_extra_dims_by_default(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_copc = tmp_path / "output.copc.laz" + input_laz.write_text("input") + seen_pipeline = {} + + def run_pipeline(command, **_): + with open(command[-1]) as handle: + seen_pipeline.update(json.load(handle)) + output_copc.write_text("copc") + return subprocess.CompletedProcess(command, 0, "", "") + + with mock.patch("tile_copc.subprocess.run", side_effect=run_pipeline): + self.assertTrue(tile_copc.convert_laz_to_copc_pdal(input_laz, output_copc)) + + writer = seen_pipeline["pipeline"][-1] + self.assertEqual(writer["type"], "writers.copc") + self.assertNotIn("extra_dims", writer) + + def test_pdal_copc_conversion_can_preserve_extra_dims_for_merged_products(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_copc = tmp_path / "output.copc.laz" + input_laz.write_text("input") + seen_pipeline = {} + + def run_pipeline(command, **_): + with open(command[-1]) as handle: + seen_pipeline.update(json.load(handle)) + output_copc.write_text("copc") + return subprocess.CompletedProcess(command, 0, "", "") + + with mock.patch("tile_copc.subprocess.run", side_effect=run_pipeline): + self.assertTrue( + tile_copc.convert_laz_to_copc_pdal( + input_laz, + output_copc, + preserve_extra_dims=True, + ) + ) + + writer = seen_pipeline["pipeline"][-1] + self.assertEqual(writer["extra_dims"], "all") + + def test_untwine_finalizer_strips_extra_dims_by_default(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + part = tmp_path / "part_0.las" + final = tmp_path / "tile.copc.laz" + part.write_text("part") + + def run_untwine(*_, **__): + final.write_text("copc") + return subprocess.CompletedProcess(["untwine"], 0, "", "") + + with mock.patch("tile_copc.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch("tile_copc._output_has_no_extra_dimensions", return_value=True): + with mock.patch( + "tile_copc.finalize_tile_to_copc_pdal", + return_value=(True, "OK"), + ) as pdal: + with mock.patch("tile_copc.subprocess.run", side_effect=run_untwine) as run: + success, message = tile_copc.finalize_tile_to_copc_untwine( + [part], + final, + tmp_path, + "tile", + ) + + self.assertTrue(success) + self.assertEqual(message, "untwine-stripped") + command = run.call_args.args[0] + self.assertIn("--dims", command) + self.assertEqual( + command[command.index("--dims") + 1], + tile_copc.UNTWINE_STRIP_EXTRA_DIMS_ARG, + ) + pdal.assert_not_called() + + def test_untwine_finalizer_strips_to_temp_laz_when_direct_dims_fails(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + part = tmp_path / "part_0.las" + final = tmp_path / "tile.copc.laz" + part.write_text("part") + + with mock.patch("tile_copc.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch( + "tile_copc.finalize_tile_to_copc_pdal", + return_value=(True, "OK"), + ) as pdal: + with mock.patch("tile_copc._strip_las_to_standard_dims", return_value=(True, "stripped")) as strip: + with mock.patch( + "tile_copc._run_untwine", + side_effect=[(False, "untwine failed"), (True, "untwine")], + ) as untwine: + with mock.patch("tile_copc._output_has_no_extra_dimensions", return_value=True): + success, message = tile_copc.finalize_tile_to_copc_untwine( + [part], + final, + tmp_path, + "tile", + ) + + self.assertTrue(success) + self.assertEqual(message, "pdal-strip+untwine") + strip.assert_called_once() + self.assertEqual(untwine.call_count, 2) + self.assertTrue(untwine.call_args_list[0].kwargs["strip_extra_dims"]) + self.assertFalse(untwine.call_args_list[1].kwargs["strip_extra_dims"]) + pdal.assert_not_called() + + def test_standard_dimension_staging_uses_minimal_las_point_format(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_laz = tmp_path / "stripped.laz" + input_laz.write_text("input") + seen_pipeline = {} + + def run_pipeline(command, **_): + with open(command[-1]) as handle: + seen_pipeline.update(json.load(handle)) + output_laz.write_text("laz") + return subprocess.CompletedProcess(command, 0, "", "") + + with mock.patch("tile_copc.subprocess.run", side_effect=run_pipeline): + success, message = tile_copc._strip_las_to_standard_dims([input_laz], output_laz) + + self.assertTrue(success, message) + writer = seen_pipeline["pipeline"][-1] + self.assertEqual(writer["type"], "writers.las") + self.assertEqual(writer["minor_version"], 2) + self.assertEqual(writer["dataformat_id"], 0) + self.assertNotIn("extra_dims", writer) + + def test_run_untwine_uses_empty_dimension_keep_list_for_stripping(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_copc = tmp_path / "output.copc.laz" + input_laz.write_text("input") + + def run_untwine(*_, **__): + output_copc.write_text("copc") + return subprocess.CompletedProcess(["untwine"], 0, "", "") + + with mock.patch("tile_copc.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch("tile_copc.subprocess.run", side_effect=run_untwine) as run: + success, message = tile_copc._run_untwine( + [input_laz], + output_copc, + "EPSG:32632", + strip_extra_dims=True, + ) + + self.assertTrue(success, message) + command = run.call_args.args[0] + self.assertIn("--dims", command) + self.assertEqual( + command[command.index("--dims") + 1], + tile_copc.UNTWINE_STRIP_EXTRA_DIMS_ARG, + ) + self.assertTrue(tile_copc._UNTWINE_EMPTY_DIMS_SUPPORTED) + + def test_run_untwine_caches_unsupported_empty_dimension_keep_list(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_copc = tmp_path / "output.copc.laz" + input_laz.write_text("input") + + def reject_empty_dims(command, **_): + self.assertIn("--dims", command) + return subprocess.CompletedProcess( + command, + 255, + "", + "Untwine Error: Missing value for argument 'dims'.", + ) + + with mock.patch("tile_copc.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch("tile_copc.subprocess.run", side_effect=reject_empty_dims) as run: + success, message = tile_copc._run_untwine( + [input_laz], + output_copc, + "EPSG:32632", + strip_extra_dims=True, + ) + self.assertFalse(success) + self.assertIn("untwine failed", message) + self.assertFalse(tile_copc._UNTWINE_EMPTY_DIMS_SUPPORTED) + + success, message = tile_copc._run_untwine( + [input_laz], + output_copc, + "EPSG:32632", + strip_extra_dims=True, + ) + + self.assertFalse(success) + self.assertEqual(message, "untwine empty --dims unsupported") + self.assertEqual(run.call_count, 1) + + def test_convert_laz_to_copc_uses_direct_untwine_dims_when_supported(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_copc = tmp_path / "output.copc.laz" + input_laz.write_text("input") + + def run_untwine(*_, **__): + output_copc.write_text("copc") + return subprocess.CompletedProcess(["untwine"], 0, "", "") + + with mock.patch("tile_copc.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch("tile_copc._output_has_no_extra_dimensions", return_value=True): + with mock.patch("tile_copc.srs_assignment_from_file", return_value="EPSG:32632"): + with mock.patch("tile_copc.append_source_geotiff_projection_evlrs"): + with mock.patch("tile_copc.copc_preserves_source_crs", return_value=(True, "ok")): + with mock.patch("tile_copc.subprocess.run", side_effect=run_untwine) as run: + self.assertTrue(tile_copc.convert_laz_to_copc(input_laz, output_copc)) + + command = run.call_args.args[0] + self.assertIn("--dims", command) + self.assertEqual( + command[command.index("--dims") + 1], + tile_copc.UNTWINE_STRIP_EXTRA_DIMS_ARG, + ) + + def test_convert_laz_to_copc_strips_to_temp_laz_when_direct_dims_fails(self): + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + input_laz = tmp_path / "input.laz" + output_copc = tmp_path / "output.copc.laz" + input_laz.write_text("input") + + with mock.patch("tile_copc.shutil.which", return_value="/usr/bin/untwine"): + with mock.patch("tile_copc._strip_las_to_standard_dims", return_value=(True, "stripped")) as strip: + with mock.patch( + "tile_copc._run_untwine", + side_effect=[(False, "untwine failed"), (True, "untwine")], + ) as untwine: + with mock.patch("tile_copc.convert_laz_to_copc_pdal") as pdal: + with mock.patch("tile_copc.append_source_geotiff_projection_evlrs", return_value=(True, "ok")): + with mock.patch("tile_copc.copc_preserves_source_crs", return_value=(True, "ok")): + with mock.patch("tile_copc._output_has_no_extra_dimensions", return_value=True): + self.assertTrue(tile_copc.convert_laz_to_copc(input_laz, output_copc)) + + strip.assert_called_once() + self.assertEqual(untwine.call_count, 2) + self.assertTrue(untwine.call_args_list[0].kwargs["strip_extra_dims"]) + self.assertFalse(untwine.call_args_list[1].kwargs["strip_extra_dims"]) + pdal.assert_not_called() + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_tile_spatial.py b/tests/test_tile_spatial.py new file mode 100644 index 0000000..594a92f --- /dev/null +++ b/tests/test_tile_spatial.py @@ -0,0 +1,99 @@ +import sys +import unittest +from pathlib import Path + +import numpy as np + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from tile_spatial import ( # noqa: E402 + compute_centroids_vectorized, + filter_by_centroid_in_buffer, + find_overlap_region, + find_spatial_neighbors, + get_border_region_mask, +) + + +class TileSpatialTests(unittest.TestCase): + def test_find_overlap_region(self): + self.assertEqual( + find_overlap_region((0, 10, 0, 10), (5, 15, 2, 8)), + (5, 10, 2, 8), + ) + self.assertIsNone(find_overlap_region((0, 10, 0, 10), (10, 20, 0, 10))) + + def test_compute_centroids_vectorized_ignores_background(self): + points = np.array([ + [0.0, 0.0, 0.0], + [2.0, 0.0, 0.0], + [10.0, 0.0, 0.0], + [99.0, 99.0, 99.0], + ]) + instances = np.array([1, 1, 2, 0]) + + centroids = compute_centroids_vectorized(points, instances) + + np.testing.assert_allclose(centroids[1], np.array([1.0, 0.0, 0.0])) + np.testing.assert_allclose(centroids[2], np.array([10.0, 0.0, 0.0])) + self.assertNotIn(0, centroids) + + def test_find_spatial_neighbors_prefers_cardinal_overlap(self): + all_tiles = { + "center": (0.0, 10.0, 0.0, 10.0), + "east": (8.0, 18.0, 0.0, 10.0), + "north": (0.0, 10.0, 8.0, 18.0), + "diagonal": (8.0, 18.0, 8.0, 18.0), + } + + neighbors = find_spatial_neighbors(all_tiles["center"], "center", all_tiles, tolerance=1.5) + + self.assertEqual(neighbors["east"], "east") + self.assertEqual(neighbors["north"], "north") + self.assertIsNone(neighbors["west"]) + self.assertIsNone(neighbors["south"]) + + def test_filter_by_centroid_in_buffer_uses_precomputed_neighbors(self): + points = np.array([ + [1.0, 5.0, 0.0], + [2.0, 5.0, 0.0], + [8.0, 5.0, 0.0], + [9.0, 5.0, 0.0], + ]) + instances = np.array([1, 1, 2, 2]) + + removed, directions = filter_by_centroid_in_buffer( + points, + instances, + (0.0, 10.0, 0.0, 10.0), + "center", + {}, + buffer=3.0, + precomputed_neighbors={"west": "west", "east": None, "north": None, "south": None}, + ) + + self.assertEqual(removed, {1}) + self.assertEqual(directions, {1: "west"}) + + def test_get_border_region_mask_preserves_edge_inclusivity(self): + points = np.array([ + [1.0, 5.0, 0.0], + [2.0, 5.0, 0.0], + [8.0, 5.0, 0.0], + [9.0, 5.0, 0.0], + ]) + + mask = get_border_region_mask( + points, + (0.0, 10.0, 0.0, 10.0), + inner_dist=1.0, + outer_dist=2.0, + neighbors={"west": "w", "east": "e", "north": None, "south": None}, + ) + + np.testing.assert_array_equal(mask, np.array([True, False, False, True])) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_union_find.py b/tests/test_union_find.py new file mode 100644 index 0000000..c111add --- /dev/null +++ b/tests/test_union_find.py @@ -0,0 +1,48 @@ +import sys +import unittest +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from union_find import UnionFind # noqa: E402 + + +class UnionFindTests(unittest.TestCase): + def test_union_keeps_larger_component_as_root(self): + uf = UnionFind() + uf.make_set(1, size=10) + uf.make_set(2, size=2) + + root = uf.union(1, 2) + + self.assertEqual(root, 1) + self.assertEqual(uf.find(2), 1) + self.assertEqual(uf.size[1], 12) + + def test_union_can_promote_larger_second_root(self): + uf = UnionFind() + uf.make_set(1, size=1) + uf.make_set(2, size=5) + + root = uf.union(1, 2) + + self.assertEqual(root, 2) + self.assertEqual(uf.find(1), 2) + self.assertEqual(uf.size[2], 6) + + def test_get_components(self): + uf = UnionFind() + uf.make_set(1) + uf.make_set(2) + uf.make_set(3) + uf.union(1, 2) + + components = {root: sorted(members) for root, members in uf.get_components().items()} + + self.assertIn([1, 2], components.values()) + self.assertIn([3], components.values()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_worker_budget.py b/tests/test_worker_budget.py new file mode 100644 index 0000000..6457db3 --- /dev/null +++ b/tests/test_worker_budget.py @@ -0,0 +1,23 @@ +import sys +import unittest +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from worker_budget import kdtree_query_workers # noqa: E402 + + +class WorkerBudgetTests(unittest.TestCase): + def test_kdtree_query_workers_share_total_budget_across_outer_workers(self): + self.assertEqual(kdtree_query_workers(total_workers=10, outer_workers=1), 10) + self.assertEqual(kdtree_query_workers(total_workers=10, outer_workers=2), 5) + self.assertEqual(kdtree_query_workers(total_workers=10, outer_workers=8), 1) + + def test_kdtree_query_workers_never_return_zero(self): + self.assertEqual(kdtree_query_workers(total_workers=0, outer_workers=10), 1) + self.assertEqual(kdtree_query_workers(total_workers=10, outer_workers=0), 10) + + +if __name__ == "__main__": + unittest.main() diff --git a/tool_appendix.txt b/tool_appendix.txt new file mode 100644 index 0000000..f170e36 --- /dev/null +++ b/tool_appendix.txt @@ -0,0 +1,135 @@ +\subsection{SmartTile} +\noindent +\begin{tabular}{@{}p{0.28\linewidth}p{0.68\linewidth}@{}} +\textbf{Main developers} & Kilian Gerberding \\ +\textbf{Current version} & v1.0.0 \\ +\textbf{Galaxy tool ID} & \texttt{3dtrees\_smart\_tile} \\ +\textbf{Access on Galaxy} & \url{https://...} \\ +\textbf{GitHub repository} & \url{https://github.com/3dTrees-earth/3dtrees} (tools/tool\_smart\_tile) \\ +\textbf{Scope} & +Subsampling, tiling, and merging of point clouds for the 3DTrees segmentation pipeline. \textbf{Tile} task: converts LAZ/LAS to COPC, builds a spatial index, computes overlapping tile bounds, creates tiles, and subsamples each tile at two resolutions (e.g.\ 1\,cm and 10\,cm) for segmentation and downstream use. \textbf{Merge} task: remaps predictions from coarser (e.g.\ 10\,cm segmented) to target resolution (e.g.\ 1\,cm), merges overlapping tiles with cross-tile instance matching, and optionally remaps to original input resolution. Implemented in Python (laspy, PDAL/untwine subprocesses, scipy cKDTree). \\ + +\end{tabular} + + +\noindent\textbf{Run order} \\ +\textbf{Tile task.} \texttt{run.py} loads \texttt{parameters.py} (Pydantic), then calls \texttt{run\_tile\_task}. \texttt{main\_tile.run\_tiling\_pipeline} converts input LAZ/LAS to COPC via \texttt{untwine} (optionally reducing to standard LAS dimensions only), builds a tindex with PDAL, computes tile bounds via \texttt{prepare\_tile\_jobs} and \texttt{get\_bounds\_from\_tindex} (tile length and buffer). If a single file is below \texttt{tiling\_threshold} (MB), tiling is skipped and the COPC is used as a single ``tile''. \texttt{plot\_tiles\_and\_copc} writes \texttt{overview\_copc\_tiles.png}. Tiles are created with PDAL. \texttt{main\_subsample.run\_subsample\_pipeline} subsamples each tile to resolution 1 and resolution 2 (PDAL subprocess per tile), writing to \texttt{subsampled\_res1} and \texttt{subsampled\_res2}. \\ +\textbf{Merge task.} \texttt{run.py} calls \texttt{run\_merge\_task}. If \texttt{--subsampled-segmented-folder} is provided, \texttt{main\_remap.remap\_all\_tiles} runs first: source (segmented) and target (subsampled) files are matched by spatial bounds, and point attributes (e.g.\ PredInstance, PredSemantic) are remapped via cKDTree nearest-neighbor to the target resolution. Then \texttt{main\_merge.run\_merge} invokes \texttt{merge\_tiles.merge\_tiles}: load segmented tiles, filter by buffer, assign global instance IDs, perform cross-tile instance matching (overlap and centroid criteria), merge and deduplicate, apply small-volume instance merging, retile to original tile layout, and optionally remap to original input files. Per-tile LAZ and a single merged LAZ are written. \\ + + +\begingroup +\small % or \footnotesize +\setlength{\tabcolsep}{4pt} % horizontal padding (optional) +\renewcommand{\arraystretch}{0.92} % <1.0 = tighter rows +\setlength{\LTpre}{2pt} % space before longtable +\setlength{\LTpost}{2pt} % space after longtable + +\paragraph{Interface specifications — Tile} +\noindent\textbf{Inputs}\par +\begin{longtable}{@{}>{\RaggedRight\arraybackslash}p{0.28\linewidth} + >{\RaggedRight\arraybackslash}p{0.68\linewidth}@{}} +\textbf{\texttt{*.las} / \texttt{*.laz}} & +Collection of point cloud files to tile and subsample. Galaxy symlinks them into \texttt{input\_dir}. \\ +\end{longtable} + +\noindent\textbf{Outputs} +\begin{longtable}{@{}p{0.28\linewidth}p{0.68\linewidth}@{}} +\textbf{\texttt{Subsampled\_Resolution\_1}} (collection) & +LAZ files at first subsampling resolution (e.g.\ 1\,cm), one per tile; discovered from \texttt{output\_dir/subsampled\_res1}. \\ +\textbf{\texttt{Subsampled\_Resolution\_2}} (collection) & +LAZ files at second subsampling resolution (e.g.\ 10\,cm), one per tile; discovered from \texttt{output\_dir/subsampled\_res2}. \\ +\textbf{\texttt{output\_png}} & +Tiling preview image \texttt{overview\_copc\_tiles.png} showing COPC extents and tile bounds. \\ +\end{longtable} + +\noindent\textbf{Parameters} +\begin{longtable}{@{}p{0.28\linewidth}p{0.68\linewidth}@{}} +\textbf{\texttt{--input-dir}} & +Directory containing input LAZ/LAS files (Galaxy: \texttt{input\_dir}). \\ +\textbf{\texttt{--output-dir}} & +Directory for tiles and subsampled outputs (default: \texttt{output\_dir}). \\ +\textbf{\texttt{--task}} & +\texttt{tile} for this pipeline. \\ +\textbf{\texttt{--tile-length}} & +Tile size in meters (default: \texttt{300}). \\ +\textbf{\texttt{--tile-buffer}} & +Overlap between tiles in meters (default: \texttt{20}). \\ +\textbf{\texttt{--tiling-threshold}} & +Optional. If a single file is below this size in MB, tiling is skipped. \\ +\textbf{\texttt{--resolution-1}} & +First subsampling resolution in meters (default: \texttt{0.01}). \\ +\textbf{\texttt{--resolution-2}} & +Second subsampling resolution in meters (default: \texttt{0.1}). \\ +\textbf{\texttt{--skip-dimension-reduction}} & +If set, keep extra dimensions in LAZ intermediates; intermediate COPC conversion still strips extra dimensions by default. \\ +\textbf{\texttt{--workers}} & +Number of parallel workers (Galaxy may pass \texttt{\${GALAXY\_SLOTS:-4}}). \\ +\textbf{\texttt{--num-spatial-chunks}} & +Spatial chunks for subsampling and bounded prod-merged creation (default: workers). \\ +\textbf{\texttt{--threads}} & +Threads per COPC writer (Galaxy: \texttt{\${GALAXY\_SLOTS:-4}}). \\ +\end{longtable} + +\paragraph{Interface specifications — Merge} +\noindent\textbf{Inputs} +\begin{longtable}{@{}p{0.28\linewidth}p{0.68\linewidth}@{}} +\textbf{\texttt{input\_segmented}} & +Collection of subsampled LAZ files with predictions (e.g.\ PredInstance, PredSemantic), e.g.\ 10\,cm segmented. Galaxy: \texttt{input\_segmented}. \\ +\textbf{\texttt{input\_res1}} & +Collection of subsampled LAZ files at target resolution for remapping (e.g.\ 1\,cm). Galaxy: \texttt{input\_res1}. \\ +\textbf{\texttt{input\_original\_laz}} & +Optional. Uploaded original LAZ/LAS files to enrich at original resolution. \\ +\textbf{\texttt{input\_original\_copc}} & +Optional. Matching original COPC LAZ files used for source matching/validation; enriched originals are still written from the uploaded LAZ/LAS files. \\ +\end{longtable} + +\noindent\textbf{Outputs} +\begin{longtable}{@{}p{0.28\linewidth}p{0.68\linewidth}@{}} +\textbf{\texttt{output\_merge\_tiles}} (collection) & +Per-tile LAZ with predictions; discovered from \texttt{output\_dir/output\_tiles/original\_with\_predictions}. \\ +\textbf{\texttt{output\_merged\_laz}} & +Single merged LAZ file \texttt{merged.laz}. \\ +\end{longtable} + +\noindent\textbf{Parameters} +\begin{longtable}{@{}p{0.28\linewidth}p{0.68\linewidth}@{}} +\textbf{\texttt{--subsampled-segmented-folder}} & +Folder with segmented (e.g.\ 10\,cm) LAZ files; Galaxy: \texttt{input\_segmented} copied to \texttt{input\_segmented/}. \\ +\textbf{\texttt{--subsampled-target-folder}} & +Folder with target-resolution LAZ for remap; Galaxy: \texttt{input\_res1} copied to \texttt{input\_res1/}. \\ +\textbf{\texttt{--original-laz-input-dir}} & +Optional. Directory of uploaded original LAZ/LAS files for final remap; Galaxy: \texttt{input\_original\_laz}. Legacy \texttt{--original-input-dir} is still accepted as the same raw-LAZ lane. \\ +\textbf{\texttt{--original-copc-input-dir}} & +Optional directory of matching original COPC LAZ files for validation/source matching; Galaxy: \texttt{input\_original\_copc}. \\ +\textbf{\texttt{--segmented-folders}} & +For final multi-collection remap, comma-separated finalized prediction collections. Prediction extra dimensions are copied as named by the producing model; duplicate dimension names across collections fail instead of being auto-suffixed. \\ +\textbf{\texttt{--remap-dims}} & +Optional comma-separated allowlist of prediction extra dimensions to transfer during multi-collection remap. If omitted, all prediction extra dimensions are transferred. \\ +\textbf{\texttt{--chunk-size}} & +Points per streaming chunk for tiling Phase 1 and multi-collection remap. Larger values reduce repeated prediction-window scans but increase peak memory. \\ +\textbf{\texttt{--merged-resolutions}} / \textbf{\texttt{--merged-output-formats}} & +Optional prod-merged products from Original-with-predictions files. COPC output preserves CRS and extra dimensions as far as LAS/COPC allows, but final direct \texttt{untwine} creation needs local scratch space for temporary hierarchy/output staging and is not guaranteed bit-for-bit point-count reproducible across chunking/tool versions. \\ +\textbf{\texttt{--staged-copc-dir}} & +Optional directory with already converted Original-with-predictions COPCs. Matching readable COPCs are reused instead of restaging raw LAZ/LAS inputs. \\ +\textbf{\texttt{--output-tiles-folder}} & +Output folder for per-tile results (default: \texttt{output\_dir/output\_tiles}). \\ +\textbf{\texttt{--buffer}} & +Buffer distance in meters for filtering (default: \texttt{30.0}). \\ +\textbf{\texttt{--overlap-threshold}} & +Overlap ratio threshold for instance matching, 0--1 (default: \texttt{0.3}). \\ +\textbf{\texttt{--max-centroid-distance}} & +Max centroid distance in meters to merge instances (default: \texttt{3.0}). \\ +\textbf{\texttt{--correspondence-tolerance}} & +Max distance for point correspondence in meters (default: \texttt{0.05}). \\ +\textbf{\texttt{--max-volume-for-merge}} & +Max convex hull volume in m$^3$ for small-instance merging (default: \texttt{4.0}). \\ +\textbf{\texttt{--min-cluster-size}} & +Minimum cluster size in points for reassignment (default: \texttt{300}). \\ +\textbf{\texttt{--border-zone-width}} & +Width of border zone in meters for instance matching (default: \texttt{10.0}). \\ +\textbf{\texttt{--disable-matching}} & +If set, disable cross-tile instance matching. \\ +\textbf{\texttt{--workers}} & +Number of workers (Galaxy uses \texttt{2} for merge). \\ +\end{longtable} +\endgroup