Statewide gen1 datum: per-line constants, and whether the control is dense enough to support them

[awickert]

Task: generalise the gen1 datum from one site to a statewide, per-flight-line correction.
The handoff (ground_control/HANDOFF_FROM_GROUND_CONTROL.md §4) records this as where
control actually pays off: "the weighted uncertainty falls from 22.75 mm toward 11 mm only
as per-line constants improve, which needs many marks per line rather than more marks at
one site."

A sketch first, because it does not obviously work

gen1's level is per flight line, so a site's constant is only as good as the constants of
the lines covering it. Improving those needs many marks per line. Whether that is
available is arithmetic on the control we have.

Measured from src/lidar_diff_icp/groundtruth/data/mn_dnr_2008_control_semn.csv:

  control extent (2.5-97.5 pct box): 159 x 93 km = 14,813 km2
  marks 1004 (241 open L1O)  ->  0.068 /km2 all, 0.0163 /km2 open

Marks a single line would sweep, by line length and swath width:

     swath      20 km      50 km     100 km     200 km
    400 m         0.1        0.3        0.7        1.3      OPEN marks
    800 m         0.3        0.7        1.3        2.6      OPEN marks
   1200 m         0.4        1.0        2.0        3.9      OPEN marks
   1600 m         0.5        1.3        2.6        5.2      OPEN marks

A 100 km line at a 1.2 km swath sweeps about two open marks. Even 200 km at 1.6 km
gives about five. "Many marks per line" is not available from open ground at this control
density — line length buys marks only linearly while the density is fixed and low.

And on the tracks actually solved so far (67 passes over 46 tiles, 614 km total), the
observed counts match:

   half-width  marks w/ >=1 line  open marks   median per PASS   per PSID
        600 m                103          29                 2          2
       1200 m                115          29                 3          4

The tension this exposes

Using all marks rather than open-only raises the density 4.2x (0.068 vs 0.0163 /km2),
which is the difference between ~2 and ~8 marks on a 100 km line. But open-only is a
deliberate scientific decision, not an oversight: pooling bakes canopy response into the
datum and pre-decides the canopy-versus-erosion question. So the statewide method appears
to need the one thing that was deliberately given up.

Three ways out, none free, and this issue is to choose among them:

1. Accept fewer marks per line and let the per-line constants stay weak. Then the
   22.75 -> 11 mm improvement does not arrive, and the honest statement is that gen1's
   absolute level is a ~20 mm quantity.

2. Model the cover dependence rather than excluding it, so non-open marks can be used
   with a cover term. That is exactly the NGV/vegetation work (issue gen1 may not resolve ground under dense grass (reads as false erosion) #1 and
   analysis/ngv.py), applied to gen1 instead of gen2 — and gen1's vegetation response is
   currently unmeasured.

3. Partial pooling across lines (hierarchical, shrinking each line toward the site or
   regional mean) rather than independent per-line means. This uses the between-line
   structure instead of fighting it, and needs no new marks. The ICC is already measured,
   though it depends on how lines are grouped -- from analysis/GEN1_DATUM_MODULE.md:

   Over the 56 marks within 20 km: F = 3.37, p = 0.000952 (df 26, 29), ICC 0.537.
   returns-based      k=19  F=3.761  p=0.01174  ICC=0.632  sd(line means)=94.4  SE over lines=21.7
   centreline-based   k=13  F=5.023  p=0.001137  ICC=0.643  sd(line means)=66.7  SE over lines=18.5
   

   Note that document also records the ICC as "reported for information beside F and p;
   nothing in the estimate depends on it" -- so partial pooling would be a change in
   estimator, not a re-use of something already load-bearing.

Option 3 is the cheapest to test and needs no new data.

Also needed regardless

• The tracks are truncated to the 46 fetched tiles (614 km total, ~9 km per pass). Real
  lines are far longer; extending them across every tile a line crosses is a prerequisite
  for any per-line estimate.
• psid is not a flight line (41 distinct psids over 67 passes); group by physical track.

Numbers above are from this repo's own control table and
ground_control/data/gen1_line_tracks.json; no new data was fetched.

🤖 Generated with Claude Code