F7 colour campaign: USB frame transport, A/B slot flashing, and a DRC fix

.gitattributes

@@ -50,6 +50,13 @@ installer/debian/*         text eol=lf
 # consistent.
 installer/files/*          text eol=lf
 
+# Measurement datasets are a primary scientific record, and `text=auto` above
+# would rewrite their line endings on a Windows checkout. That does not corrupt
+# the JSON, but it means the file no longer round-trips byte-for-byte, so a
+# checksum taken against a copy on one platform disagrees on another for a
+# reason that has nothing to do with the data. Pin them.
+docs/screens/*/measurements/data/* text eol=lf
+
 # pi-gen build inputs. `config` is sourced by bash, so a CR becomes part of
 # every value it sets (IMG_NAME, TARGET_HOSTNAME, ...); 00-packages is fed to
 # apt one line at a time, which then looks for a package literally named

AGENTS.md

@@ -49,6 +49,32 @@ rules directly. They are absolute and override "it's easier", "to be safe", and
    state exactly what you will erase/write and wait for it. Otherwise flash only the
    app partitions you changed; leave the bootloader and the known-good slot intact.
 
+## Colour calibration — USB upload ONLY, never over HTTP
+
+**Getting a calibration target onto a panel MUST use the direct USB/serial frame
+upload.** The `frame` console command (see
+[`docs/screens/bigme_f7/custom_firmware.md`](docs/screens/bigme_f7/custom_firmware.md))
+plus [`tools/f7_send_frame.py`](tools/f7_send_frame.py) push the exact bytes down
+the UART. That is what the firmware feature exists for.
+
+**Do NOT propose, and do not fall back to, serving the target over HTTP from a
+hokku server** — not production, not the test Pi, not a throwaway local server.
+It does not work for this purpose and the human has ruled it out. Concretely, the
+server route is wrong because:
+
+- the render pipeline re-processes the image (autocontrast, gamma, CLAHE,
+  unsharp, error diffusion), so what reaches the glass is not the target that was
+  generated — measuring it measures the pipeline, not the panel;
+- it requires temporarily rewriting a live server's preset and the screen's
+  `server_url`, both of which are disruptive and have to be restored;
+- it makes the measurement depend on WiFi, poll timing and which host the screen
+  happens to point at, so a session cannot be reproduced.
+
+The USB route has none of those properties: exact bytes, on demand, no server,
+no config churn. Use it. If the firmware on a unit is too old to have `frame`,
+the answer is to flash it (subject to the STOP rules above), not to reach for
+HTTP.
+
 ## Python environment
 - Venv: `.venv/` at repo root
 - Windows: `.venv/Scripts/python` | Linux/macOS: `.venv/bin/python`

docs/color_calibration.md

@@ -0,0 +1,225 @@
+# Measuring a panel's colour with a colorimeter
+
+The dither pipeline is built on one array per screen model —
+`palette_measured_rgb` in `python/hokku/screens/<model>/display.py`. Every
+palette LUT, the hue gating, the dynamic-range anchors and the quality metrics
+are derived from it (see [dithering.md](dithering.md) §1, §8).
+
+Its provenance is weak. `bigme_f7` borrowed its values from a third party's
+`default-palettes.json`; `seeedstudio_e1004` inherits huessen's wholesale;
+huessen's own numbers have no recorded method. This document is how to replace
+them with values measured on our own glass, and how to check whether the panel
+actually mixes inks the way the renderer assumes.
+
+---
+
+## 1. What the instrument can and cannot do
+
+E-paper is **reflective**. It has no emission of its own, so what a meter reads
+is entirely a function of the light you put on it.
+
+That splits instruments into two classes:
+
+| Instrument | Lamp? | How to use it here |
+|---|:---:|---|
+| Calibrite ColorChecker Display / Display Plus, X-Rite i1Display Pro / Studio, ColorMunki Display | no | Emissive mode under your own controlled light, normalised against a white reference. |
+| X-Rite i1Pro / i1Pro 2 / 3, ColorMunki Photo, i1Studio | yes | Reflective mode, 45°/0°, calibrate on the white tile. Absolute L\*a\*b\* directly. |
+
+A colorimeter has no lamp and no reflective mode — ArgyllCMS will refuse
+`spotread` in reflective mode on one. It is still a perfectly good tristimulus
+sensor; you just have to supply and control the illumination yourself, and
+divide out its brightness and colour cast afterwards. That is what
+`tools/color_read.py` does.
+
+**Two error sources to accept on the colorimeter path.** Its filters approximate
+the CIE observer for *display* spectra, not for broadband light reflected off
+pigment — expect a few ΔE of systematic error. And the result is only as good
+as your knowledge of the white reference. Repeatability will be far better than
+absolute accuracy, which is the right trade: the LUTs care mostly about where
+the six anchors sit *relative to each other*.
+
+---
+
+## 2. The rig
+
+```
+        lamp
+          \  45°
+           \
+            \        ┌──────────┐   meter, perpendicular,
+             \       │  patch   │ ← a few mm off the glass
+              ＿＿＿＿│＿＿＿＿＿＿│＿＿＿
+                     panel
+```
+
+- **Light**: one stable, high-CRI broadband source at roughly 45° to the panel
+  normal. A D50 viewing lamp is ideal; a good high-CRI LED works. Kill every
+  other light in the room, including daylight — one illuminant only, and it
+  must not change between the first patch and the last.
+- **Geometry**: 45°/0° is the standard for reflective media and it keeps the
+  lamp's specular reflection off the front lamination out of the aperture.
+  Rotate the meter's **ambient diffuser out of the way** — that cap is for
+  measuring room light and would integrate the whole hemisphere.
+- **Consistency beats correctness**: clamp or jig the meter so every patch is
+  read at the same distance and angle. A patch-to-patch geometry change is
+  indistinguishable from a colour difference.
+- **A white reference**, measured in the identical geometry. A ColorChecker
+  white patch or a white calibration tile with known Lab. This is what makes
+  the numbers mean something — see §5.
+
+The generated target's patches are ~29 mm square on the F7 against a meter
+aperture of roughly 10–15 mm, so there is ~7 mm of centring slack per side.
+
+---
+
+## 3. Generate and display the target
+
+```powershell
+python tools\color_target.py --model bigme_f7
+```
+
+Writes to `build/colorcal/`:
+
+| File | Purpose |
+|---|---|
+| `colorcal_<model>.png` | Upload this to the server |
+| `colorcal_<model>.json` | Patch manifest — feeds `color_read.py` |
+| `colorcal_<model>.bin` | Packed panel bytes, for `screen_sim.py --file` |
+
+The target is 15 patches on a 5×3 grid: six flat ink anchors, seven
+Bayer-dithered ramp patches at exactly k/8 black-ink coverage, and repeats of
+white and black at the end of the sequence as a drift check.
+
+### Route A — over the serial console (preferred, Bigme F7)
+
+If the unit runs firmware with the `frame` command, skip the server entirely:
+
+```powershell
+python tools\f7_send_frame.py --port COM9 --target
+```
+
+This pushes the exact bytes down the console UART (~17 s at 115200) and refreshes
+the panel. Nothing about the picture depends on the server's configured preset,
+on WiFi, or on which host the screen happens to point at — which is what you want
+while metering, and what makes a measurement session reproducible.
+
+`--cycle` walks every ink and then the target, which is also the quickest way to
+confirm all six inks actually fire on a given unit.
+
+### Route B — through the server
+
+For screens without the `frame` command, upload the PNG and let the server render
+it. To get it onto the glass **unmodified**:
+
+1. Set the screen's preset to **`calibration_raw`**. This matters. The normal
+   presets run autocontrast, gamma, CLAHE, unsharp mask and error diffusion —
+   any one of which destroys a flat patch or shifts a ramp's coverage away from
+   the k/8 it is supposed to be. `calibration_raw` neutralises the whole chain
+   and uses `noop` quantisation, so each pixel maps to the ink it was painted
+   in. `test_color_target.py` asserts bit-exactness through the real renderer,
+   so this cannot silently rot.
+2. Upload the PNG and pin it with **show next**.
+3. Wait for the screen to poll (or power-cycle it) and refresh.
+
+Sanity-check the glass before metering: the six anchor patches must be visibly
+flat, with no speckle. If they are not, the preset did not take.
+
+**Put the preset back afterwards** — photos rendered with `calibration_raw`
+band badly.
+
+---
+
+## 4. Take the readings
+
+`tools/color_read.py` prompts patch by patch; you paste a reading into it. It
+deliberately does not drive the instrument: that keeps it working with any
+meter and any vendor software, and means getting numbers is not blocked on the
+ArgyllCMS USB driver swap (which on Windows displaces the X-Rite/Calibrite
+driver, so you will be swapping back and forth if you also use i1Profiler).
+
+If you are using ArgyllCMS, `spotread -e` gives emissive readings; paste the
+whole result line, the XYZ triple is picked out of it. Bare `X Y Z` works too.
+
+```powershell
+python tools\color_read.py build\colorcal\colorcal_bigme_f7.json --white-ref-lab 96.5,-0.4,1.2
+```
+
+Order matters only in that the two repeat patches come last — they are the
+drift check. Readings are saved as you go, so a session can be re-analysed with
+`--from-file` without re-metering.
+
+---
+
+## 5. Why the white reference matters
+
+Raw readings carry the lamp's brightness and colour cast. Dividing every patch
+componentwise by a reading of the white reference cancels both at once (a von
+Kries adaptation); multiplying back up by the reference's *known* Lab is what
+preserves the difference between "the reference is perfectly white" and "the
+reference is a real, slightly off-white object".
+
+Without a reference you can pass `--relative-to-panel-white`, which normalises
+against the panel's own white ink instead. Understand what that costs:
+
+- panel white is **defined** as neutral L\*=100, so the white ink's real
+  blue-grey cast — a thing the pipeline models — is thrown away;
+- absolute lightness is meaningless, so the panel-white anchor that
+  `compress_dynamic_range()` depends on cannot be obtained.
+
+It still yields usable *relative* geometry for the coloured anchors. It is a
+fallback, not the plan.
+
+---
+
+## 6. Reading the output
+
+### Ink anchors
+
+A table of measured L\*a\*b\* per ink, the sRGB triple, and **ΔE vs the values
+currently in the repo** — i.e. how wrong the borrowed palette was. The `repeat`
+column is the spread between the two readings of white and of black; if that is
+not small (say under 1 ΔE), the session drifted and the rest of the numbers are
+suspect.
+
+Then a ready-to-paste `palette_measured_rgb` block. Inks you skipped are
+carried through unchanged and marked `NOT MEASURED`.
+
+### Tone response
+
+This is the part that can change more than the anchors do.
+
+Error diffusion in this codebase propagates error in sRGB units, so it
+implicitly assumes that a patch which is `f` black by area reads as the
+`f`-weighted **sRGB** average of the two inks. Physically, area mixing is
+linear in **reflectance**, not in sRGB — those are materially different curves.
+On top of that, real electrophoretic ink has a dot-gain analogue.
+
+The report gives measured L\* against both predictions:
+
+```
+  black area  measured L*  pred L* (sRGB)    error  pred L* (refl)
+       50.0%        54.65           45.49    +9.17           56.58
+```
+
+The `error` column is the actionable one: it is the systematic mid-tone error
+the renderer is making on every image. Under ~2 L\* the linear assumption is
+fine. Much above that and a transfer curve applied before dithering is worth
+building — it would be a global quality win, not a per-image tweak.
+
+---
+
+## 7. After measuring
+
+1. Paste the new `palette_measured_rgb` into the model's `display.py`.
+2. Re-run the quality metrics — the LUTs, the DRC anchors and the metrics all
+   move with the palette:
+   `pytest -m time_intensive -k test_dither_quality_metrics`
+3. Update the table in [dithering.md](dithering.md) §1 (measured RGB / Lab /
+   hue / chroma per ink) and the benchmark table in §13.
+4. Note in the model's `display.py` docstring that the values are now measured,
+   with the date and the instrument — that provenance is exactly what is
+   missing today.
+
+`seeedstudio_e1004` currently inherits huessen's palette. If it is ever metered
+and differs, override `palette_measured_rgb` on that subclass rather than
+editing the huessen base.

docs/dithering.md

@@ -26,6 +26,12 @@ its 1200 × 1600 pixels can be exactly one of:
 | 4 | Blue   | (5, 64, 158)       | (29.83,  22.18, −55.47) | −68°      | 59.7   |
 | 5 | Green  | (39, 102, 60)      | (38.30, −30.62, 17.87)  | 150°      | 35.5   |
 
+These values, and the equivalents for the other screen models, come from
+`palette_measured_rgb` in each model's `display.py`. Their provenance is
+weak — see [color_calibration.md](color_calibration.md) for how to re-measure
+them on real glass with a colorimeter, and how to check whether the panel
+mixes inks the way the error diffusion below assumes.
+
 Two key properties:
 
 - **The palette is sparse.** Six anchors in a 3D colour space leave huge gaps.

docs/screens/bigme_f7/custom_firmware.md

@@ -75,6 +75,23 @@ OTA. The version is reported in `X-Firmware-Version` and the frame-state `fw`.
 - `wifi <ssid> <password>` — persist WiFi creds to sysinfo + connect.
 - `cfg show | server <url> | name <n> | ip <ip> <gw> <nm> | dhcp | static | sleep <s> | power <auto|sleep|awake> | save`
 - `ota` — trigger an OTA from the configured server right now (test hook).
+- `frame` — upload one ready-made 192000-byte panel buffer over this console and
+  display it. No server, no WiFi, no render pipeline: the host sends exact bytes,
+  the device shows them. For bring-up and colour measurement, where the picture
+  on the glass has to be known precisely (see [`../../color_calibration.md`](../../color_calibration.md)).
+  Wire exchange is in [`firmware/common/all/frame_proto.h`](../../../firmware/common/all/frame_proto.h);
+  drive it with `tools/f7_send_frame.py`. Takes the same OTA/refresh mutex as
+  `ota`, so it refuses rather than racing a refresh.
+
+  The device holds no patterns and no list — the host decides what to display
+  and uploads it, so new test images never need a rebuild or a reflash.
+
+  While a frame is in flight the console's UART RX callback is detached
+  (`console_disable()`), because the transfer needs the raw byte stream. Every
+  exit path restores it — success, CRC mismatch, and the bounded per-chunk
+  timeout alike — so a host that dies mid-transfer costs ~5 s, not the console.
+  `console_disable()` state is pure RAM, so a reboot restores it regardless, and
+  the mask-BROM replug+press catch is unaffected either way.
 - `upgrade` — SDK command that drops to the mask-BROM (used by the flashers).
   **Only our firmware answers `upgrade`; stock OEM does not.**
 

docs/screens/bigme_f7/measurements/README.md

@@ -0,0 +1,85 @@
+# Bigme F7 panel measurements
+
+Instrument readings taken off real glass. Raw data lives here so a later analysis
+can be re-run without repeating a physical session.
+
+Two datasets, in the order they were taken:
+
+- **[`data/`](data/)** — the full campaign. 1700 readings with spectra over 20
+  sessions, covering 1434 planned patches across eight phases (tone, LUTs,
+  algorithms, skin, dense gamut). This is the one to analyse.
+- **`f7_panel_2026-08-08.json`** — the first absolute-colorimetry run, described
+  below. Superseded in coverage, but it is what the dot-gain and contrast numbers
+  in this file were derived from, so it stays.
+
+## `f7_panel_2026-08-08.json`
+
+First absolute-colorimetry run on the F7. X-Rite ColorMunki Photo (s/n 2017853)
+via ArgyllCMS 3.5.0 `spotread -O -N -i D65`, reflective 45°/0°, instrument
+calibrated against its internal white tile immediately before the run.
+
+Method: the meter was **clamped in one position** on the glass and never moved.
+Each measurement field therefore fills the WHOLE panel — 6 flat inks, then 7
+Bayer levels at exactly k/8 black coverage — uploaded one at a time over USB with
+the `frame` protocol (`tools/color_measure_f7.py`). This is the meter-can't-move
+counterpart to the hand-aimed grid in `color_target.py`; it removes patch-aiming
+as an error source entirely. Firmware 1.2.11, poll URL parked so nothing could
+repaint mid-run.
+
+XYZ is percent-scaled (Y=100 for a perfect diffuser), two readings per field
+except `ramp 5/8` which has one — the second lost the instrument to a transient
+`Communications failure` and was not retried. Repeatability across held-still
+reads was ~0.4 %, so a single reading there is not a material loss.
+
+### Results
+
+Paper-white is **Y = 37.8 % (L\* 67.9)** and black **Y = 1.18 %** — a contrast
+ratio of **31.9 : 1**. That is the physical ceiling: nothing the render pipeline
+does can produce a brighter white or a deeper black than the ink itself.
+
+**Tone response is strongly non-linear.** The dither pipeline assumes palette
+inks mix linearly by area. They do not:
+
+| nominal | measured Y% | linear Y% | effective coverage | dot gain | ΔL\* |
+|--------:|------------:|----------:|-------------------:|---------:|-----:|
+| 0.125 | 30.38 | 33.21 | 0.202 | +0.077 | 2.3 |
+| 0.250 | 23.99 | 28.63 | 0.377 | +0.127 | 4.4 |
+| 0.375 | 17.47 | 24.06 | 0.555 | +0.180 | 7.3 |
+| 0.500 | 11.84 | 19.48 | 0.709 | **+0.209** | 10.3 |
+| 0.625 | 8.87 | 14.91 | 0.790 | +0.165 | 9.8 |
+| 0.750 | 6.28 | 10.33 | 0.861 | +0.111 | 8.3 |
+| 0.875 | 3.67 | 5.76 | 0.932 | +0.057 | 6.3 |
+
+A 50 % dither renders like **71 %** ink coverage — ΔL\* ≈ 10, far above any
+just-noticeable threshold. The curve is the classic dot-gain arch, largest in the
+mid-tones and closing at both ends, so it cannot be corrected by a gain or an
+exposure shift; it needs an inverse-transfer LUT applied before quantisation.
+
+**Ink anchors disagree with `palette_measured_rgb`.** Comparing absolute
+measurements to the palette directly is invalid — the palette encodes a rendering
+intent with white near sRGB white, while paper-white here is L\* 67.9. After
+white-normalising (so the panel's own white maps to the palette's white), the
+remaining error is hue and chroma:
+
+| ink | palette sRGB | measured, normalised | ΔE76 | Δchroma |
+|---|---|---|---:|---:|
+| black | (31, 34, 38) | (45, 30, 53) | 15.5 | +14.1 |
+| white | (185, 199, 201) | (185, 199, 201) | 0.0 | — |
+| yellow | (193, 187, 30) | (205, 199, 0) | 12.7 | +11.9 |
+| red | (98, 32, 30) | (142, 30, 23) | 24.4 | +22.4 |
+| blue | (35, 63, 142) | (13, 93, 170) | 15.2 | −2.7 |
+| green | (53, 86, 58) | (41, 108, 82) | 12.6 | +6.7 |
+
+Mean ΔE76 13.4, worst 24.4 on red. Δchroma is positive almost everywhere: the
+real inks are **more saturated** than the palette assumes, red substantially so.
+
+### Caveats before acting on this
+
+- **One unit, one session.** Panel-to-panel spread is unmeasured, so treat these
+  as this screen's numbers, not the model's, until a second unit is run.
+- **The meter rested on the glass.** 45°/0° geometry rejects the specular
+  component, which is the right way round, but contact pressure on a flexible
+  panel can shift the surface slightly. Unquantified.
+- Changing `palette_measured_rgb` moves every rendered image. The dot-gain
+  correction and the anchor update are separate changes and should be evaluated
+  separately — applying both at once makes a regression impossible to attribute.