F7 colour campaign: USB frame transport, A/B slot flashing, and a DRC fix - #39
Open
defl wants to merge 41 commits into
Open
F7 colour campaign: USB frame transport, A/B slot flashing, and a DRC fix#39defl wants to merge 41 commits into
defl wants to merge 41 commits into
Conversation
Restores the colour-calibration work onto a feature branch. It briefly went to main directly (2fcaa0c) while still in progress and was backed out there in 34da1e0; this is the same content, now landing through review. Every dither decision derives from one array per screen model (palette_measured_rgb in display.py), and the provenance of those numbers is weak: bigme_f7 borrowed a third party's, seeedstudio_e1004 inherited huessen's wholesale, huessen's own have no recorded method. tools/color_target.py generates a target of flat ink patches plus a Bayer tone-response ramp at exactly known ink coverage. tools/color_read.py walks the manifest, takes pasted tristimulus readings from any instrument, and emits a palette block plus the measured tone curve against what the renderer assumes. The "calibration_raw" preset renders that target pixel-exact — noop dither, every tonal and chromatic stage at its identity value — so the patches land on the inks they were painted in. The normal presets run autocontrast, gamma, CLAHE, unsharp mask and error diffusion, any one of which destroys a flat patch or shifts a ramp off its intended coverage. The tone-response half may matter more than the anchors: error diffusion propagates error in sRGB, so it assumes coverage mixes there, while area mixing is physically linear in reflectance. Those are different curves, and the gap is a systematic mid-tone error on every rendered image. docs/color_calibration.md covers the procedure; dithering.md points at it where the palette table is introduced. pyright/ruff extraPaths gain "tools" so the two new test modules can import the scripts the same way the scripts import each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a `frame` console command that receives one ready-made 192000-byte panel
buffer over the UART and streams it straight to the EPD. No server, no WiFi, no
render pipeline — the host sends exact bytes and the device shows them.
Motivation: the render pipeline is the wrong tool for bring-up and for the
colour measurement the calibration workflow needs. Metering a panel means
putting an *exact*, known raster on the glass on demand; going through the
server makes the result depend on the configured preset, on WiFi, and on which
host the screen happens to point at. This removes all of that from the loop.
The device stays a dumb pipe: it holds no patterns, no list and no state, so new
test images never require a rebuild or a reflash. Everything clever stays
host-side where it is already unit-tested — frames come from
Display.indices_to_panel_bytes(), the same packing the server uses.
common/all/frame_proto.{c,h}
Portable protocol: chunk arithmetic plus an IEEE CRC32 that is bit-identical
to Python's zlib.crc32 (pinned against five vectors in the host test).
Chunking is not decoration — the UART RX FIFO is shallow and the panel is fed
a byte at a time, so an unthrottled 192 KB blast overruns the device. The
per-chunk ACK is the flow control; this board wires no RTS/CTS.
bigme_f7 hokku_frame_receive()
Takes the existing OTA/refresh mutex, so it refuses rather than races a
refresh. A CRC mismatch or short transfer refuses to refresh at all — during
colour measurement a subtly wrong picture is worse than no picture.
Console recovery is the load-bearing property: taking the UART detaches the
console's RX callback, so every exit path restores it — success, CRC failure
and the bounded per-chunk timeout alike. A host that dies mid-transfer costs
~5 s, not the console (and with it the `upgrade` route into BROM).
console_disable() state is RAM-only so a reboot restores it regardless, and
the mask-BROM replug+press catch is unaffected either way. The rollback
watchdog is already stopped by hokku_rollback_commit() before any console
command runs, so the ~17 s transfer cannot trip it.
tools/f7_send_frame.py
--cycle walks every ink then the calibration target (the quickest check that
all six inks fire on a unit), --solid/--target/--bin for single frames.
docs: `frame` documented in the bigme_f7 console command list;
color_calibration.md gains Route A (serial) alongside Route B (server).
Verified: 7/7 common/all host tests including the new frame_proto, and the
firmware builds to a packed xr_system.img under -Wall -Werror. Not yet
exercised on hardware.
FIRMWARE_VERSION 1.2.10 -> 1.2.11 (N increment; no wire-protocol or config change).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turns the colour-calibration workflow from "paste numbers in by hand" into one
loop that can run every screen: put the target on the glass, meter each patch,
emit a palette block and a tone curve.
Written for the X-Rite ColorMunki Photo (CMUNPH) but instrument-agnostic.
Why a reflective spectro changes the design
-------------------------------------------
It has its own lamp, self-calibrates against a built-in white tile and reports
ABSOLUTE colorimetry, so the white-reference normalisation the emissive
colorimeter path needs is not just unnecessary here but wrong: applying it would
re-reference every reading to whatever patch was chosen as white and discard the
absolute lightness compress_dynamic_range() depends on. color_read.analyse() is
therefore split — assemble() holds the anchor/ramp derivation both paths share,
and only the colorimeter path runs normalise() ahead of it. No duplication.
The white-point trap this closes
--------------------------------
Reflective measurement is a print convention and instruments default to D50,
while this codebase is sRGB/D65 throughout. Feeding D50 XYZ into xyz_to_lab
silently shifts every ink anchor and nothing downstream would notice. Readings
are now Bradford-adapted to D65 first. The tests pin the adaptation's defining
property (source white maps exactly onto destination white), prove a D50 neutral
stays neutral in D65 Lab, and — so that test cannot pass vacuously — prove that
skipping the adaptation shifts b* by more than 3 units.
Scale is the other silent hazard: Argyll reports reflective XYZ as percentages
while the codebase uses Y = 1.0. That is judged once across the whole set, not
per reading, because a dark patch reads below 2 on either scale.
Structure
---------
tools/colorimeter.py instrument backends — a driven ArgyllCMS spotread, or
manual paste for any other meter/vendor software (which
also avoids Argyll's Windows USB driver having to
displace the X-Rite one).
tools/color_calibrate.py the loop, with target delivery abstracted per screen:
serial (Bigme F7 `frame` command — preferred, nothing
depends on server preset or WiFi), server (upload+pin,
and it REFUSES unless the server is on calibration_raw,
rather than measuring a target it cannot trust), or
manual. --all walks every registered model.
Verified end to end against a synthetic absolute session: a panel that measures
exactly the current display.py values round-trips to dE 0.00 on all six inks and
reproduces the palette block byte-identically. spotread's exact flags still need
confirming against real hardware, which is why they are configurable.
886 passed, ruff/pyright clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e active slot
The USB flasher could only write slot 0, so updating a unit already running Hokku
firmware from slot 0 had no USB path at all — the only option was the network OTA,
which needs a live console window to trigger and loses to the firmware's refresh
lock ("OTA busy"). Writing slot 0 on such a unit would have destroyed the only
known-good image.
Generalise the write instead of copying it: slot0.py becomes slots.py, with
flash_slot(f, img, slot=...) parameterised on the slot base, the region limit
(slot 0 is bounded by the OTA cfg, slot 1 by sysinfo at 0x300000) and the fdcm
raw-seq code. build_fdcm_seq0() becomes build_fdcm(seq). The validated sequence is
unchanged: assert the bootloader header + F2 xz guard, sector-bounded erase,
read-back verify, cfg flip last.
Add the safety check BROM makes possible: the OTA cfg records which slot the
bootloader would launch, so the active slot is knowable even though nothing is
running. flash_slot now refuses to write it unless allow_active_slot=True. The
bootstrap call sites (web GUI, f7_initial_flasher) pass that opt-in, since they
deliberately replace a stock/OEM image for which a full-flash restore exists;
behaviour there is unchanged. catch.py gains --slot/--allow-active-slot.
The image is slot-agnostic: main.c derives its XIP bias as
0x13040 + boot_seq * 0x179000 at runtime, so the same bytes boot from either slot.
Tests pin the slot argument at the bootstrap call sites so "writes slot 0" stays a
checked property rather than a comment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…for slot writes parse_cfg() crashed with struct.error on a real device. fdcm appends a new 4-byte entry per rewrite rather than updating in place, so this unit — flashed and OTA'd many times — had ~140 used entries and its live one sat at offset 684. Every existing caller only ever parsed a cfg we had just written ourselves (one entry, at offset 132), so a 512-byte read always sufficed and the bug stayed latent until the new active-slot pre-flight became the first caller to parse the LIVE cfg. Read OTA_SIZE (4 KB) instead of 512 at both call sites, and make parse_cfg return None on a short/empty buffer rather than raising, so a truncated read can never be mistaken for a valid answer. Rename flash_candidate_slot0_upgrade.py -> f7_flash_slot.py and give it --slot/--allow-active-slot. Its `upgrade`-console BROM entry is the route to prefer: it needs no USB replug, is unaffected by the firmware's "OTA busy" refresh lock, and never touches the network. Measured twice at ~2 s to sync with no button press. Correct catch.py's instructions, which claimed a long-press triggers the BROM catch. It does not: a long-press drops the CH340 for ~1.1 s and the window closes before the port reopens. The trigger is unplug -> replug -> short-press. Also record the diagnosis, since this cost a 14-minute failed attempt: the loop prints "waiting for port" only when the port actually drops, so a log of nothing but "hammering 0x55" means the cable is not being unplugged — not that the timing is off. Verified on hardware: slot 1 written and read back byte-identical, cfg -> seq1 VERIFIED, slot 0 preserved (AWIH intact). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… REFRESHED First run of `frame` against real hardware (1.2.11), and both bugs were in the host client, not the firmware. 1. `! no ACK after 0 B (got b'\n')`. _read_line() returns on the FIRST of \r or \n and leaves its partner queued, so the stray byte from the READY line was read as chunk 0's ACK and every upload aborted immediately. Drain the buffer after parsing READY; safe because the device sends nothing between READY and the ACK for a chunk we have not written yet. 2. `! expected REFRESHED, got 'hokku: frame received ... - refreshing'`. The device emits an hlog line between DONE and REFRESHED, and hlog can interleave at any moment, but the client tested a single line and declared failure. A 192000 B upload that had fully succeeded was reported as failed — and a caller retrying on that false negative repainted the panel four more times (18 s and a full ~30 s refresh each). Read until REFRESHED, treat ABORTED as a real failure, and surface anything else as device output. Verified on hardware: 192000 B at 10.3 KiB/s, CRC 0xF1BAD87D matched device-side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… results The grid target assumes the operator re-aims the instrument at each of 15 patches. A spectrophotometer resting on the glass cannot be moved without disturbing the geometry it measures, so invert it: the meter stays clamped and the PANEL changes, one full-screen field at a time. color_measure_f7.py uploads each field over USB (`frame`, never HTTP), waits for REFRESHED, and reads. color_target.fullscreen_sequence() builds the 13 fields off the same BAYER_8 and palette indices as build_index_raster, so a level here and the corresponding grid patch are the identical pattern at identical coverage. Ramp coverage is exact to 0.00e+00 on all seven levels. Two robustness fixes the first hardware run forced: * colorimeter: redirect spotread to FILES, not pipes. capture_output=True makes subprocess.run wait for the pipes to close, and the ColorMunki driver's background switch-monitor thread can hold those handles open after measuring — so the call blocked on a reading that had already succeeded (the meter visibly took both readings; the second never returned). The reading is parsed from the logfile, so the pipes were pure risk. On timeout, still try the logfile. * color_measure_f7: preflight one reading before displaying anything. A power-cycled ColorMunki has no stored calibration, so spotread forces one that fails the same way for every field — 13 uploads and 13 ~30 s refreshes to learn nothing. Instrument timeout 45 s, since a calibrated read takes ~1.5 s. Results (docs/screens/bigme_f7/measurements/): paper-white Y=37.8 %, black Y=1.18 %, contrast 31.9:1. Tone response is strongly non-linear — a 50 % dither renders like 71 % coverage (dot gain +0.209, dL* 10.3), a classic arch that a gain or exposure shift cannot fix. White-normalised ink anchors differ from palette_measured_rgb by mean dE76 13.4 (worst 24.4, red), with chroma positive almost everywhere: the real inks are more saturated than the palette assumes. Data is one unit and one session; caveats recorded alongside the numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cterisation
Plans a full characterisation run offline so it can be reviewed before eight
hours of hardware time are spent on it. Same seed -> identical patches in
identical order.
Five phases (8.10 h at the measured 56 s/patch):
tone_fine 65 Bayer k/64 — exact known coverage, no pipeline in the loop
algo_gain 91 is dot gain pattern-dependent? bayer vs FS/atkinson/stucki,
serpentine on and off
skin 180 60 colours sampled from the CIELAB skin band x 3 configs
gamut 125 5x5x5 RGB cube for a general correction LUT
control 60 white/black/mid re-measured every 25 patches and at both ends
Patches are shuffled WITHIN each phase: drift over an 8-hour run is slow, so
measuring a factor's levels in order would alias drift straight onto that factor.
The control patches then measure whatever drift remains, which is what allows two
patches measured hours apart to be compared at all.
Skin is sampled in Lab, not RGB — an RGB grid clusters in the bright end, and the
deep end (L* < 35) is both under-sampled by generic charts and where a 31.9:1
panel struggles most. Out-of-gamut candidates are rejected, never silently
clipped, so every patch is a colour sRGB can actually express.
Only dither() runs; the tonal chain is bypassed. A flat colour through the full
pipeline hits autocontrast, which has no dynamic range on a uniform field and
destroys the patch — the trap behind the bogus 0.00 % result in the earlier
dither search.
Validation on the real dither(): the same gray 128 yields DIFFERENT ink coverage
per algorithm (atkinson 0.393, FS 0.429, stucki 0.380). The algorithms do not
merely arrange the same ink differently, so algo_gain must be analysed against
achieved ink fraction rather than nominal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drives a multi-hour characterisation run: build each patch's raster, upload over USB, wait for REFRESHED, take N readings, record, repeat. The meter never moves. Crash-safety drove the file format. Results append to JSONL, one line per patch, flush + fsync each time, so a run that dies at hour six keeps every patch it measured and re-running skips completed uids. Nothing waits in memory for a clean exit that may never come. Each record carries what is needed to re-derive any conclusion without touching hardware again: requested colour, dither config, the EXACT achieved ink histogram (we generate the raster, so this is known rather than inferred), every raw reading, the median, timestamp and battery voltage. Median rather than mean over repeats: a spurious 'Communications failure' has already been seen mid-run, and a mean would quietly drag the result toward whatever that read produced. Battery is polled from the device every 20 patches. This unit reports usb_present=0 even when plugged in, so the voltage TREND is the only trustworthy evidence that a long run will survive; zeros (the ADC read during a refresh) are filtered out of the trend. color_target gains fullscreen_bayer_raster(k64) for exact k/64 coverage — BAYER_8 holds each value 0..63 once, so thresholding selects precisely k64 cells. fullscreen_ramp_raster now delegates to it rather than repeating the tiling. Validated on hardware: 4/4 patches, ink fractions exact (63/64 = 0.984375), three reads agreeing to ~0.01 %, battery 4180 -> 4180 mV held on USB. Measured throughput ~45 s/patch, so the 521-patch plan is ~6.5 h, not the 8.1 h estimated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e missing
The 521-patch campaign never measured yellow, red, blue or green as solid
fields. tone_fine and the controls are Bayer black/white only, and the gamut
phase requests colours like (255,0,0) but sends them THROUGH the dither, which
for a 6-ink palette returns a mixture, not 100 % of one ink.
That is not a missing data point, it is a missing basis. The six solid inks are
the primaries of the area-coverage model — every ink fraction in the dataset is a
weight on them — so without them this campaign has no anchors of its own and
would have to borrow from a different session, leaving the drift between the two
unmeasurable.
Adds source="ink" and:
* an `inks` phase, each primary measured 3x and shuffled through the run, so a
drifting primary shows as spread between its own repeats rather than
silently biasing every fitted coverage;
* a full six-ink block bracketing the campaign at BOTH ends, which is what lets
this session be compared with any other — without it a shift in the rig
between sessions is indistinguishable from a real change in the panel.
554 patches, 8.62 h. The 31 records measured before this fix are kept as
campaign_superseded_no_inks.jsonl rather than reused: uid assignment is
positional, so those uids now denote different patches and resuming against them
would mislabel data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ColorMunki loses its calibration when it sleeps on USB, and a run that hits that keeps going: every subsequent patch still uploads (18 s) and still refreshes the panel (~30 s), and records nothing. That is the expensive failure — the hardware time is spent either way — and on a 554-patch plan it could burn most of a day producing an empty file. Abort after --max-consecutive-failures (default 3) with an explanation of the likely cause and the exact recovery: recalibrate via the dial, re-run the same command, already-measured patches are skipped. Observed pattern behind the default: the instrument sustained 26 reads in the 13-field run and 12 in the validation without complaint, then failed after sitting idle between runs. So the risk is idleness, not read count, and a continuous campaign should keep it awake — but "should" is not worth eight hours of unattended hardware time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This unit discharges even while plugged in — measured 4200 -> 4160 mV over 30
min, about -80 mV/h — so a multi-hour campaign walks the battery down. Earlier I
called the power "confirmed" off a three-minute validation run, which was far too
short to show a trend; an hour of data contradicted it.
Three levels, so it cannot be missed between cycles:
* per-check "LOW — CHARGE THE SCREEN SOON" at or below --battery-warn-mv
* end of run "CHARGE THE SCREEN before the next cycle" in the summary
* hard floor stop cleanly at --battery-stop-mv rather than let it die
mid-refresh; a half-written panel plus an unmeasured patch is a
worse place to resume from than a deliberate stop
Defaults 3800 / 3600 mV. At the observed drain that is roughly five hours of
warning before the floor.
The battery stop sets a flag consumed after the normal record write, rather than
writing-and-breaking inside the battery block. The latter happened to work only
because break skipped the second write — one edit away from silently duplicating
every stopped record.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the moment it does The reflective calibration expires on a TIMER, not on use — measured at ~58 min of continuous reading, and ArgyllCMS exposes no way to extend it (-N only skips auto-calibration; there is no interval flag). So every campaign cycle ends the same way and needs a human to rotate the dial. That should be announced, not discovered. Two changes: * Predictive. At --calibration-warn-min (45 min) the run prints a heads-up that the cycle will stop soon and need the dial, so the interruption can be planned. * Precise. colorimeter now classifies a failed read: "calibration" when the output mentions it, else "unknown". The runner stops immediately on the former instead of burning through --max-consecutive-failures first. Those two failures are not free — each still costs an 18 s upload and a ~30 s panel refresh to produce nothing. The distinction matters because the responses are opposite: a glitched read should be retried, an expired calibration cannot be. Also fixes a mangled multi-line string introduced by editing this block with sed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records a rule the human had to state three times because I kept proposing the server route. Serving a target over HTTP does not measure the panel: the render pipeline re-processes the image (autocontrast, gamma, CLAHE, unsharp, error diffusion), so what reaches the glass is not the target that was generated. It also requires rewriting a live server's preset and the screen's server_url, and makes the result depend on WiFi and poll timing — so a session cannot be reproduced. The `frame` console command plus tools/f7_send_frame.py push exact bytes down the UART with no server involved. If a unit's firmware is too old to have `frame`, the answer is to flash it (subject to the STOP rules above), not to reach for HTTP. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Campaign 1's algo_gain held the LUT at "bw" on my assumption that a neutral grey
lands on black and white ink whatever the colour space. That is false, and
measurably so. At grey 119, atkinson+serpentine:
bw black .447 white .553
hue_aware black .260 white .486 red .088 green .166
oklab_hue_aware black .216 white .466 red .053 green .265
cam16ucs_hue_aware black .022 white .416 red .174 blue .066 green .322
The hue-aware LUTs deliberately build neutrals out of chromatic ink — 56% of the
panel under CAM16-UCS. Consequences: campaign 1's gain curves describe the `bw`
preset only, while `general` and `faces` ship `hue_aware` and are unmeasured; and
a single black-coverage curve is the wrong model when five inks are in play, so
the fit must be multi-ink off per-LUT ink histograms.
Campaign 2 (1074 patches, ~14 h at the measured 47 s/patch):
lut_gain 65 4 LUTs x 13 greys, algorithm pinned to the production
atkinson+serpentine, plus bayer at matched coverage
lut_algo_cross 13 is the LUT effect separable from the diffusion kernel?
gamut_dense 729 9x9x9 — campaign 1's 5x5x5 is too coarse to invert on a
device this non-linear
skin_dense 120 40 NEW skin colours (offset seed) x 3 configs
inks + control 147 its own primaries and drift controls
The shuffle-and-interleave logic is now shared by both campaigns rather than
copied: a campaign without its own anchors cannot be tied to any other session,
so that must not be something a new campaign can forget.
Campaign 1's spec regenerates byte-identical (sha256 of the patch list unchanged),
so the run currently in flight is unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… provenance Three things learned the hard way today, folded back into the measurement path. SPECTRA. We were recording XYZ only. XYZ is one projection of a reflectance curve through one observer and one illuminant, so a session without spectra can never answer "how does this look under tungsten?" — and these screens hang in rooms lit by tungsten, LED and daylight. It is the only part of a measurement that cannot be reconstructed later. `spotread -s` appends the full 380-730 nm curve (36 bands) to the same logfile row, so it costs nothing per reading and needs no second file. Wavelengths are parsed from the header rather than assumed, and a spectrum whose length disagrees with the header is dropped rather than mis-aligned. SESSION ANCHORS. The instrument is recalibrated between sessions and a recalibration is not a perfect restoration of the previous one. Every run now re-measures the six primaries FIRST, before any spec patch. Without that, a difference between two patches measured in different sessions is unattributable — and this campaign already spans several calibrations. Anchors carry session-scoped uids (s003_anchor_red) so resume never skips them: a new session must always produce a fresh set. PROVENANCE. Each row gains the session number and a sha1 of the exact bytes sent to the panel, so a re-derived raster can be proven to be the same stimulus that was measured. Each session emits a header row with the spec hash, instrument arguments, firmware/config string and starting battery, so a session can be interpreted without consulting anything outside the file. Idempotency is preserved and now tested: session numbers derive from the file rather than any stored state, a torn final line from a hard kill is skipped rather than fatal, and spec patches are still matched by uid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…asses Splitting the work into two campaigns only made sense while campaign 1 was already measured. Starting from scratch it is strictly worse: the coarse 5x5x5 gamut is superseded by the dense one, and a second campaign pays for its own ink anchors and drift controls all over again. I kept the split after the restart because that was the earlier instruction, without revisiting it when the economics changed. --campaign 3 emits the de-duplicated union, 1436 patches / ~18.75 h: inks 18 the basis everything is expressed in tone_fine 65 exact k/64 coverage, no pipeline in the loop lut_gain 52 4 LUTs on the SHIPPING algorithm (atkinson+serpentine) algo_gain 91 ordered vs diffused, and scan order lut_algo_cross 13 is the LUT effect separable from the kernel? skin 300 100 distinct colours (both seeds, deduped) x 3 configs gamut_dense 729 9x9x9 control 168 interleaved drift + brackets Phases run in that order and the runner resumes, so the answers arrive front-loaded: tone curve, per-LUT gain, pattern dependence and all the skin work land before the gamut sweep. gamut_dense is deliberately last because it is the only phase that can be abandoned part-way without invalidating anything measured before it — 9.5 h of the 18.75 is optional in a way nothing else is. Two skin seeds are merged and de-duplicated to 100 distinct colours rather than concatenated, so the same colour is not silently measured twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ches
Cycles halved (75 patches/58 min -> ~32/25 min) after I added "precise"
calibration detection. The detection was the cause.
When a one-shot read fails, spotread falls back to its interactive loop and
prints a banner containing "'k' to do a calibration". The classifier matched the
bare substring "calibration" anywhere in the output, so a transient USB glitch
was indistinguishable from an expired calibration — and the runner stopped the
whole run on the first one. Session 2 ended after 32 patches with 0 failed
patches and exactly 2 failed reads out of 114, one of them this:
Spot read failed due to communication problem.
Hit Esc or Q to give up, any other key to retry:
which is self-clearing; the next read succeeded. Intermittent ColorMunki comms
failures are a documented Argyll/USB issue, not a fault — 2 in 114 is a glitch
rate.
Three changes:
* classify on SPECIFIC phrases ("needs a calibration", "calibration failed",
"set instrument sensor to calibration position", "got abort or error from
calibration") and separately mark communication failures as "transient".
Calibration is tested first, so a genuinely failed calibration attempt — whose
text contains both — still stops correctly;
* retry transient failures (--transient-retries, default 2, 2 s apart), used by
both the patch loop and the session anchors;
* require the calibration signal TWICE before stopping, since one has already
proven to be a glitch wearing a calibration message.
Consequence for the record: the "~58 minute calibration expiry" I quoted is not
established. Some or all of those stops may have been this same
misclassification. The predictive warning still fires at 45 min but its
--calibration-expect-min is a guess, not a measurement, and is now labelled as
such.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…on a stale one
Correcting my own correction. When I found the misclassification I declared the
~58 min calibration expiry "not established". It is established — both things
were true at once:
calibration -> session 2 ran 29 min, killed by the misclassification bug
-> +57 min: a 3-read test PASSED, so I resumed without recalibrating
-> +62 min: session 3 died after 5 min with 14x "Calibration failed"
So the expiry is real at roughly 60 minutes, AND the bug was separately wasting
half of each cycle. Fixing the bug did not make the expiry go away.
The trap in "resume without recalibrating" is that a successful reading proves
the calibration is valid at that instant, not that it has any life left. I
resumed with about three minutes remaining and lost the cycle.
tools/f7_calibrate.py now performs the calibration AND records when, since Argyll
exposes no way to ask the age. color_campaign_run reads that stamp and aborts if
the calibration is older than --max-calibration-age-min (default 15), printing
how much cycle is left when it does start. --ignore-calibration-age opts out.
This makes the operational rule enforceable rather than something I have to
remember: calibrate, then start immediately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…les cleanly at it
The expiry is not physical and not a guess. ArgyllCMS's ColorMunki driver
hardcodes it, spectro/munki_imp.c:
#define DCALTOUT (1 * 60 * 60) /* [1 Hrs] Dark Calibration timeout in seconds */
#define WCALTOUT (24 * 60 * 60) /* [24 Hrs] White Calibration timeout in seconds */
if ((curtime - cs->ddate) > DCALTOUT) { ... dark_valid = 0; }
Three things follow. It is the DARK calibration that expires, not the white one
(the white tile is good for 24 h). It expires on elapsed time alone — no drift or
temperature is measured. And the source gives no justification for the hour; it
is a conservative policy constant, not an instrument property. It matches
observation exactly: valid at 57 min, expired by 62.
Since the deadline is known to the second rather than guessed, the runner now
stops CLEANLY five minutes before it, instead of measuring until reads start
failing and losing patches to errors that cannot succeed. The heads-up now
reports real minutes remaining rather than minutes elapsed against a guess.
Consequence worth stating plainly: a ~19 h campaign needs ~19 calibrations, and
no amount of robustness work changes that — it is a fixed hourly tax. Raising
DCALTOUT and rebuilding Argyll is possible (AGPL, source public) but should not
be done before measuring whether the dark reading actually drifts over a longer
window.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…-cal drift Argyll invalidates the ColorMunki's dark calibration after exactly one hour on elapsed time alone, measuring nothing. On this rig that is a dial rotation every hour — roughly 19 manual interruptions for the campaign — so whether the constant protects anything is worth real money. First attempt was to forge it: rewrite the calibration date in Argyll's cache and measure for hours on a knowingly stale dark cal. The date fields are there (two int32 time_t at offsets 85 and 3221, confirmed against a calibration we had just performed), but the file is checksummed: Checksum didn't verify, got 0xc845799b, expected 0x4445836c The algorithm is rotate-left-13-and-add (update_chsum in munki_imp.c) and the stored value is the last four bytes, but no contiguous byte range reproduces it and the implied seeds are not meaningful values — it covers the serialised fields, not the file bytes. Forging it convincingly means reverse-engineering Argyll's serialiser, and a subtly corrupt calibration is worse than no experiment. Abandoned; the cache file was restored and verified byte-identical. The same question answers itself from data we already collect. Anchors now run at BOTH ends of a session, bracketing one calibration: if a stale dark cal biases readings, the same six inks measured ~50 minutes apart on that one calibration must differ; if they agree to the 0.3 dE noise floor, the hour is over-conservative and we will have measured it rather than assumed it. The clean stop now reserves ANCHOR_BLOCK_S for the closing bracket, so a session ends measured rather than cut off. --no-closing-anchors buys the ~6 min back once the question is settled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pipeline assumes inks mix linearly by area; the panel disagrees by up to dE 12. The standard correction is Yule-Nielsen — R^(1/n) = SUM a_i R_i^(1/n) — where n describes how far light spreads under the halftone relative to its period. This device suits the model unusually well: one ink per pixel means no overprints, so the Neugebauer primaries ARE the six measured inks and there are no interaction terms to invent. Validated on the 263-patch archive: Yule-Nielsen n = 1.62 mean dE76 2.49, median 2.30, p95 6.35 linear-mixing baseline mean dE76 5.23 -> the model removes 52% of the error Per config, the fitted n independently reproduces the algo_gain result by an entirely different route: bayer 1.90 (most spreading), stucki_serp 1.74, FS 1.69, atkinson 1.61, hue_aware variants 1.45-1.47. The earlier finding came from an ad-hoc arch fit on effective coverage; this comes from a physical model on ink histograms, and they agree on the ordering. n is found by scanning rather than gradient descent: it is one bounded scalar with a cheap objective, and a scan cannot fall into a local minimum or wander somewhere unphysical. A residual of 2.49 dE against a 0.3 dE noise floor says the model is still missing something — most likely that n varies with coverage rather than being constant across the arch. Spectral fitting (now that spectra are recorded) and per-config n are the next refinements. The measured anchors are printed for palette_measured_rgb, with the warning that they are ABSOLUTE reflectance: paper white sits far below sRGB white, so adopting them unchanged darkens everything. Choosing the gamut-mapping intent is a decision, not something the measurement settles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… re-measuring The 9x9x9 gamut cube collapses hard on a device this gamut-limited: 729 input colours produce only 313 distinct rasters. The other 416 are byte-identical panel bytes, so measuring them can only reproduce a reading we already have — 5.43 h of panel time for nothing. Deduplication is keyed on the raster hash at RUN time, not applied to the spec. That distinction matters: patch uids are positional and the interleaved controls are numbered after all the phases, so shrinking a phase would silently renumber every control and remap already-measured uids onto different stimuli. Hashing leaves every uid untouched, and the campaign in flight stays valid. A random 3% of duplicates is measured anyway (--dedup-sample-rate). "Identical raster implies identical reading" is a very good assumption, but it is still an assumption, and sampling keeps it under continuous test for ~12 patches instead of trusting it across 416. The sampler is seeded per session so a resumed run makes the same choices rather than re-rolling. Rasters measured during the current run are added to the index as they complete, so duplicates within a single session are caught too, not just against history. Inherited rows carry duplicate_of and are excluded from being sources themselves, so a chain can never form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Screens candidate pipeline changes offline: dither a colour, take the EXACT ink fractions, predict what the panel really shows via the fitted Yule-Nielsen model, compare against what was asked for. No hardware time, and the model was fitted to real measurements of this panel rather than assumed. Two results, and the first one corrects me. ANCHORS DO NOT MATTER MUCH. I recommended fixing palette_measured_rgb first as "the root cause, biggest expected gain". On this evidence that is wrong: white-normalised measured anchors move skin dE 15.17 -> 14.32 (inside the model's own 2.5 dE residual) and make greys clearly worse (|dhue| 65 -> 89). Absolute anchors are worse everywhere. Aggregate dE turns out to be dominated by gamut limits no anchor choice can move — the gamut set scores dE 50 regardless — which is exactly why a single number was the wrong headline. THE LUT DOES. Reporting hue separately shows the artifact plainly. On skin, with everything else fixed: oklab blue shift -1.72 cam16ucs -2.73 euclidean -4.03 hue_aware -9.46 <- what general and faces ship oklab_hue_aware -10.09 hue_aware_weighted -10.19 and the pattern is the hue_aware modifier itself, not the colour space: euclidean -4.03 vs hue_aware -9.46, oklab -1.72 vs oklab_hue_aware -10.09, cam16ucs -2.73 vs cam16ucs_hue_aware -9.11. Adding hue_aware multiplies the blue shift in every space. That fits the known defect that its gate tests the drifted value's hue rather than the source pixel's, so warm tones misfire and recruit blue. This independently reproduces the human observation that started this work — "all but oklab make lips blue" — from measurements taken without reference to it. Caveat kept in the tool's own output: flat fields, a model with a 2.5 dE residual, and no tonal chain. A swatch-derived recommendation has already failed photo validation once in this project, so this is a screening result that says what to put on glass next, not a decision. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A swatch-derived recommendation has already failed photo validation once in this
project, so the flat-field result that hue_aware drives skin blue needed testing
on photographs before anyone acts on it.
It holds, and photographs are affected MORE than swatches. Predicting perceived
colour by locally averaging each ink's coverage (the eye integrates dots) and
applying the fitted Yule-Nielsen relation per position, then scoring only warm
skin pixels:
Robert De Niro (33.5% skin) r=2 r=6 r=12
hue_aware (production) -16.38 -16.72 -17.13
oklab +1.56 +0.88 +0.02
cam16ucs +0.69 -0.03 -0.93
euclidean -0.33 -1.06 -1.96
Anna Unterberger hue_aware -19.53 vs oklab -8.26
Wayuu woman hue_aware -7.05 vs oklab -0.91
Same ordering on all three portraits at every blur radius, and the swatch estimate
(-9.46) understated it: on faces the production config shifts skin -16 to -19 b*,
where oklab is near zero. An 18-unit b* gap is not subtle.
Skin is scored on its own because whole-image error is dominated by the panel's
gamut and hides the artifact people actually notice. The blur radius stands in for
chromatic acuity and is a modelling choice, so several are reported rather than
picking one and hiding behind it.
Still not a decision: no tonal chain, and changing the LUT moves every image, not
just faces. It says clearly what to put on glass next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Consolidates the characterisation into one place, including the two results that overturned my own earlier recommendations. The blue skin cast is caused by the hue_aware MODIFIER, not by the colour space it modifies — adding it multiplies the shift in euclidean, oklab and cam16ucs alike. Confirmed on three portraits, where the effect is stronger than swatches suggested (-16 to -19 db* against oklab's ~0). cam16ucs is better or equal to the current hue_aware on every stable metric while nearly matching oklab on blue shift, so it is the recommendation — to be validated on glass, since this is model-based screening with the tonal chain off. The palette anchors, which I had called the root cause and the first thing to fix, turn out not to matter: substituting measured values moves every metric less than the model's own residual, and makes greys worse. Aggregate error is dominated by gamut limits no anchor choice can move. Also records the rig lessons that cost the most time: the 1 h dark-calibration expiry is a hardcoded ArgyllCMS constant rather than physics, a passing reading says nothing about remaining calibration life, intermittent ColorMunki comms failures are normal and must not be treated as fatal, and the 6 mm aperture's apparent under-sampling of the Bayer period is bounded at ~0.004 coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… dry run mask a patch Two ways the resume logic could quietly lose data, both found by testing it rather than by it going wrong. A dry run displays patches and measures nothing, but its rows were written with a uid like any other, so done_uids counted them. A single rehearsal against the real output file would have permanently masked those patches from every later run — the data simply absent, with nothing in the file to say why. Dry-run rows are now tagged and skipped when computing what is done. More generally, resume treated "attempted" as "done", so a patch whose upload glitched or whose calibration expired mid-read was recorded with an error and never revisited. Every failure mode seen in this project is transient, so those should come back around; a permanent hole is the worst outcome because it is invisible in the results. done_uids now requires an actual measurement. On the live campaign this reclaims 6 patches — one tone_fine and five session-3 anchors lost when the calibration expired — which will now be re-measured instead of silently missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A 2.4-minute dry run reported "battery 4200 -> 4160 mV (-40) DRAINING". A 40 mV discharge in 2.4 minutes is not physical — it is the ADC. Every sample taken all session sits inside a ~+/-40 mV band (4200, 4180, 4160), which means the earlier "-80 mV/h, roughly a campaign-length from cutoff" reading was almost certainly the same noise, and I reported it to the user as a real constraint. The threshold now sits outside the observed noise, and the message states the window the delta was measured over so a small drop across two minutes cannot read as a trend. The warn/stop floors are unchanged: those compare an absolute voltage rather than a difference, so noise costs at most one early stop instead of a wrong conclusion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…global check Recommending a global LUT change from a skin metric was only half an argument. Whole-image mean dE over the 12 readable test images, radius 6: hue_aware (production) 19.34 oklab 18.95 cam16ucs 17.37 euclidean 17.31 cam16ucs beats production on 11 of 12 images; the exception is the Wikipedia logo by +0.08. The gains are not confined to faces — the RGB corner gradient improves 40.22 -> 33.84 and the Albi panorama 20.69 -> 16.55. This also disqualifies oklab as a global default, which the skin metric alone would have recommended: despite the best blue shift, it is WORSE than production on the Duerer hare, Berlin Wall, Fitz Roy, the B&W forest road, the grayscale bar and the Wikipedia logo. It fixes skin by spending accuracy everywhere else. cam16ucs gives nearly the same blue-shift improvement without that cost. The 100 MP synthetic image is skipped rather than decoded: the server gates images that size at ingest, and raising PIL's limit here would hide that policy instead of respecting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cam16ucs The skin metric alone would have picked oklab. Scoring all 12 readable test images shows that would have been a mistake: oklab is worse than production on six of them, fixing faces by spending accuracy everywhere else. cam16ucs gives nearly the same blue-shift improvement and beats production on 11 of 12 images. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…had none lut_gain measured bw plus the three hue_aware variants, because I designed that phase before the evidence pointed at the hue_aware MODIFIER as the cause of the blue skin. So the LUT now being recommended — plain cam16ucs — has no measured grey data at all, and the whole case for it rests on the model. That gap matters because the measured greys showed something the model did not predict. For a requested neutral grey, mid-range: atkinson_serp_bw chromatic ink 0.000 |cast| 6.36 (b* -5.17) atkinson_serp_cam16ucs_hue_aware 0.343 |cast| 5.26 atkinson_serp_oklab_hue_aware 0.221 |cast| 4.91 atkinson_serp_hue_aware 0.132 |cast| 3.64 bw — black and white ink only — has the WORST neutral cast, and it is blue: the black ink's own violet bias showing through. The hue-aware LUTs REDUCE it by spending chromatic ink to cancel it, which is presumably why the mechanism exists. So hue_aware is best on greys and worst on skin, and a plain LUT might trade one for the other. Campaign 4 (--campaign 4) adds 39 patches: oklab, cam16ucs and euclidean over the same 13 greys on the production algorithm. 60 patches with controls, ~47 min. It is a SEPARATE spec with its own uid prefix rather than an addition to the existing one. uids are positional and the interleaved controls are numbered after the phases, so appending would renumber every control and remap already-measured uids onto different stimuli — the same trap that ruled out spec-level gamut dedup. Verified: 0 uid collisions, main spec sha unchanged, 1436 patches intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plain-LUT grey supplement answered the gap it was built for, and the answer was not the one I expected. Mid-range neutral cast, production algorithm: euclidean 3.56 cam16ucs 5.55 hue_aware 3.64 bw 6.36 oklab_hue_aware 4.91 oklab 9.12 cam16ucs_hue_aware 5.26 Plain cam16ucs costs grey neutrality (5.55 vs hue_aware's 3.64), so the previous recommendation was buying skin at the price of neutrals — the exact risk that justified measuring it. And plain oklab is catastrophic on greys at 9.12, worse than black-and-white-only, landing at a* -8.0 because it recruits more chromatic ink than anything else. That is the LUT the skin metric alone would have chosen. euclidean is best or tied-best on grey cast (measured), whole-image dE, skin dE and skin hue angle, losing only on blue shift where it is still less than half of production's. It is also the simplest option — plain CIELAB nearest-neighbour, no hue machinery, no perceptual space. Every more elaborate alternative measured worse. The ranking is identical using only freshly-measured patches (23 of 39 were inherited from byte-identical rasters), so dedup did not manufacture it. Also fixes the calibration guard, which refused a perfectly usable 39-minute window because 21 minutes had already been spent on a short run. It gated on AGE when what matters is time REMAINING; --min-usable-min replaces --max-calibration-age-min. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s, and the metric misled Two failures, one technical and one methodological. TECHNICAL. Every LUT comparison in this work used hue_cutoff_deg=30, neutral_chroma=12. Production ships 95, 8, and those two numbers ARE the hue-aware gate. I picked 30/12 when designing the campaign and never checked it against presets.py, so a day of conclusions describe a config we do not run. At the shipped values hue_aware is BIT-IDENTICAL to euclidean on skin colours (verified by raster hash), so two of my three successive recommendations — cam16ucs, then euclidean — were no-ops I did not know I was proposing. Re-run correctly, production measures -5.61 db* (general) and -4.77 (faces), not -9.46, and oklab is the genuine outlier at -1.15. That vindicates the original human observation that "all but oklab make lips blue". METHODOLOGICAL, and the more useful lesson. Shown side by side on the panel at correct parameters, the human verdict was that PRODUCTION looks better: oklab reads "too light, overlighted" while the shipped config gives "more human" skin. That matches whole-image dE (17.16 shipped vs 18.27 oklab) and contradicts the blue-shift metric I had made the headline. The dE column predicted preference; blue shift did not. Continuing to optimise blue shift would have kept generating recommendations a human rejects. No LUT change is recommended. f7_ab_compare now defaults its gate parameters to the shipped values with a comment recording why guessing them was expensive. Unaffected, because none of it involves the hue gate: ink primaries, gamut, contrast, black-ink non-neutrality, the dot-gain curve, and the Yule-Nielsen model. The dot-gain finding (a 50% dither prints like 71%, mean dE 6.85) remains the largest measured error and is still uncorrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… Huessen's compress_dynamic_range squeezes an image's lightness into the panel's reachable range. It read that range from the module-level PALETTE_LAB / PALETTE_OKLAB, which dither_streaming derives from the Huessen EPF1301 reference display — regardless of which screen is being rendered for. The import comment said so plainly; the consequence had never been measured. On the F7 it is severe. Measured on our glass the panel spans L* 10.21..68.02; the DRC was compressing into 0.55..79.86, 37% wider, so both ends clipped: image clipped before after Robert_De_Niro_KVIFF_portrait.jpg 52.0% 0.0% Albi_Panorama_Sunset_Panini_General 16.9% 0.0% Forest_road_Slavne_2017_BW_G9 10.1% 0.0% Fitz_Roy_1.avif 4.7% 0.0% Half the De Niro portrait was mapped below the panel's black floor and collapsed into one flat tone. The tanh soft shoulder could not help either: it began rolling off at L* 68.0 and ran to 79.9, entirely beyond the panel's white, so the mechanism meant to protect highlights never engaged at all. The renderer now supplies anchors from the display it was constructed with. Displays gain an optional drc_anchor_l for panels that have been photometrically measured; without it the range comes from that display's own palette, still correct-by-construction for the screen in hand. Huessen and Seeed are unaffected (0.55..79.86 either way). bigme_f7 carries its measured 10.21..68.02. drc_anchor_l is deliberately separate from palette_measured_rgb. That table drives ink SELECTION, where a few dE hardly matters because the gamut dominates — an earlier experiment substituting measured values there changed almost nothing. This drives what range the entire image is squeezed into, where being wrong clips everything past the end. Same numbers, different mechanism, opposite sensitivity. Verified side by side on the panel through the full production path with only the anchors differing, and the human verdict was that the corrected side looks better. That matters: three earlier candidate improvements today measured as better and were rejected on sight. This is the first that survived a human looking at it. 886 tests pass; the DRC helpers keep their previous defaults, so existing callers and tests are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Session 10 started with 20 minutes of calibration left, passed the guard, and produced 6 spec patches. The two six-ink anchor blocks are ~11 minutes of that window and the safety margin another 5, so only ~4 minutes went to actual patches — a dial rotation spent almost entirely on overhead. The guard compared the remaining time against a flat 12-minute floor as if all of it were usable. It now adds the overhead it can predict (one anchor block, or two when closing anchors are on, plus the margin) to the minimum PATCH time, so --min-usable-min means what its name says. A cycle now needs ~31 min remaining rather than 12, i.e. it must start within ~29 min of calibrating. The abort message breaks the requirement down rather than printing one number, so the next person can see where the time goes instead of rediscovering it. Root cause was operational, not code: the hour starts when f7_calibrate runs, not when the run does, and I calibrated ~40 minutes before the operator was ready. Calibrate only when the panel is ready to start immediately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… not waste The raster dedup inherited 56 control patches instead of measuring them, silently disabling the drift instrumentation it was supposed to leave alone. Controls are ctl_white / ctl_black / ctl_mid interleaved every 25 patches. They produce byte-identical rasters BY DESIGN — repeatedly measuring an unchanged stimulus is the only way drift becomes visible. To the dedup that looked exactly like the 416 redundant gamut-cube duplicates it was written to skip, so it skipped them, and each "saving" was a drift sample not taken. Controls are now excluded both as dedup targets and as dedup sources; a control must never seed the index either, or an ordinary patch could later inherit from one. The 56 already lost are not force-re-measured: that is ~44 min of panel time for data the six session-anchor brackets already cover better (six inks bracketing a whole calibration, ~0.44 dE consistently). The remaining 114 controls, most of them interleaved through the seven hours of gamut_dense still to run, will now be measured properly. Checked at the same time and fine: the skin phase is not degenerate — 86 of 100 colours produce three distinct rasters across the three configs, 11 produce two, only 3 collapse to one. Its 40 inherited patches came from uniform-field colours matching solid-ink rasters, which is legitimate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oughput The header printed "1436 patches, 1438 already done", which is impossible. It counted every uid on file, and that includes the per-session anchors — extra measurements that live outside the spec by design. It now reports done/to-go against the spec itself. The per-run ETA also still used the original 56 s/patch estimate. Measured throughput across nineteen sessions is 47 s, so every ETA has been ~20 % long. The line now says "if none inherit" too, since dedup means the real figure is lower again — currently about half the remaining gamut rows cost no panel time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The campaign is closed. The measurements lived only in build/, which is gitignored, so twenty hours of panel time across twenty human-attended calibration cycles existed on exactly one disk. This commits them. 1700 readings over 20 sessions covering 1434 of 1436 planned patches, every one carrying its full 380-730 nm reflectance spectrum. The spectra are the reason to store this rather than a summary: XYZ is a projection of the spectrum through one observer and one illuminant, so a dataset recorded without them can never answer a question about a different illuminant without going back to the hardware. The two uncollected patches are `control` rows, dropped deliberately — they would have cost a full calibration cycle to add two drift samples to the twenty-plus brackets already collected. No gamut, tone or skin patch is missing. The `firmware` provenance field carried the test unit's LAN address on 20 rows; ip=/gw=/nm= are redacted. A field-by-field diff against the working copy differs only in `firmware`, only on those rows — no measured value is altered. The README documents the two things a reader would otherwise get wrong: that `duplicate_of` rows were inherited rather than measured (~54 % of gamut_dense, so they are real data but not independent samples), and that the session anchor brackets put a ~0.44 dE noise floor under every conclusion drawn from this file. .gitattributes pins the dataset to LF. `text=auto` would have rewritten it on a Windows checkout — harmless to the JSON, but it breaks byte-for-byte round-trip, so a checksum taken on one platform would disagree on another for a reason unrelated to the data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…generable The findings document was written incrementally while measuring, so each section used whatever subset existed on the day it was written. Now that the campaign is closed at 1700 readings over 20 sessions, every number is re-derived from the committed dataset. That immediately caught a figure that was wrong on its own terms: the gamut was reported as both 55 % of sRGB and 18.5 % of the visible locus. Those two references differ by a fixed ratio, so the pair could never both be right — 55 % of sRGB is 32 % of the locus. Measured over the full dataset it is 46 % and 26 % (xy hull area 0.05165). No remeasurement was needed to know the old pair was broken, only a division, which is the argument for deriving numbers rather than typing them. tools/color_findings.py now regenerates every headline figure, so the document can be checked rather than trusted. Other numbers that moved with the larger n: contrast 31.9 -> 33.0 : 1, Yule-Nielsen n 1.62 -> 1.51 (residual 2.49 -> 1.72 dE, now removing 70 % of linear-mixing error), and a noise floor of 0.35 dE established from 206 anchor readings rather than asserted. Dedup is now validated on 4705 independent pairs of identical rasters rather than 166: mean 0.497 dE, indistinguishable from the noise floor. The LUT retraction is promoted from a banner into the body. The campaign's honest result is that no dither change is warranted, and the reason is worth keeping: every LUT comparison ran at hue_cutoff_deg=30/neutral_chroma=12 while production ships 95/8, where hue_aware is bit-identical to euclidean on skin. Three successive recommendations were no-ops. Of four candidates put on glass, the three argued from preference were rejected by a human on sight; the one that survived was the DRC correctness bug. display.py records that the campaign corroborates its DRC anchors (L* 10.18/67.96 vs the 10.21/68.02 it carries) without editing them — the difference is an order of magnitude inside the noise floor, so changing them would be churn that reads like a real change in git blame. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in presets.py, and both sides were wanted: main restructured the module around three measured per-pipeline defaults, this branch added the `calibration_raw` service preset. Kept main's structure and re-added the preset below the alternatives, where it belongs — it is a passthrough for images already authored in the panel's inks, not a starting point for photos. The merge makes the DRC fix on this branch matter more, not less. main moved the general and black-and-white pipelines' dynamic-range compression into OKLAB; `drc_anchor_l` feeds both the CIELAB and OKLAB compressors, so the correction still applies. Had it been written against one space only, this merge would have silently disabled it for most photos. The hue-aware gate is unchanged at 95/8, so the central finding on this branch — that hue_aware is bit-identical to euclidean on skin at shipped values — survives the merge intact. 1041 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gain CI could not build the F7 host tests at all: test_logic.c compiles main.c as one translation unit against mocked SDK headers, and this branch's `frame` command added #include "console/console.h" and "driver/chip/hal_uart.h", neither of which had a mock. Adds both, plus frame_proto.c to the translation unit — its crc32 and chunk_size were unresolved at link once the compile succeeded. The mocks are exercised rather than left as scaffolding. hokku_frame_receive() borrows the UART from the console for the length of a transfer, and the property worth pinning is that the borrow is always balanced: every path out must re-enable the console. A path that does not leaves the device with no console, which on real hardware means no way back in short of a USB replug — during a routine whose entire purpose is to be run repeatedly while measuring colour. Three tests: a complete transfer ACKs once per chunk and balances the handover; a host that vanishes mid-transfer still gets the console restored; and a busy OTA lock refuses without touching either the console or the UART. The UART mock generates its stream rather than copying from a buffer — a frame is 192000 bytes, which is a lot of .bss for a test that never inspects the content — and "how many bytes will the host still deliver" makes the truncated case a one-line setup. 67 host assertions pass, up from 56. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Characterises the Bigme F7 panel with a spectrophotometer, and fixes the one real
bug that fell out of it. Merged with
main; 19/19 checks pass.The product change
One, and it is a correctness bug rather than a preference:
The dynamic-range compressor derived its lightness anchors from
palette_measured_rgb, a third-party table whose white sits at L* 79.3. Thisglass tops out at L* 67.96. Every image was being compressed into a range
37 % wider than the panel can display, so both ends clipped — on one test
portrait, half the image collapsed into flat black.
Fixed with
drc_anchor_lon the display spec. Screens that have not beenmeasured keep the old behaviour exactly. Confirmed better on glass, side by side.
This merge makes it matter more:
mainmoved the general and B&W pipelines' DRCinto OKLAB, and the fix feeds both the CIELAB and OKLAB compressors. Written
against one space only, it would have been silently disabled for most photos.
No dither LUT change is recommended. That is not where this was expected to
land — see below.
The dataset
docs/screens/bigme_f7/measurements/data/— 1700 readings over 20 sessions,every one carrying its full 380–730 nm reflectance spectrum, covering 1434 of
1436 planned patches. Roughly 20 hours of panel time across 20 human-attended
calibration cycles, previously living only in a gitignored
build/.Spectra are the reason to store rather than summarise: XYZ is a projection
through one observer and one illuminant, so a dataset without them can never
answer a question about a different illuminant without going back to hardware.
The device's LAN address appeared in a provenance field on 20 rows and is
redacted; a field-by-field diff confirms no measured value was altered.
What the panel actually is
A 50 % dither prints like 71 % coverage. That is real, but it does not imply
the pipeline needs a dot-gain correction — error diffusion is a closed loop and
already compensates. Applying an inverse-transfer LUT on top would double-correct.
The part worth reading
Three successive LUT recommendations from this work were wrong, for one
reason: every comparison ran at
hue_cutoff_deg=30, neutral_chroma=12whileproduction ships 95, 8. At shipped values
hue_awareis bit-identical toeuclideanon skin. The effect being characterised belonged to a configurationthat has never shipped.
Four candidates went onto real glass. The three argued from preference
(
cam16ucs,euclidean,oklab) were rejected by a human on sight. The one thatsurvived was the DRC bug. Whole-image ΔE predicted human preference; the
blue-shift metric the analysis had been optimising did not.
findings.mdrecords this in the body rather than as a footnote, andtools/color_findings.pyregenerates every number in it from the dataset — whichimmediately caught a gamut figure that was impossible on its own terms (reported
as both 55 % of sRGB and 18.5 % of the visible locus; those differ by a fixed
ratio and cannot both be true).
Firmware
frameconsole command (main.c,command.c) — receives a full panelimage over the UART with per-chunk ACK and a CRC32 check. The protocol itself
lives in
firmware/common/all/frame_proto.{c,h}, SoC-agnostic and shared, withits own unit tests.
the UART from the console for the length of a transfer, and the tests pin that
the borrow is always balanced — a path that returns without re-enabling the
console leaves a device with no way in short of a USB replug. 67 host
assertions, up from 56.
Tooling
tools/f7_send_frame.py— the host side of the above. Calibration targetsgo over USB/serial only, never HTTP;
AGENTS.mdnow records why.tools/f7_flash_slot.py+hokku/common/xr872/slots.py— generalised A/Bslot flashing, replacing a tool that could only write slot 0 (the running
slot). Refuses to write the active slot unless explicitly overridden.
calibration_rawpreset — neutralises every stage so an image authored inthe panel's own inks reaches the glass unchanged.
tools/f7_calibrate.py— records calibration time, because ArgyllCMS's darkcalibration expires after a hardcoded hour (
DCALTOUT) and a passing readingproves validity now, not remaining life.
Not included
The 3-D correction LUT the dense gamut phase was collected for. That build is
entirely offline and has not been started.
🤖 Generated with Claude Code