Skip to content

feat(#289): I2C dropout back-off and re-probe for every bus device - #294

Merged
Strycher merged 10 commits into
mainfrom
fc/289-i2c-backoff
Sep 9, 2026
Merged

Strycher merged 10 commits into
mainfrom
fc/289-i2c-backoff

Conversation

@Strycher

@Strycher Strycher commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Closes #289.

What changed and why

On 2026-09-08 the touch controller, then the IMU breakout, went silent on the I2C bus for hours. Every reader retried its device on every loop pass, each failure costing two error lines to serial and SD, so the loop fell from 44 Hz to 12-25 Hz and the logs filled with a million identical ESP_ERR_INVALID_STATE (NACK) lines. Nothing recovered without a reboot, and the boot log could not say why the last run ended.

New unit src/i2c_health.{h,cpp}, and a serviceX() in each I2C unit called from loop():

  • Health check is a chip-ID register read compared to the expected value, on Wire directly: LSM6DSOX 0x0F=0x6C, LIS3MDL 0x0F=0x3D, FT6336U 0xA8=0x11, BME688 0xD0=0x61. An address probe was the first design and the bench disproved it: the LIS3MDL ACKed its address on every scan while failing every data byte. SHT41 and MAX17048 have no ID register; their own read/isDeviceReady() results feed the same counters.
  • Drop after 5 consecutive failures or 8 of the last 16 checks. Dropping clears the unit's available flag, which gates its reads in loop(), and logs once.
  • Re-probe every 10 s: check, on failure a bus recovery (nine SCL clocks with SDA released, START+STOP, Wire.end()/begin()), check again at the recorded address and at the chip's alternate address (0x6A/0x6B, 0x1C/0x1E, 0x77/0x76). When it answers, the unit's normal init runs.
  • Init read-back: initIMU() clears both flags first and reads a configuration register back on each chip; a chip that ACKs WHO_AM_I but NACKs its configuration writes (the 17:38 boot did exactly that and the library said OK) is refused.
  • Flap back-off: a device that drops again within 30 s of returning waits 10, 20, 40 ... 320 s; 30 s of good checks clears it.
  • Boot log prints the reset reason (panic, watchdog, brownout, power-on).
  • Cadences: IMU every 200 ms, touch 1 s, gauge 2 s, SHT41 on its read, BME688 on the 3 s BSEC cadence.

Docs: bench STEMMA chain order and the 4517 board facts in HARDWARE.md; the redirected-output flash hang in DIAGNOSTICS.md; the scan now names 0x29 as the TSL2591.

What the hardware said

Flashed to the UM bench board at commit 8c96702 and run for 100+ minutes:

  • The fault is the LIS3MDL's floating address-select input on the 4517 board. On the old firmware it looked like a dead bus. On this firmware the log names it: 53 drops, most returns at the other address (answers at 0x1E now, not 0x1C), some at the same one, a re-drop within a second of each re-init. The 8 of 16 rule fired on the intermittent phases. Hardware fix: bridge the ADM jumper (owner).
  • Loop 44-53 Hz throughout. About 100 error lines an hour against 80,000 before.
  • The first live bus recovery exposed a digitalWrite()-before-pinMode() ordering; fixed in this branch (4a9c9aa).
  • Touch wake, GPS, SHT41, BME688, gauge all normal for the whole run.

Builds: um_feathers3 RAM 22.7% Flash 27.6%; feather_s3 RAM 31.6% Flash 64.7%.

Gemini gate (four passes, $0.10)

Fixed: BME688 status narrowed to BME68X_E_COM_FAIL; the 8-of-16 window (an intermittent chip kept resetting a consecutive counter); init read-back for a chip that NACKs its configuration; the failure window cleared only by a successful init, not by the re-probe; LIS3MDL read-back on the operation mode rather than the data-rate bits; back-off decays with good behaviour.

Rejected with evidence: "begin() leaks the old I2C device" (all five libraries delete before new); "Wire re-entrancy between endTransmission(false) and requestFrom" (straight-line code, one task); "a long SD write makes the next probe time out" (the Wire timeout is per transaction); "bus recovery aborts an OLED transaction" and "web task reads flags mid-init" (single loop task, synchronous WebServer); "the LIS3MDL ignores the 155 Hz data rate in medium mode" (the library sets ultra-high mode itself; the read-back now checks the operation mode anyway).

Accepted as designed: a noisy SHT41 can be dropped for 10 s after 8 bad reads in a third of a second; re-init of one IMU chip re-inits both (one breakout); no hot-plug of a device absent at boot (never existed).

Verification still owed

Owner flashes the branch tip and confirms on hardware. The board currently runs 8c96702; the three commits after it (recovery pin order, back-off, back-off decay) are built but not yet flashed.

🤖 Generated with Claude Code

Strycher and others added 10 commits September 8, 2026 17:25
A device that stops answering cost a driver timeout per loop pass and an
error line per read: on 2026-09-08 the touch controller and then the IMU
breakout went silent for hours, the loop fell from 44 Hz to 12-27 Hz and
the log filled with a quarter of a million identical errors.

i2c_health: per-device consecutive-failure count; after 5 the device is
dropped (its reads stop, logged once), re-probed every 10 s (probe, then
a 9-clock bus recovery and a second probe), and re-initialised through
its normal init when it answers. Only devices present at boot are
re-probed. The failure signal per library, checked in the pinned sources:
SHT4x getEvent() is false on a NACK; BME68x leaves a negative
sensor.status, sampled on the 3 s BSEC cadence; MAX17048 isDeviceReady()
reads the version register; LSM6DS and LIS3MDL getEvent() return true
unconditionally and FT6206 touched() has no error path, so those are
address-probed (per read pass for the IMU, every 1 s for touch).

The boot log now states the reset reason, so a crash boot is attributable
from the next boot's log.

Builds: um_feathers3 RAM 22.7% Flash 27.6%; feather_s3 RAM 31.6% Flash 64.7%.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Any negative sensor.status was being counted as a bus failure. Null-pointer, self-test and invalid-length codes are not bus faults and a re-probe would not clear them, so a chip with an internal fault would have been dropped and re-initialised every 10 s for nothing (Gemini finding on #289). Both envs rebuilt, sizes unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The live bench disproved the first design while it was under review. The
LIS3MDL on the 4517 breakout has been NACKing nearly every data byte for
hours, yet the boot bus scan lists 0x1C on every boot: the chip ACKs its
address and fails the transaction after it. An address-only probe would
have called it healthy all night. At the 17:38 boot its WHO_AM_I read
succeeded, every configuration write after it NACKed, and begin_I2C()
still returned true. A reset did not clear the fault; the LSM6DSOX on the
same board and cable reads fine, so it is the chip, not the wiring.

- i2cReadReg(): write register, repeated start, read one byte, on Wire
  directly (the drivers' own transaction shape). i2cCheck() compares the
  chip-ID register to the expected value: LSM6DSOX 0x0F=0x6C, LIS3MDL
  0x0F=0x3D, FT6336U 0xA8=0x11, BME688 0xD0=0x61. SHT41 and MAX17048 have
  no ID register and keep the address probe for the re-probe only.
- IMU check moves off the read path onto a 200 ms timer in serviceIMU().
- Two-address chips record the alternate; the re-probe tries it after
  the bus recovery and swaps addr if the chip answers there.
- Drop on 5 consecutive failures OR 8 of the last 16 checks, so a chip
  that answers one read in ten cannot keep resetting the count
  (review pass 2, finding 2).
- initIMU() clears both flags first and reads one configuration register
  back on each chip; a chip whose configuration writes were NACKed is
  refused instead of declared available while powered down
  (review pass 2, finding 3; the 17:38 boot did exactly that).

Builds: um_feathers3 RAM 22.7% Flash 27.6%; feather_s3 RAM 31.6% Flash 64.7%.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the operation mode

Third review pass. The re-probe cleared a device's failure window on a good WHO_AM_I read before its init had run, so a chip that passes the probe but fails its configuration forgot it was flaky; now only a unit's init clears the window, through i2cNoteReady(), when it succeeds. The LIS3MDL read-back moved from the data-rate bits to the operation mode, the setting whose failure yields zero data and whose register default and failed-read value are both power-down. Both envs rebuilt, sizes unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Chain on the bench, from the owner on 2026-09-08: Feather -> BME688 -> SHT41 -> LSM6DSOX/LIS3MDL (one 4517 board) -> TSL2591 (unused). Moving the IMU board up the chain did not change the failure rate (one failed IMU-board transaction per loop pass, 22.4/s, steady for 75 minutes either side), and the failing read moved from the LIS3MDL to the other chip on the same board with no reboot. The scan labelled 0x29 as a VEML7700; the device there is the TSL2591. Both envs rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
From the product page and the pinouts guide, read 2026-09-08: one shared regulator, shared FET level shifters with 10k pull-ups, ADM/ADAG address jumpers (open = 0x1C / 0x6A), INT pins unwired. Shared parts would take both chips down together; the bench fault takes one at a time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pio-flash confirm hung twice at what looked like Connecting... for ten minutes. PlatformIO's echo thread died with UnicodeEncodeError (cp1252) on esptool's Unicode progress bar because stdout was a file, esptool blocked on the full pipe mid-write. PYTHONUTF8=1 PYTHONIOENCODING=utf-8 fixed it; the third run flashed and verified all four regions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first live recovery (2026-09-08 20:44) logged two core refusals per run: digitalWrite() on a pin still routed to the I2C peripheral. Inputs first, latch high, then SCL to open-drain output. Both envs rebuilt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On the bench the LIS3MDL, whose address-select input is floating, came back at the other address every 10 s and was dropped again within a second: a re-init and a burst of failed reads every 11 s. A device that drops again within 30 s of returning now waits twice as long before the next re-probe, 10 s doubling to 320 s; a return that holds for 30 s resets it.

Correction: the previous commit (4a9c9aa) said both envs were rebuilt. That build had failed, on a tool-esptoolpy package that PlatformIO reinstalled as a stub and could not complete under MSYS. Rebuilt from PowerShell: um_feathers3 RAM 22.7% Flash 27.6%, feather_s3 RAM 31.6% Flash 64.7%, and the package is whole again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fourth review pass: the back-off ladder could only be cleared by the timing of the next drop, so a device with glitches spaced just under 30 s apart climbed to the 320 s interval with 29 s of perfect operation between them. i2cNoteOk() now clears the ladder once the device has answered for I2C_FLAP_MS since it came back. Both envs rebuilt from PowerShell, sizes unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Strycher

Strycher commented Sep 9, 2026

Copy link
Copy Markdown
Owner Author

Agent: EmberDelta (session f0e32f9f)

@Strycher
Strycher merged commit e36a79f into main Sep 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Touch controller stops ACKing after ~100 s; firmware polls it at 30 Hz forever with no backoff, and boot never logs the reset reason

1 participant