Repository navigation
feat(#289): I2C dropout back-off and re-probe for every bus device - #294
Merged
Merged
Conversation
A device that stops answering cost a driver timeout per loop pass and an error line per read: on 2026-09-08 the touch controller and then the IMU breakout went silent for hours, the loop fell from 44 Hz to 12-27 Hz and the log filled with a quarter of a million identical errors. i2c_health: per-device consecutive-failure count; after 5 the device is dropped (its reads stop, logged once), re-probed every 10 s (probe, then a 9-clock bus recovery and a second probe), and re-initialised through its normal init when it answers. Only devices present at boot are re-probed. The failure signal per library, checked in the pinned sources: SHT4x getEvent() is false on a NACK; BME68x leaves a negative sensor.status, sampled on the 3 s BSEC cadence; MAX17048 isDeviceReady() reads the version register; LSM6DS and LIS3MDL getEvent() return true unconditionally and FT6206 touched() has no error path, so those are address-probed (per read pass for the IMU, every 1 s for touch). The boot log now states the reset reason, so a crash boot is attributable from the next boot's log. Builds: um_feathers3 RAM 22.7% Flash 27.6%; feather_s3 RAM 31.6% Flash 64.7%. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Any negative sensor.status was being counted as a bus failure. Null-pointer, self-test and invalid-length codes are not bus faults and a re-probe would not clear them, so a chip with an internal fault would have been dropped and re-initialised every 10 s for nothing (Gemini finding on #289). Both envs rebuilt, sizes unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The live bench disproved the first design while it was under review. The LIS3MDL on the 4517 breakout has been NACKing nearly every data byte for hours, yet the boot bus scan lists 0x1C on every boot: the chip ACKs its address and fails the transaction after it. An address-only probe would have called it healthy all night. At the 17:38 boot its WHO_AM_I read succeeded, every configuration write after it NACKed, and begin_I2C() still returned true. A reset did not clear the fault; the LSM6DSOX on the same board and cable reads fine, so it is the chip, not the wiring. - i2cReadReg(): write register, repeated start, read one byte, on Wire directly (the drivers' own transaction shape). i2cCheck() compares the chip-ID register to the expected value: LSM6DSOX 0x0F=0x6C, LIS3MDL 0x0F=0x3D, FT6336U 0xA8=0x11, BME688 0xD0=0x61. SHT41 and MAX17048 have no ID register and keep the address probe for the re-probe only. - IMU check moves off the read path onto a 200 ms timer in serviceIMU(). - Two-address chips record the alternate; the re-probe tries it after the bus recovery and swaps addr if the chip answers there. - Drop on 5 consecutive failures OR 8 of the last 16 checks, so a chip that answers one read in ten cannot keep resetting the count (review pass 2, finding 2). - initIMU() clears both flags first and reads one configuration register back on each chip; a chip whose configuration writes were NACKed is refused instead of declared available while powered down (review pass 2, finding 3; the 17:38 boot did exactly that). Builds: um_feathers3 RAM 22.7% Flash 27.6%; feather_s3 RAM 31.6% Flash 64.7%. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the operation mode Third review pass. The re-probe cleared a device's failure window on a good WHO_AM_I read before its init had run, so a chip that passes the probe but fails its configuration forgot it was flaky; now only a unit's init clears the window, through i2cNoteReady(), when it succeeds. The LIS3MDL read-back moved from the data-rate bits to the operation mode, the setting whose failure yields zero data and whose register default and failed-read value are both power-down. Both envs rebuilt, sizes unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Chain on the bench, from the owner on 2026-09-08: Feather -> BME688 -> SHT41 -> LSM6DSOX/LIS3MDL (one 4517 board) -> TSL2591 (unused). Moving the IMU board up the chain did not change the failure rate (one failed IMU-board transaction per loop pass, 22.4/s, steady for 75 minutes either side), and the failing read moved from the LIS3MDL to the other chip on the same board with no reboot. The scan labelled 0x29 as a VEML7700; the device there is the TSL2591. Both envs rebuilt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
From the product page and the pinouts guide, read 2026-09-08: one shared regulator, shared FET level shifters with 10k pull-ups, ADM/ADAG address jumpers (open = 0x1C / 0x6A), INT pins unwired. Shared parts would take both chips down together; the bench fault takes one at a time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
pio-flash confirm hung twice at what looked like Connecting... for ten minutes. PlatformIO's echo thread died with UnicodeEncodeError (cp1252) on esptool's Unicode progress bar because stdout was a file, esptool blocked on the full pipe mid-write. PYTHONUTF8=1 PYTHONIOENCODING=utf-8 fixed it; the third run flashed and verified all four regions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first live recovery (2026-09-08 20:44) logged two core refusals per run: digitalWrite() on a pin still routed to the I2C peripheral. Inputs first, latch high, then SCL to open-drain output. Both envs rebuilt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
On the bench the LIS3MDL, whose address-select input is floating, came back at the other address every 10 s and was dropped again within a second: a re-init and a burst of failed reads every 11 s. A device that drops again within 30 s of returning now waits twice as long before the next re-probe, 10 s doubling to 320 s; a return that holds for 30 s resets it. Correction: the previous commit (4a9c9aa) said both envs were rebuilt. That build had failed, on a tool-esptoolpy package that PlatformIO reinstalled as a stub and could not complete under MSYS. Rebuilt from PowerShell: um_feathers3 RAM 22.7% Flash 27.6%, feather_s3 RAM 31.6% Flash 64.7%, and the package is whole again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fourth review pass: the back-off ladder could only be cleared by the timing of the next drop, so a device with glitches spaced just under 30 s apart climbed to the 320 s interval with 29 s of perfect operation between them. i2cNoteOk() now clears the ladder once the device has answered for I2C_FLAP_MS since it came back. Both envs rebuilt from PowerShell, sizes unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
|
Agent: EmberDelta (session f0e32f9f) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #289.
What changed and why
On 2026-09-08 the touch controller, then the IMU breakout, went silent on the I2C bus for hours. Every reader retried its device on every loop pass, each failure costing two error lines to serial and SD, so the loop fell from 44 Hz to 12-25 Hz and the logs filled with a million identical
ESP_ERR_INVALID_STATE(NACK) lines. Nothing recovered without a reboot, and the boot log could not say why the last run ended.New unit
src/i2c_health.{h,cpp}, and aserviceX()in each I2C unit called fromloop():Wiredirectly: LSM6DSOX0x0F=0x6C, LIS3MDL0x0F=0x3D, FT6336U0xA8=0x11, BME6880xD0=0x61. An address probe was the first design and the bench disproved it: the LIS3MDL ACKed its address on every scan while failing every data byte. SHT41 and MAX17048 have no ID register; their own read/isDeviceReady()results feed the same counters.availableflag, which gates its reads inloop(), and logs once.Wire.end()/begin()), check again at the recorded address and at the chip's alternate address (0x6A/0x6B, 0x1C/0x1E, 0x77/0x76). When it answers, the unit's normal init runs.initIMU()clears both flags first and reads a configuration register back on each chip; a chip that ACKs WHO_AM_I but NACKs its configuration writes (the 17:38 boot did exactly that and the library said OK) is refused.Docs: bench STEMMA chain order and the 4517 board facts in
HARDWARE.md; the redirected-output flash hang inDIAGNOSTICS.md; the scan now names 0x29 as the TSL2591.What the hardware said
Flashed to the UM bench board at commit 8c96702 and run for 100+ minutes:
answers at 0x1E now, not 0x1C), some at the same one, a re-drop within a second of each re-init. The8 of 16rule fired on the intermittent phases. Hardware fix: bridge theADMjumper (owner).digitalWrite()-before-pinMode()ordering; fixed in this branch (4a9c9aa).Builds:
um_feathers3RAM 22.7% Flash 27.6%;feather_s3RAM 31.6% Flash 64.7%.Gemini gate (four passes, $0.10)
Fixed: BME688 status narrowed to
BME68X_E_COM_FAIL; the 8-of-16 window (an intermittent chip kept resetting a consecutive counter); init read-back for a chip that NACKs its configuration; the failure window cleared only by a successful init, not by the re-probe; LIS3MDL read-back on the operation mode rather than the data-rate bits; back-off decays with good behaviour.Rejected with evidence: "begin() leaks the old I2C device" (all five libraries delete before new); "Wire re-entrancy between endTransmission(false) and requestFrom" (straight-line code, one task); "a long SD write makes the next probe time out" (the Wire timeout is per transaction); "bus recovery aborts an OLED transaction" and "web task reads flags mid-init" (single loop task, synchronous
WebServer); "the LIS3MDL ignores the 155 Hz data rate in medium mode" (the library sets ultra-high mode itself; the read-back now checks the operation mode anyway).Accepted as designed: a noisy SHT41 can be dropped for 10 s after 8 bad reads in a third of a second; re-init of one IMU chip re-inits both (one breakout); no hot-plug of a device absent at boot (never existed).
Verification still owed
Owner flashes the branch tip and confirms on hardware. The board currently runs 8c96702; the three commits after it (recovery pin order, back-off, back-off decay) are built but not yet flashed.
🤖 Generated with Claude Code