Skip to content

Add graceful I2C error handling to prevent printer shutdowns#38

Closed
thetic wants to merge 2 commits into
mainfrom
graceful-i2c-error-handling
Closed

Add graceful I2C error handling to prevent printer shutdowns#38
thetic wants to merge 2 commits into
mainfrom
graceful-i2c-error-handling

Conversation

@thetic

@thetic thetic commented Feb 19, 2026

Copy link
Copy Markdown
Owner

Summary

Add graceful I2C error handling to prevent printer shutdowns from transient I2C communication failures.

Context

Klipper PR #7013 (merged Feb 7, 2026) changed I2C error handling from MCU-side to host-side. I2C errors now raise mcu.error exceptions in Python that trigger printer shutdowns if not caught.

This PR adds try-except blocks around I2C operations in the measurement loop to catch these exceptions, log warnings, and continue operating with previous sensor values rather than shutting down the printer.

Changes

  • Wrap I2C operations in try-except to catch communication failures
  • Log warnings and continue measurement cycle on errors

Compatibility

Klipper (current): This change is effective immediately for users on Klipper v0.13.0+ with PR #7013's host-side error handling.

Kalico (current): Kalico does not yet have Klipper's PR #7013 changes. On Kalico, I2C errors still trigger MCU-side shutdowns that occur below the Python layer, so this error handling won't prevent those shutdowns. However, the changes are forward-compatible and will become effective when Kalico syncs these upstream changes.

With Klipper PR #7013, I2C errors now trigger host-side shutdowns instead
of MCU-side shutdowns. This change catches mcu.error exceptions during
measurement cycles, allowing the sensor to log warnings and continue
operating with previous values rather than shutting down the printer on
transient I2C communication failures.
@thetic

thetic commented Feb 19, 2026

Copy link
Copy Markdown
Owner Author

I'm gonna leave this on a branch while I look into Kalico's plans to pull in upstream changes.

@ndanyluk

Copy link
Copy Markdown

Thank you!!

I'm gonna test this out tonight on my Stealthmax by simply unplugging one of the sensor stacks during operation to see what happens.

So glad we might finally at long last have sane error handling for non-critical I2C sensors :)

@thetic

thetic commented Feb 19, 2026

Copy link
Copy Markdown
Owner Author

let me know how it goes. i haven't done any testing yet.

Comment thread src/klipper_sgp40/__init__.py Outdated
@ndanyluk

ndanyluk commented Feb 19, 2026

Copy link
Copy Markdown

Attempted testing, but klippy still crashes when I yank the wire going to my sensor stacks. attaching shortened klippy.log here for you, but the relevant commands when we crash are at line 924:

Sent 98 24297.787794 24297.787632 9: seq: 1a, i2c_transfer oid=4 write=b'' read_len=3
...
Receive: 98 24297.788159 24297.787794 12: seq: 1b, i2c_response oid=4 i2c_bus_status=START_READ_NACK response=b'\xc0>\x02'
...
Unhandled exception during run
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/klippy.py", line 177, in run
    self.reactor.run()
  File "/home/nick/klipper/klippy/reactor.py", line 329, in run
    g_next.switch()
  File "/home/nick/klipper/klippy/reactor.py", line 374, in _dispatch_loop
    timeout = self._check_timers(eventtime, busy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/reactor.py", line 177, in _check_timers
    t.waketime = waketime = t.callback(eventtime)
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 230, in _handle_step
    # Calculate VOC index
               ^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 291, in _read
    params = self.i2c.i2c_read([], reply_len)
                         ^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
BME280: Error reading data
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/extras/bme280.py", line 415, in _sample_bme280
    data = self.read_register('PRESSURE_MSB', 8)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bme280.py", line 771, in read_register
    params = self.i2c.i2c_read(regs, read_len)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 271, in i2c_read
    return self.i2c_transfer(write, read_len, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response
BME280: Error reading data
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/extras/bme280.py", line 415, in _sample_bme280
    data = self.read_register('PRESSURE_MSB', 8)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bme280.py", line 771, in read_register
    params = self.i2c.i2c_read(regs, read_len)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 271, in i2c_read
    return self.i2c_transfer(write, read_len, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response
Repeat unhandled exception during run
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/klippy.py", line 185, in run
    self.reactor.run()
  File "/home/nick/klipper/klippy/reactor.py", line 329, in run
    g_next.switch()
  File "/home/nick/klipper/klippy/reactor.py", line 374, in _dispatch_loop
    timeout = self._check_timers(eventtime, busy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/reactor.py", line 177, in _check_timers
    t.waketime = waketime = t.callback(eventtime)
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 230, in _handle_step
    # Calculate VOC index
               ^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 290, in _read
  File "/home/nick/klipper/klippy/extras/bus.py", line 271, in i2c_read
    return self.i2c_transfer(write, read_len, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response

Now this was an extreme example (I yanked the entire module out effectively, so not quite a read failure the way it was expected

@ndanyluk

ndanyluk commented Feb 19, 2026

Copy link
Copy Markdown

Commented out my BME280 to clean up the error messages (since you're not touching the BME280 code I think that still fails ungracefully):

Unhandled exception during run
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/klippy.py", line 177, in run
    self.reactor.run()
  File "/home/nick/klipper/klippy/reactor.py", line 329, in run
    g_next.switch()
  File "/home/nick/klipper/klippy/reactor.py", line 374, in _dispatch_loop
    timeout = self._check_timers(eventtime, busy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/reactor.py", line 177, in _check_timers
    t.waketime = waketime = t.callback(eventtime)
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 231, in _handle_step
    response = self._read()
               ^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 292, in _read
    response = bytearray(params["response"])
                         ~~~~~~^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
Repeat unhandled exception during run
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/klippy.py", line 185, in run
    self.reactor.run()
  File "/home/nick/klipper/klippy/reactor.py", line 329, in run
    g_next.switch()
  File "/home/nick/klipper/klippy/reactor.py", line 374, in _dispatch_loop
    timeout = self._check_timers(eventtime, busy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/reactor.py", line 177, in _check_timers
    t.waketime = waketime = t.callback(eventtime)
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 231, in _handle_step
    response = self._read()
               ^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 291, in _read
    params = self.i2c.i2c_read([], reply_len)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 271, in i2c_read
    return self.i2c_transfer(write, read_len, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response
webhooks client 548164851984: Disconnected
Starting Klippy...

From what I can tell, it looks like the code after the I2C exception is unable to handle the i2c_response having no data (I think). But it's tough to say since Python's not my strongest language

Log here: klippy_i2c_fail_sgp40_only.log

Comment thread src/klipper_sgp40/__init__.py
Comment thread src/klipper_sgp40/__init__.py Outdated
Comment thread src/klipper_sgp40/__init__.py Outdated
@ndanyluk

Copy link
Copy Markdown

Unfortunately, even with the changes I made above, we still crash out:
From klippy_fail_NoneTypeFixed.log, line 1025:

SGP40 SGP_IN: I2C read returned None
SGP40 SGP_IN: Measurement read failed: SGP40 I2C read failed
Unhandled exception during run
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/klippy.py", line 177, in run
    self.reactor.run()
  File "/home/nick/klipper/klippy/reactor.py", line 329, in run
    g_next.switch()
  File "/home/nick/klipper/klippy/reactor.py", line 374, in _dispatch_loop
    timeout = self._check_timers(eventtime, busy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/reactor.py", line 177, in _check_timers
    t.waketime = waketime = t.callback(eventtime)
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 271, in _handle_step
    self.i2c.i2c_write(cmd)
  File "/home/nick/klipper/klippy/extras/bus.py", line 262, in i2c_write
    self.i2c_transfer(data, minclock=minclock, reqclock=reqclock,
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response
BME280: Error reading data
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/extras/bme280.py", line 415, in _sample_bme280
    data = self.read_register('PRESSURE_MSB', 8)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bme280.py", line 771, in read_register
    params = self.i2c.i2c_read(regs, read_len)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 271, in i2c_read
    return self.i2c_transfer(write, read_len, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response
Repeat unhandled exception during run
Traceback (most recent call last):
  File "/home/nick/klipper/klippy/mcu.py", line 97, in _do_send
    return xh.get_response(cmds, self._cmd_queue, minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/serialhdl.py", line 339, in get_response
    raise error("Unable to obtain '%s' response" % (self.name,))
serialhdl.error: Unable to obtain 'i2c_response' response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/nick/klipper/klippy/klippy.py", line 185, in run
    self.reactor.run()
  File "/home/nick/klipper/klippy/reactor.py", line 329, in run
    g_next.switch()
  File "/home/nick/klipper/klippy/reactor.py", line 374, in _dispatch_loop
    timeout = self._check_timers(eventtime, busy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/reactor.py", line 177, in _check_timers
    t.waketime = waketime = t.callback(eventtime)
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 231, in _handle_step
    response = self._read()
               ^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/sgp40/__init__.py", line 291, in _read
    params = self.i2c.i2c_read([], reply_len)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 271, in i2c_read
    return self.i2c_transfer(write, read_len, retry=retry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/extras/bus.py", line 283, in i2c_transfer
    param = cmd.send([self.oid, write, read_len],
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 102, in send
    return self._do_send([self._cmd.encode(data)], minclock, reqclock,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nick/klipper/klippy/mcu.py", line 100, in _do_send
    raise self._error(str(e))
gcode.CommandError: Unable to obtain 'i2c_response' response
webhooks client 547251101072: Disconnected

- Catch generic Exception instead of mcu.error (which wasn't being raised)
- Use logging.exception() to get full traceback
- Remove unused mcu import

Fixes the actual exception type being raised (gcode.CommandError) which
was causing crashes during testing.
@ndanyluk

ndanyluk commented Feb 20, 2026

Copy link
Copy Markdown

Okay @thetic, I tried the new code. Unfortunately still not working, but I got some insight from the Klipper dev who wrote PR #7013. It seems the problem for my test is here in bus.py:

    def i2c_transfer(self, write, read_len=0, minclock=0, reqclock=0,
                     retry=True):
        if self.async_write_only:
            raise mcu.error("I2C dev is write only")
        cmd = self.i2c_transfer_cmd
        if self.mcu.is_fileoutput():
            cmd.send([self.oid, write, read_len],
                     minclock=minclock, reqclock=reqclock)
            return
        param = cmd.send([self.oid, write, read_len],
                         minclock=minclock, reqclock=reqclock, retry=retry)
        status = param["i2c_bus_status"]
        if status == "SUCCESS":
            return param
        err_msg = "MCU '%s' I2C request to addr %i reports error %s" % (
            self.mcu.get_name(), self.i2c_address, status)
        self.mcu.get_printer().invoke_shutdown(err_msg)

It seems like unless the MCU reports that the I2C command was a SUCCESS (not I2C_READ_NACK, I2C_NACK, etc), it automatically shuts down the printer. So by force-disconnecting the device I'm automatically killing things because I guarantee a NACK.

Going to let my printer run overnight and see what happens - I haven't seen what happens with a simple read failure.

@astrolemonade

Copy link
Copy Markdown

Hey! I volunteer myself to test the new thing when it's test ready. Thank you! It's frustrating to me, I have a nevermore stealthmax pcb v2 which is basically a glorified fan controller now. Thanks to this project I'll be able to bring it back to life.

@thetic

thetic commented Apr 5, 2026

Copy link
Copy Markdown
Owner Author

@ndanyluk any news/progress? It sounded like you were following some supporting changes upstream, but I've been preoccupied with some personal issues these past few months. Thanks for all your help so far.

@thetic

thetic commented May 2, 2026

Copy link
Copy Markdown
Owner Author

Closing in favor of a new approach at #38

@thetic thetic closed this May 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants