[BUG] A refused block read drops its members from the snapshot with no per-register fallback

[sphings79]

Affected area

Battery communication or availability

Omnibattery version

release/v1.5.0 (89d3a96) plus #500, built locally as 1.5.0-blockread.1. The code path below is unchanged since block reads were introduced in #361.

Home Assistant version

2026.9.3 (HA OS 18.3)

Installation details

Marstek Venus D, EMS v150, seven packs, Modbus TCP. Measured both through a Modbus proxy and against the battery directly; the behaviour is the same, the proxy only changes how fast a refusal comes back.

What happened?

When a block read fails, its members are dropped from the snapshot and no attempt is made to read them individually. In drivers/marstek.py:

pending.difference_update(member_keys)
regs = await self._client.async_read_block(...)
if regs is None:
    continue                      # <- members are gone from `pending` already

The keys leave pending before the request is sent, so the per-register loop below never sees them. One refused span therefore takes every register in it out of that cycle, and the only trace is a generic Modbus read error at register N, which does not say which entities just went without a value.

Today this is mostly invisible, because the spans in the table are known-good on the models that have them. It stops being invisible the moment a span covers an address a given device does not implement — and it is what makes an otherwise obvious optimisation unusable, which is how I ran into it.

I measured bridging small gaps on the Venus D, since the tables only ever group registers that are already adjacent:

	fast schedule	full pass	values returned
no gap	11 requests	41	57
gap of 2	11 requests	38	57
gap of 10	9 requests	35	53

With a gap of 10, four of the five new spans answer fine:

30001+6   high      battery_power, ac_power                 answered in 360 ms
30200+5   very_low  ems/vms/bms_version                     answered in 163 ms
30300+4   low       wifi_status, cloud_status, signal        answered in 300 ms
34003+7   low       battery_cycle_count, balancing_mode     answered in 299 ms
42010+12  high      force_mode, charge_to_soc,
                    set_charge_power, set_discharge_power   REFUSED after 503 ms

Some address between 42012 and 42019 is not implemented, so the whole span is refused — and that span carries the four control registers. They simply vanish from every fast cycle, silently, and the two requests the gap "saves" are exactly the ones whose answers are lost.

Steps to reproduce

1. Point the Marstek driver at a battery and block-read a span that covers an address the firmware does not implement (on a Venus D: 42010, 12 registers).
2. Request all of that span's keys in one read_telemetry call, the way a read group does.
3. The returned snapshot contains none of them, async_read_register is never called for them, and nothing in the log names the keys that were dropped.

Expected behavior

A block is a way to fetch registers that would otherwise be fetched one at a time, so a failed block should cost no more than the block read itself: fall back to reading its members individually. Failing that, log at warning which keys were dropped, so it is visible rather than silent.

One caveat from trying it: a blanket fallback is not free. regs is None also covers a connection outage, and then the fallback fires N doomed requests per cycle at the weakest Modbus stack in the supported range — which is the thing #361 set out to relieve. It probably needs to tell a refusal (the device answered, with an exception) from a timeout, and to remember a span that was refused rather than retrying it forever.

Relevant Home Assistant logs

Modbus read error at register 42010 (0xA41A) on attempt 1

That is the whole trace. Nothing says that force_mode, charge_to_soc, set_charge_power and set_discharge_power are missing from this cycle.

Additional context

Not a regression, and #500 does not introduce it: the derived table still only groups registers that are already adjacent, all of which would be read — and refused — individually anyway. I am filing it because the measurement above is worth recording either way, so nobody else spends a night finding out that gap bridging looks free and is not, and because the silent drop is worth fixing on its own terms.

Nothing of this is in #500 — that one changes which registers get grouped, not what happens when a group is refused. Happy to prepare a separate PR for it, once it is clear which of the two shapes above you would rather have.

Confirmations

• I am using the latest available Omnibattery release and restarted Home Assistant after updating.
• I searched the documentation, existing issues, and existing discussions.
• I can reproduce this unexpected behavior; this is not a request for installation or configuration help.
• I removed passwords, tokens, API keys, complete serial numbers, IP addresses, and other sensitive information.

[ffunes]

@sphings79, confirmed and fixed on release/v1.5.0 in e30502e, so no PR is needed. Your measurement and the caveat around it made the right shape clear.

The client now distinguishes a device ExceptionResponse from a timeout/connection failure. A refused block falls back to its member registers in the same cycle, and that span is remembered so it is not block-read again. Transport failures retain the existing no-fallback behavior. Both paths now name the keys that went without a value — warning for a refusal, debug for a transport failure — and the error line itself carries the span length and the Modbus exception code.

I have recorded the gap-bridging result, but have not adopted it: the derived table still groups adjacent registers only. With refusal now bounded, your numbers make it worth revisiting as a separate change.

One deliberate limitation: the refused-span memory is reset on integration reload, so firmware that changes its implemented register set is probed again.

One concrete question, if you can check it with your setup: through your Modbus proxy, does a v3/A/D rejection reach the client as a Modbus exception response? A proxy that parses the frame itself could answer differently, and this fallback depends on that distinction.

[sphings79]

Measured, and the answer is yes — but only because of something sitting between the client and the battery, which makes it a qualified yes.

Through the proxy: a proper exception response

Raw socket against the proxy, one request each:

32104 x2    OK          fc=0x03                     3 ms
30001 x6    OK          fc=0x03                    10 ms
42010 x12   EXCEPTION   fc=0x83 code=2             54 ms
42010 x1    OK          fc=0x03                    76 ms
42012 x8    EXCEPTION   fc=0x83 code=2            132 ms

pymodbus 3.15.0 on the same path:

type                          = pymodbus.pdu.exceptionresponse.ExceptionResponse
isError()                     = True
isinstance(ExceptionResponse) = True
exception_code                = 2
function_code                 = 131

So isinstance(result, ExceptionResponse) fires, BLOCK_REFUSED is returned, and the fallback runs. Promptly, too — 54 ms, nowhere near a timeout.

The catch: the frame arrives repaired

The reply I captured was 00 03 ... 01 83 02. The length field reads 3. The Venus D does not send 3.

The firmware builds a nine byte exception frame whose MBAP length field reads 4 where the protocol requires 3. It is in Modbus_Dispatcher, the exception branch writing the length low byte:

0801eaf4  04 20        movs r0, #4           ; the length byte
0801eaf6  48 71        strb r0, [r1, #5]
0801eb06  09 22        movs r2, #9           ; nine bytes sent

Confirmed on the device, not only in the image. My proxy runs a local patch that reframes an exception reply by the protocol instead of by the header and re-emits it with a corrected length, which is why the capture above looks clean. That patch is not published anywhere yet.

Without it, pymodbus never produces an ExceptionResponse

I replayed both frame shapes against pymodbus 3.15.0 from a local socket server — no hardware, so this is reproducible anywhere:

length 3 (protocol-correct)  -> ExceptionResponse, code 2,           after    2 ms
length 4 (what the Venus D sends) -> ModbusIOException "No response received", after 6003 ms

The framer receives the frame and discards it:

>>>>> recv: 0x0 0x1 0x0 0x0 0x0 0x4 0x1 0x83 0x2 extra data:
>>>>> extra:  unexpected data: 0x0 0x1 0x0 0x0 0x0 0x4 0x1 0x83 0x2

It then retries and discards again, so one refused span costs two full client timeouts.

_read_raw catches ModbusIOException and returns None. A Venus D talking to Omnibattery directly therefore lands on the timeout path, BLOCK_REFUSED is never produced, and the fallback in e30502e never runs — on the device that motivated it. Everything you built is correct; its precondition just does not hold here unless a repairing proxy is in the way.

This is also why the caveat in my original report read the way it did. I had the proxy patch in place by then and did not realise it was carrying the distinction.

Your commit message already names the gap — the unit tests build the ExceptionResponse by hand and would pass either way. The socket-level test above is the one that separates the two, and it needs no device: a server that emits the nine bytes with the length field set to 4.

What follows is your call

Two shapes, and neither is obviously right:

1. Tolerate the frame in the client. An exception PDU is always exactly two bytes, so the length field is not needed for it. That means a custom framer or a pre-read, which is intrusive for one vendor's off-by-one, and it would have to drain any surplus so a padded frame from a future firmware cannot desync the next reply.
2. Leave the client strict and document it. Venus D users need something in the path that repairs the frame. That is honest, but today there is nothing to point them at.

I have no preference strong enough to argue for. If option 1 is interesting I can prepare it with the socket-level test; if you would rather keep the client strict, I can look at getting the proxy patch upstream so there is a link to give people. tiagocoutinho/modbus-proxy has merged nothing in about a year, the HA add-on repo is active, so it would likely have to go in at the add-on end.

Incidentally, the hole is exactly bounded

42010 ok (1)    42011 ok (0)    42012-42019 REFUSED
42020 ok (2005) 42021 ok (0)    42022-42024 REFUSED

Eight unimplemented registers between the two control pairs, each rejected with exception code 2.

Tested with pymodbus 3.15.0; the manifest allows >=3.8.0, and I have not checked where in that range the framer behaviour changes.