DEV-1023: one timestamp-unwrap rule — invalid zeros, reorders, duplicates and wraps

[marknolan]

Supersedes #301, whose reorder and duplicate detection is adopted here with credit on
the commit. Depends on log-and-stream-common#136,
which specifies the rule and ships the conformance vectors this runs.

The fault

Firmware stamps a packet when its sample tick starts it and does not publish a packet it
never stamped, so 0x000000 in the timestamp field means the record is invalid — not that
the counter reached its origin. LogAndStream v1.00.x–v1.01.003 could emit one under SD
write back-pressure.

unwrapTimeStamp() read any backward step as a roll-over, and a zero is always below its
predecessor, so each such record added 2²⁴ ticks = 512.002 s to every later sample,
permanently. A 9 minute 30 second trial holding four of them imported as 43 minutes 38.

Across six devices in one session, every reported duration was an integer multiple of
512 s above the true one, against a single clean device as the control — the evidence
table on DEV-1030 that made the mechanism unambiguous.

The rule

Each sample is classified by its modular forward distance from the previous one, with
forward motion as the default:

  if forward == 0:            hold                          // duplicate
  elif backwards <= W:        lastUnwrapped - backwards     // reordered packet
  elif modulo == 2^24 and raw == 0 and lastRaw < modulo - 32768:
                              reject, state untouched       // never stamped
  else:                       lastUnwrapped + forward       // forward; a wrap iff raw < lastRaw

  W = 0 when the rate is unknown; else min(8 * 32768 / rateHz, modulo / 8)

Three things in it are load-bearing, and all three were verified by mutation rather than
assumed:

	why
Modular distance, not unwrapped values	Asking "is the candidate below the last one?" misses a packet arriving late from before a wrap boundary — its candidate sits nearly a modulo ahead, so it is accepted and the next real sample reads as a second wrap. 16777206, 5, 16777206, 70 costs 512 s twice
Forward motion is the default	A roll-over preceded by a long dropout is still a roll-over, however much was lost
An unknown rate gives zero, never infinity	32768.0 / 0.0 is POSITIVE_INFINITY in Java; an infinite window makes every backward step a reorder and loses every wrap — worse than the naive rule

The window is eight sample periods, not a fraction of the modulo: a reorder swaps adjacent
packets, a dropout spanning the wrap point is most of a modulo, and modulo / 8 confuses
them — on the 2-byte counter every dropout between 1.75 s and 2.0 s would read as a reorder
and lose the wrap, which is an ordinary Bluetooth gap.

It is derived from getRtcClockFreq() (32768 Hz), not getSamplingClockFreq(), which
is 312500 or 255765.625 Hz on a TCXO board and would widen the window about nine and a half
times.

Scope and behaviour notes

• Shimmer2/2R get a window of zero. Their tick domain is unsettled — this class divides
  their 16-bit counter by 32768 while the C# API divides by 1024 — so a rate-derived window
  would be wrong in one of the two. They keep today's behaviour; the invalid-zero rule
  never applied to a 2-byte counter anyway. Follow-up ticket to settle the clock.
• The window is derived per sample, so a rate written mid-session is picked up by the
  next one and there is no cached window to reset.
• The first sample of a stream is passed through rather than measured against the reset
  state, which would otherwise let a first raw value near the top of the range read as a
  packet reordered across a boundary and place a whole recording one modulo early.
• Reordered output is deliberately not monotonic — a late packet belongs at the time it
  was sampled. Checked against the SD-import consumers (ParserLoggedDataToCSV,
  …ToDatabase, …ToObjectCluster, DockManager, UtilDock): none sorts, diffs or throws
  on a backward step, and none derives duration from the timestamp span. In a file it
  should not fire at all — the firmware writes records sequentially.
• Two limits, stated rather than hidden: a packet more than eight sample periods late is
  indistinguishable from a roll-over, and a gap longer than a whole modulo cannot be
  recovered from the counter at all.
• isLastTimestampRejected() lets the SD importer drop the record (the Advance-API PR);
  a live stream carries the previous timestamp for one packet. calculateTrialPacketLoss()
  is skipped for a rejected sample, and resetCalibratedTimeStamp() clears the flag.

Tests

22 tests, 0 failures.

API_00010_TimestampUnwrapVectorsTest runs the 26 shared conformance vectors from
log-and-stream-common, byte-identical at revision 1 / commit 267f693. The C# API,
pyshimmer and the web SDK run the same cases — if the four disagree, one is wrong, which is
the thing nobody could see last time. It asserts the file's revision and its full id list
too, so a vector dropped upstream fails here instead of quietly reducing coverage, plus the
window derivations and the values JSON cannot spell (infinity, NaN, negative).

API_00009_TimestampUnwrapTest keeps covering the rule in a form meant to be read, and
gains cases for the three properties above.

Four mutations were checked, each caught by two or three tests:

mutation	caught by
compare unwrapped values instead of modular distances	2
size the window as modulo / 8	3
let an unknown rate become an infinite window	3
drop the first-sample sentinel	2

Running these locally needs the Gradle 8.14.3 wrapper and Guava 33.4.0-jre — master's
Gradle 6.1 cannot resolve Guava's dual-capability metadata (DEV-1013, DEV-1020). Both were
applied temporarily and reverted; the only build change here is the test-only Gson
dependency, which is a plain POM and resolves on 6.1.

Landing

Shimmer-Advance-API #472 drops the rejected rows on SD import and depends on this; its CI
is red until this merges, because it builds against the driver's default branch.

🤖 Generated with Claude Code

[marknolan]

Follow-up commit: an adversarial review of this branch found that "no sample
yet" was not distinguishable from a real sample.

This API keeps an unwrapped value and a cycle count rather than the previous raw
value, so the reset state had to be encoded somehow, and (0, 0) was the
encoding. That state is also reachable: a reordered packet landing exactly on the
counter's origin leaves both at zero in the middle of a stream. The next packet
is then read as a first sample and passed through, so one arriving from just
before the origin is placed a whole modulo late rather than sixteen ticks behind.

Found by running the two formulations of the rule against each other rather than
by reading them:

	[520, 0, 16777200]
keeps lastRaw — web SDK, pyshimmer	[520, 0, -16]
(0, 0) as the reset state — here, before	[520, 0, 16777200]

512 seconds apart on the same input, between APIs that are meant to be
identical, and no vector covered it.

The unwrap is now told outright. The existing overload stays, still inferring it
from (0, 0), for callers written before the distinction existed — with a test
stating what it does with this sequence, so the compatibility boundary is
deliberate rather than discovered.

The sequence is now the shared conformance vector
reorder-onto-origin-then-earlier-packet-24bit
(ShimmerResearch/log-and-stream-common#136), so all five host APIs are held to
the same answer. It is the first vector whose final cycle is negative — a real
state here, since the raw value is derived back out of it.

Probability in the field is negligible: it needs a reorder to land exactly on the
origin, within one window of it. The reason to fix it rather than note it is that
the class of problem is the one this whole workstream exists to stop — two
families of implementation that cannot represent the same state, with nothing
testing the difference.

[MAzalya]

ok tested, looks good