DEV-1023: unwrap the sample counter by the shared rule, and run the shared vectors

[marknolan]

The Swift API is the fifth host implementation of the same wire format, and it
carries the same timestamp-unwrap defect as the other four — in its original,
unmitigated form. This brings it onto the shared rule and the shared conformance
vectors.

What is wrong today

TimeSensor.calibrateTimeStamp reads any backward step in the counter as a
roll-over:

if (LastReceivedTimeStamp > (timeStamp + (TimeStampPacketRawMaxValue * CurrentTimeStampCycle)))
{
    CurrentTimeStampCycle = CurrentTimeStampCycle + 1;
}

Right for a roll-over, wrong for the three other things that step the counter
backwards — and each of them then adds a whole modulo, 512 seconds, to every
later sample for the rest of the session:

	how far back	cost
a reordered packet	one sample period	+512 s
a duplicated packet	nothing at all	+512 s
a record the firmware never stamped	to 0x000000	+512 s

The third is the reported one. Firmware stamps a packet when the sample tick
starts it and does not publish a packet it never stamped, so an exact zero in
the counter field marks an invalid record rather than the counter reaching
its origin. LogAndStream v1.00.x–v1.01.003 could produce one under SD write
back-pressure; a 9 minute 30 second recording containing four of them was
reported as 43 minutes 38.

It is also stated the wrong way round — on a comparison of unwrapped values
rather than on the modular forward distance. A packet arriving late from just
before a boundary has an unwrapped value above its predecessor, so the
comparison accepts it and then reads the next real sample as a second roll-over.
[2²⁴ − 10, 5, 2²⁴ − 10, 70] lands at 33554502 — two modulos out, from one
out-of-order packet.

The rule

TimestampUnwrap.swift, mirroring TimestampUnwrap.cs and
TimestampUnwrap.java method for method:

forward   = (raw − lastRaw) mod modulo
backwards = modulo − forward
forward == 0                          → hold (a duplicate)
backwards ≤ W                         → placed where it was taken (reordered)
3-byte, raw == 0, lastRaw mid-range   → invalid; flagged, state untouched
otherwise                             → forward motion, a roll-over when the raw value fell
W = min(8 × 32768 / rateHz, modulo / 8)

Forward motion is the default. That is what keeps a roll-over preceded by a
long dropout classified as a roll-over: however much was lost, the counter still
wrapped.

The window is eight sample periods, not a fraction of the counter's range —
the same two quantities the old rule confused. A reorder swaps packets adjacent
in time; a dropout spanning the wrap point is most of a modulo.

An unknown rate disables the branch. 32768 / 0 is +Infinity in Swift, not
an error, and an infinite window reads every backward step as a reorder and
loses every roll-over — the original bug restored, with no symptom until a
recording comes out short. reorderWindowTicks answers 0 for infinity, NaN,
zero and negatives.

Wiring

Shimmer3Protocol.interpretDataPacketFormat sets the window. Both inquiry paths
(interpretInquiryResponseShimmer3 and …Shimmer3R) set CurrentSamplingRate
immediately before calling it, so the rate is always current there.

ObjectCluster gains timestampValid. A rejected record keeps its sensor values
and loses its time; both Time Stamp signals are still added, so a consumer
reading by signal name is unaffected, and one that needs a true time axis has a
flag to filter on.

The unwrap now runs whether or not calibrationEnabled is set. An unstamped
record has no time either way, so the flag has to mean something in both modes —
and a timeline that only advanced while someone was watching would jump by
however much was missed the moment calibration was switched on mid-stream.

Tests

TimestampUnwrapVectorsTest.swift carries 27 conformance vectors and 11 window
derivations, the same set the Java, C#, Python and TypeScript APIs run. They
are specified in the firmware repository beside the prose they encode, where a
reference implementation regenerates and re-checks them in CI. Those four APIs
load the JSON directly; a classic Xcode project would need four hand-edited
project.pbxproj entries to carry a resource nothing compiles, so this target
gets generated source instead:

python Test/host/crosscheck_timestamp_unwrap.py --emit swift

Regenerating produces a byte-identical file, so a stale copy is detectable.

Plus seven tests on TimeSensor itself: a swapped pair placed where it was taken,
the same swap costing a modulo with the window off (what wiring the rate buys,
stated as the difference it makes), an unstamped record rejected without
disturbing the next sample, a genuine roll-over onto zero still accepted, a
roll-over after heavy loss still a roll-over, a reorder onto the origin not
looking like a fresh stream, and timestampValid on the ObjectCluster.

Two things to be aware of

Not built or run locally. There is no Swift toolchain on the machine this was
written on, so this PR's CI run was the first execution of any of it. Before
pushing, the rule was compared statement by statement against the C#
implementation, whose own suite passes, and the generated vectors come from the
same script that produces the C# array.

CI has since run it: ** TEST SUCCEEDED **, with all 11 new cases passing
(testAllSharedVectorsAgree covers the vectors). So it is verified now — but
by the runner, not by anything that happened before the push, which is worth
knowing if you are weighing how much review the Swift-specific parts deserve.

The one-line wiring in interpretDataPacketFormat is not covered by a test.
Shimmer3Protocol.init takes a concrete BleByteRadio, which needs a live
CBPeripheral, so no unit test can construct the protocol — and no existing test
does. The window derivation and everything downstream of it are tested; that the
protocol passes the rate through is not.

Raised as DEV-1034: widening the three protocol initialisers to take
ByteCommunication makes a loopback stub possible, which is how the TypeScript
and C# APIs test their protocol layers. It needs two members added to that
protocol (deviceName and delegate, both of which BleByteRadio already
declares) — not the one-word change it first looks like.

Since the first push — one review finding, fixed here

Keeping (lastUnwrapped, cycle) rather than the previous raw value means the
reset state has to be encoded somehow, and (0, 0) was the encoding. It is also
reachable: a reordered packet landing exactly on the counter's origin leaves both
at zero mid stream, so the next packet is read as a first sample and one from
just before the origin is placed a whole modulo late.

	[520, 0, 2²⁴ − 16]
web SDK, pyshimmer (keep lastRaw)	[520, 0, -16]
this API, Java, C# (before)	[520, 0, 16777200]

512 seconds apart, and nothing in the vector set could see it. unwrap is now
told outright, on both overloads rather than by default: the C# and Java copies
keep a form that infers it from (0, 0) because they have callers older than the
distinction, and this file does not, so a default would only be a quiet way to
get the first sample of a stream wrong. The sequence is now
reorder-onto-origin-then-earlier-packet-24bit in the shared file, and it is the
first vector whose final cycle is negative — a real state here, since the raw
value is derived back out of it.

Related

• Spec + vectors: Specify the host timestamp-unwrap rule, and give it conformance vectors log-and-stream-common#136
• Java driver: DEV-1023: one timestamp-unwrap rule — invalid zeros, reorders, duplicates and wraps Shimmer-Java-Android-API#300
• C#: DEV-1023/DEV-1024: one timestamp-unwrap rule, and a Shimmer3 SD file that parses Shimmer-C-API#203
• TypeScript: DEV-1023: size the timestamp reorder window from the sampling rate, and run the shared vectors shimmer-web-sdk#148
• Python: Do not read a reordered or invalid timestamp as a counter overflow seemoo-lab/pyshimmer#44

Reorder and duplicate detection came out of @MAzalya's parallel fix for this
defect on the Java driver. The polarity and the window sizing here differ, but
the observation that a backward step is not always a roll-over is hers.

🤖 Generated with Claude Code