Unit 7: On-device verification with dispersion

[mairas]

Goal: Verify on a device that the stack meets its resource and latency criteria, with dispersion reported.

Requirements: All success criteria

Dependencies: Unit 6

Files:

• Extend: the Unit 1 measurement harness
• Create: results under docs/

Approach:

• Three conditions: control with no history plugin, sqhp plus QuestDB, and this provider. The harness already exists from Unit 1, and Units 3a, 3b and 5a have reported through it, so figures are comparable rather than improvised per unit.
• Configure all conditions to the same workload — same path set, same rate cap, self only — and report rows/s for each. Divergence beyond a stated tolerance invalidates the comparison rather than being absorbed by the criterion band. The earlier baseline recorded 111.7 against 84.4 rows/s under configurations declared identical and never explained it.
• Report dispersion. Per-window values or a standard deviation. The earlier baseline reported point estimates, and its own numbers show the noise floor is high enough to swallow a 0.15-point CPU delta — the sqhp condition measured lower disk writes than the no-history control.
• Keep steady state and transients separate. Steady-state RSS is the continuous cost; the roll's peak is a transient. Do not sum them. The query service is neither: it starts on the first history request and stays, at 92 MB idle rising to the high-water mark of the largest answer it has served, so it is a standing cost from the first query onwards. A playback session adds no process of its own — Unit 4c reads its windows through that same service — so what a session costs is what it moves that mark to.
• Test an aged tree, not a fresh one. The 16–37 ms spike figures came from a tree the size of a single roll — roughly one file per path, the cheapest possible case. At the chosen roll interval a 30-day single-path range touches far more files, and an all-paths query over the same range multiplies that by path count. Measure a long-range query against a tree holding a full retention window's file count.
• Measure end-to-end wall time from spawn to result, not in-engine query time.
• Note whether QuestDB's Parquet storage policy was active during the sqhp condition; if it was, the reference figures need re-measuring.
• This unit can fail. If the criteria are not met, that is a finding and a decision, not an adjustment of the bar.

Test scenarios:

• Integration: steady-state added RSS across plugin and writer is under 150 MB.
• Integration: total CPU across plugin, writer and amortised roll is under +2.3 points of one core at 70–130 rows/second.
• Integration: Signal K process disk writes stay within noise of the 12 KB/s control.
• Integration: a single-path range and a bucket aggregate return within sqhp's range against a fresh tree.
• Integration: a 30-day single-path range and a 7-day all-paths range against an aged tree return within a stated bound.
• Integration: the roll's transient peak and the query service's resident size — idle, after ordinary queries, and over a playback session of several minutes — are each recorded and each fit available memory with the full marine stack running.
• Integration: two playback clients and a dashboard together. The service answers one request at a time and refuses past eight queued, so this is queue depth, refusals and what the service holds — not a per-client process count.
• Integration: a v1 snapshot in both of its branches — of the present, which reads the sidecar and the hot store, and of an instant the tree has rolled past, which reads two date directories. Neither is measured anywhere yet.
• Edge case: a full day of continuous recording followed by a roll shows no unbounded growth in writer RSS.
• Edge case: a roll coinciding with a query, with a playback session running — the summed transient is measured, since nothing admission-controls the roll against the query service.

Verification:

• A results table across all three conditions with dispersion shown and rows/s stated for each.
• Each criterion marked met or not met against a measured number.

[mairas]

Updated the body for what Unit 4c settled about playback, so this unit does not go looking for something the implementation does not have.

A playback session has no process of its own. The plan written before Unit 4a assumed spawn-per-query, which made a session "a ~120 MB process every 60 s per connected client". Unit 4a replaced that with one query service held for the life of the plugin, and Unit 4c reads playback windows through it like every other query — asserted by test: three chunk reads, one spawn. So what a session costs is what it moves the service's resident size to, not a per-client process count.

That also moves the query service out of the transient column. It starts on the first history request and stays, so from that moment it is a standing cost — 92 MB idle, and the high-water mark of the largest answer it has served after that.

Three scenarios added: a playback session's effect on the service over several minutes, two playback clients plus a dashboard (queue depth and refusals, since the service answers one at a time and refuses past eight queued), and the v1 snapshot in both of its branches — neither of which is measured anywhere yet.

[mairas]

First on-device run, 26 August, on a HALPI2 that has been recording continuously since 23 August. Not the complete unit — no aged-tree long-range query, no playback session, no concurrent-client test — but enough to settle the criterion the plan called neither met nor missed.

Method

The sqhp and InfluxDB2 columns are not re-measured. They come from the 22 August matrix run's final/2000ms cell, which is the production QuestDB configuration at the production sampling rate, with 480 s settle, 420 s window, 30 s sampling and a half-window warmup check. That file (qdb-final.out) is the only one of the three matrix outputs that still exists on the device; the other two were deleted. A copy is now off the device.

The sphp column was measured the same way — 120 s settle, 420 s window, 30 s sampling, halves reported — with InfluxDB running untouched alongside, as it was during the earlier run.

Ingest rates: sphp 136.5 rows/s, QuestDB 147.7 values/s. 8% apart, which is the comparison's stated tolerance. sphp recorded one context; QuestDB ran with recordOthers: true.

Standing cost

	CPU (% of one core)	anon RSS	total RSS	swap	write I/O
sphp writer	0.11 (halves 0.11/0.11)	16 MB (16–17)	58 MB (58–59)	0	39.5 kB/s
QuestDB, tuned	2.88 (2.88/2.87)	317 MB (290–336)	379 MB	21 MB	0 IOPS
InfluxDB2	1.30	159 MB (146–174)	—	—	0 IOPS

Per unit of ingest the writer costs 24× less CPU and 20× less anonymous memory than tuned QuestDB.

The plugin's in-process cost is below this method's noise floor

A paired control — server restarted, plugin off, then restarted with it on:

control/off | skcpu=21.91 | skrss=521/445-603 | n=15 | dt=449
control/on  | skcpu=22.11 | skrss=504/422-585 | n=14 | dt=420

The memory delta is −17 MB: the server measured less resident with the plugin on. That is noise, not an effect — within-cell RSS swings 160 MB as Node's heap grows and collects, so 30 s sampling cannot resolve a plugin of this size. The CPU delta is +0.20 points against a 22% baseline, and this plan already records that the noise floor swallows 0.15 points.

What it does establish is an upper bound: the in-process cost is not one or two points of a core. Resolving it further needs many alternating cells and a paired confidence interval, not one pair.

Transients

The 09:00 UTC roll, measured two independent ways that agree to 0.6%:

• The roll's own VmHWM, reported in its summary line: 166 MB, 401,304 rows to one date, 574 sidecar rows, all 401,304 truncated afterwards.
• An external 0.5 s sampler reading /proc/<pid>/VmHWM: 167 MB.

It ran from 08:59:59Z to 09:00:01Z — two seconds — and the writer returned to 59 MB. The exit does return the memory, as the design requires. This sits inside the 146–163 MB range docs/layout-decision.md records for the shipped roll.

The query service, started by the first history request and kept: 145 MB RSS / 65 MB anon immediately after 13 queries, settling to 120 MB / 62 MB.

Query latency — the criterion, and it is missed

Same window (07:00–08:00 UTC), one path, 60 s resolution, all three returning 60 rows:

provider	median	range	n
InfluxDB2	11.0 ms	9.9–18.1	12
sqhp / QuestDB	14.2–29.8 ms	five readings, 21–22 Aug	5
sphp, warm	132.0 ms	120.2–152.4	12
sphp, cold	466.9 ms	engine start	1

sphp is 4.4–9.3× slower than sqhp and 12× slower than InfluxDB2.

This is not a regression. docs/query-layer.md predicted a single-path hour at 127–174 ms with the service warm. The measurement is 132 ms — the middle of the prediction. The architecture is doing exactly what it was designed to do; what it was designed to do is an order of magnitude slower than a resident database for this shape.

What this costs in total, which is the comparison that matters

Once any history has been requested, the query service is a standing cost — this plan says so itself. So:

	resident, steady state	CPU	1-hour query
sphp	179 MB (59 writer + 120 service)	0.11% + below noise	132 ms
InfluxDB2	159 MB anon (22 Aug)	1.30%	11 ms
QuestDB	379 MB	2.88%	14–30 ms

Against QuestDB the trade is real: less than half the memory, a twenty-fourth of the CPU per row, for 4–9× the query latency.

Against InfluxDB2 there is no memory win to trade. 179 MB against 159 MB, for 12× the query latency.

Two things found along the way

• The hot store never returns deleted pages. 259 MB file, 98.4% freelist, holding 4.2 MB of rows. Filed as Hot store never returns deleted pages: 259 MB file holding 4.2 MB of rows #179.
• InfluxDB2's current state cannot be reconciled with its 22 August figures. It used 145 CPU-seconds in 4.5 days (0.037% average) and measured 0.04% over a 420 s window today, against 1.30–1.42% in all four cells on 22 August — while still writing to its WAL. Its resident size is now 1 MB, almost entirely paged out. The figures above are the 22 August ones; nothing today reproduces them, and the reason is unknown. A publish decision resting on the InfluxDB2 comparison needs that explained first.

Still unmeasured

Long-range and all-paths queries against an aged tree; a playback session; two clients plus a dashboard; a roll coinciding with a query; disk writes by the Signal K process against a no-history control.

[mairas]

Follow-up to the run above: a proper no-history baseline, three alternating pairs rather than one, 120 s settle and 480 s window per cell, six restarts. Adds write_bytes, which the first control did not measure and which is the only evidence for the disk-writes criterion.

p1-off|skcpu=22.06|skkbps=11.8|skrss=513/412-591|n=17|dt=509
p1-on |skcpu=22.26|skkbps=11.7|skrss=526/414-577|n=17|dt=509
p2-off|skcpu=22.06|skkbps=11.7|skrss=523/438-577|n=17|dt=509
p2-on |skcpu=22.30|skkbps=11.7|skrss=519/418-597|n=17|dt=509
p3-off|skcpu=21.90|skkbps=11.9|skrss=509/396-596|n=17|dt=509
p3-on |skcpu=21.94|skkbps=14.1|skrss=526/441-588|n=17|dt=509

Disk writes: the criterion is met

The control measures 11.8 kB/s, which independently reproduces the 12 KB/s no-history figure this plan cites. Five of the six cells sit in 11.7–11.9 whether the plugin is on or off. The Signal K process's disk writes do not move when the plugin runs, which is the design's central claim measured directly for the first time.

One on-cell reads 14.1 kB/s. It is reported rather than averaged away: one excursion in six windows, and nothing in the design accounts for it.

In-process CPU: positive, small, and not pinned

Four paired deltas, counting the earlier standalone pair: +0.20, +0.24, +0.04, +0.20. Mean +0.17 points of one core.

All four agree on sign, so the cost is real rather than noise. But the spread (0.04–0.24) is as wide as the effect, and the off-cells themselves scatter across 21.90–22.06 — the same order. The defensible statement is "between roughly zero and 0.25 points", not a point estimate.

Worth recording as a method note: after two pairs the deltas were +0.20 and +0.24 and looked converged. The third pair returned +0.04. A two-pair result here would have been reported as a resolved number and would have been wrong.

In-process memory: unresolvable, and more pairs will not fix it

Deltas +13, −4, +17 MB, and −17 MB from the earlier pair. Within-cell RSS ranges span 160–200 MB as Node's heap grows and collects, so 30 s sampling cannot see a plugin holding single-digit megabytes. This needs a different instrument, not more windows.

Against the criteria

• Total CPU under +2.3 points: writer 0.11 + in-process ≤0.25 + roll amortised ≈0.06 = ≤0.42. Met, with margin, on the pessimistic reading of every term.
• Signal K disk writes within noise of the 12 KB/s control: met.
• Steady-state added RSS under 150 MB: 58 MB writer plus an in-process term too small to measure — met. Counting the query service's 120 MB, which this plan calls a standing cost from the first query onwards, gives 179 MB and misses. Which of those two readings applies is a decision, not a measurement.

Six restarts of the Signal K container cost roughly four minutes of gaps in InfluxDB's production collection.