The query service started in Unit 4a is always on: the plugin starts it on the first history request and keeps it until the plugin stops. That was chosen deliberately — an engine costs ~345 ms to start and a warm query answers in 96–175 ms — and it has a standing cost that this issue is about bounding.
What it costs today
Measured on a HALPI2 through ./run bench query, which drives the shipped client. The service reports its own resident size with every answer.
Forty identical single-path hour queries against one service:
| after |
wall |
service RSS |
| 1 (cold) |
558 ms |
117 MB |
| 2 |
166 ms |
128 MB |
| 5 |
165 ms |
142 MB |
| 15 |
136 ms |
164 MB |
| 25 |
140 ms |
165 MB |
| 40 |
168 ms |
170 MB |
So it is not a leak: repeated work converges, at 117 → 164 MB over fifteen queries and +6 MB over the twenty-five after that. What it does do is settle at the high-water mark of the largest shape it has been asked for, and never come back down — DuckDB's allocator does not return it, which is the same property that makes the roll a process that exits. An all-paths ten-minute query (82,000 rows) takes the same service to 317 MB and leaves it there.
Idle, before any query, the service is 92 MB: 79 MB of node with the addon mapped, 6 MB for the instance, 8 MB for sqlite_scanner and the attach.
Against the plan's budget in #152 — 150 MB steady state across plugin and writer, where the writer is 87 MB — a service that has answered a few queries puts the total near 260 MB. That is well under QuestDB's ~366 MB standing cost, and it is over the number this project set itself.
What to decide
When to recycle. Three signals, and they are not exclusive:
- Idle timeout. History use is bursty: a Grafana or Skip panel fires a cluster of requests and then nothing for hours. An idle exit restores 0 MB standing for the hours nobody is looking, at the cost of one cold request when someone returns.
- RSS ceiling. Bounds the damage from one large answer. Needs a figure; 150 MB would recycle after roughly fifteen ordinary queries, 200 MB after the first big one.
- Query count. Simplest to reason about, and the least related to what actually costs memory.
What a recycle does to a request in flight. Nothing, if it happens between requests. The service must not be replaced underneath an answer that is being streamed.
Whether the replacement is started eagerly or lazily. Starting the next one before the old one exits overlaps two engines — 92 MB plus whatever the outgoing one holds. Starting it on the next request means that request pays ~350 ms.
Whether the operator sees it. A recycle is normal, and a recycle every few seconds is not. Whatever counts them should be visible in the plugin status rather than in a debug log, which is the same argument as #175.
What it does not need to solve
The cold path itself. ~220 ms of the ~345 ms is mapping @duckdb/node-api's native addon, and nothing in this design avoids that — the addon is why a query runs in its own process at all.
Verification
- A service that crosses the ceiling is replaced, and the request that crossed it still gets its answer.
- A recycle between two requests is invisible to both.
- The idle timer does not hold the plugin open at shutdown.
- Measured on a device: standing RSS after an hour of a dashboard polling, and after an hour of nothing.
Follows Unit 4a (#166) and signalk-parquet-history-provider#14.
The query service started in Unit 4a is always on: the plugin starts it on the first history request and keeps it until the plugin stops. That was chosen deliberately — an engine costs ~345 ms to start and a warm query answers in 96–175 ms — and it has a standing cost that this issue is about bounding.
What it costs today
Measured on a HALPI2 through
./run bench query, which drives the shipped client. The service reports its own resident size with every answer.Forty identical single-path hour queries against one service:
So it is not a leak: repeated work converges, at 117 → 164 MB over fifteen queries and +6 MB over the twenty-five after that. What it does do is settle at the high-water mark of the largest shape it has been asked for, and never come back down — DuckDB's allocator does not return it, which is the same property that makes the roll a process that exits. An all-paths ten-minute query (82,000 rows) takes the same service to 317 MB and leaves it there.
Idle, before any query, the service is 92 MB: 79 MB of node with the addon mapped, 6 MB for the instance, 8 MB for
sqlite_scannerand the attach.Against the plan's budget in #152 — 150 MB steady state across plugin and writer, where the writer is 87 MB — a service that has answered a few queries puts the total near 260 MB. That is well under QuestDB's ~366 MB standing cost, and it is over the number this project set itself.
What to decide
When to recycle. Three signals, and they are not exclusive:
What a recycle does to a request in flight. Nothing, if it happens between requests. The service must not be replaced underneath an answer that is being streamed.
Whether the replacement is started eagerly or lazily. Starting the next one before the old one exits overlaps two engines — 92 MB plus whatever the outgoing one holds. Starting it on the next request means that request pays ~350 ms.
Whether the operator sees it. A recycle is normal, and a recycle every few seconds is not. Whatever counts them should be visible in the plugin status rather than in a debug log, which is the same argument as #175.
What it does not need to solve
The cold path itself. ~220 ms of the ~345 ms is mapping
@duckdb/node-api's native addon, and nothing in this design avoids that — the addon is why a query runs in its own process at all.Verification
Follows Unit 4a (#166) and signalk-parquet-history-provider#14.