Skip to content

fix(storage): probe before the pipe, not inside it (#462) - #904

Open
fujibee wants to merge 1 commit into
mainfrom
fix/probe-before-the-pipe
Open

fix(storage): probe before the pipe, not inside it (#462)#904
fujibee wants to merge 1 commit into
mainfrom
fix/probe-before-the-pipe

Conversation

@fujibee

@fujibee fujibee commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Declared reviewers: 1

Change class: performance, no behaviour change. Every statement runs as
before; some of them stop starting a second sqlite3 process to ask a question
that was already answered.

What is wrong

agmsg_sqlite memoises the escape-flag probe so it costs one sqlite3 process
per shell rather than one per call — that is #462, and the reason is written
above the function.

The right-hand side of a pipeline is a subshell. It inherits a memo, but a
memo it sets there dies with it. So a process whose first database access is
piped records nothing, and every piped call after it probes again.

Measured, in a shell whose first access is piped:

pipe#1 = 2 sqlite3 processes   probed_in_parent=unset
pipe#2 = 2                     probed_in_parent=unset
pipe#3 = 2                     probed_in_parent=unset

Against the same call after an unpiped one has warmed it:

argv (cold) = 2      pipe#1 = 1      pipe#2 = 1

It does not converge: the piped form pays the probe forever.

Where

Ten sites, derived with grep -rn '| agmsg_sqlite' scripts/ rather than
enumerated. Three of them arrived with #895 — my change, which moved
data-sized SQL onto stdin and was reviewed and cleared without the shell rule
underneath it being noticed. The rest predate it, including the migration
helper and the legacy import path.

A redirection is not a pipe. agmsg_sqlite db < file runs in the current
shell and memoises normally, so the four < "$sql_file" sites need nothing —
including the one #899 just added.

The fix

agmsg_sqlite_warm beside agmsg_sqlite, called on the line before a
pipeline. Two of the driver's statements go through a helper that warms, so the
rule has one place there rather than one per statement.

Measured

tests/test_remote_sync.bats     33/33   (1 new)
tests/test_storage.bats         24/24
check-enforced-assertions       638, at the baseline

The new case is a scan, not a list: it derives the piped sites and fails on
any without a warm above it. Written that way because the sites move — three
appeared in a single day, in a change that two people read.

warm removed from the helper       -> red
an unwarmed pipe planted in sqlite.sh -> red, and it names the file and line
none (control)                     -> green

It also asserts the scan can see at least eight sites, so an instrument that
matched nothing could not pass as "everything is warm".

agmsg_sqlite memoises the escape probe so it costs one sqlite3 process per
shell rather than one per call (#462). The right-hand side of a pipeline is a
subshell: it inherits the memo, but a memo it sets there dies with it. A
process whose first database access is piped therefore records nothing and
every piped call after it probes again -- measured at two sqlite3 processes
per call, and it does not converge.

Ten piped sites, derived rather than listed. Three arrived with #895's move
onto stdin, which was reviewed and cleared without the shell rule underneath
it being noticed; the rest predate it. A redirection is not a pipe and needs
nothing.

The scan that found them is now a test, so the eleventh is caught rather than
counted on being noticed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant