Skip to content

fix(server): set hasResolvedPath before the startup load reads it (#184 part 2) - #186

Merged
dborup merged 2 commits into
masterfrom
codex/issue-184-resolved-path-startup
Oct 3, 2026
Merged

dborup merged 2 commits into
masterfrom
codex/issue-184-resolved-path-startup

Conversation

@adminopenclaw8-sketch

Copy link
Copy Markdown
Collaborator

Relates to #184 (part 2: the possible start-up race on hasResolvedPath).

Can the race happen?

Not in the form the issue describes. Issue #184 assumed: the OpenDB probe sees no resolved_path column, waitForDBSchema passes, and the start-up load runs before forceTrue(). That case is already covered:

  • dbschema.AssertReady requires observations.resolved_path (internal/dbschema/dbschema.go:316). main.go exits if the gate fails.
  • After the gate passes, waitForDBSchemaWith calls db.detectSchema() again (schema_wait.go). That re-probe sees the column the ingestor added and sets the flag before RunStartupLoad starts.

A narrower form can happen. detectSchema gives up silently when its PRAGMA table_info(observations) fails (db.go, if err != nil { return }). Suppose the OpenDB probe missed the column and the post-gate re-probe then fails. The flag stays false when RunStartupLoad starts.

The forceTrue() in main.go cannot help, because it ran only after <-store.FirstChunkReady(), so after the first (newest) chunk was already loaded. The 2 s healer does not bound the window either. With the flag false, LoadChunked takes the NULL branch: the chunk's relays go into byNode only, with no byPathHop credit until the next restart.

How likely is it? It needs a failed PRAGMA right after AssertReady succeeded on the same read-only handle. On a WAL database that is rare: a reader gets SQLITE_BUSY only in edge cases such as WAL recovery. It is not reproduced on staging.

Fix

waitForDBSchema sets hasResolvedPathFlag as soon as AssertReady has passed, since AssertReady requires the column. The late forceTrue() in main.go is removed. The flag is now true before RunStartupLoad starts, whatever the re-probe does.

DB.schemaProbeHook is a new test seam, nil in production and in the same style as channelsRowsHook. It lets a test make a detectSchema pass fail.

Test

TestStartupLoadIndexesResolvedPathAfterSchemaGate_184 uses the e2e fixture. The column is dropped, so the probe at open sees the flag as false. The "ingestor" then runs dbschema.Apply and writes a resolved_path for one observation. The test then runs waitForDBSchema → graph load → RunStartupLoad, in main.go's order. It asserts the flag, and that byPathHop[<resolved pubkey>] holds the transmission.

Subtest On master (bedbe1f1) With the fix
column added after the first probe (the issue's scenario) pass pass
post-gate probe fails fail: flag false, byPathHop 0 tx pass

The DB is built without OpenDB's healer goroutine, so the window is deterministic.

Mutant, run in a copy of the tree: put forceTrue() back after RunStartupLoad in main.go and remove it from waitForDBSchema. Result: post-gate probe fails is red on both assertions.

Runs

  • cd cmd/server && go test ./...: ok. Master has 2,050 top-level tests and 624 subtests. This branch has 2,051 and 626, with 0 failures. 2 top-level tests are skipped on both.
  • go test -race -count=10 -run 'TestStartupLoadIndexesResolvedPathAfterSchemaGate_184|TestWaitForDBSchema|TestWaitForServerSchema|TestWaitForSchema': ok.
  • go vet and gofmt: clean.
  • No changes to cmd/ingestor, to store.go's ingest paths (fix(store): index live observations from the persisted resolved_path, as Load does (#158) #182 changes those) or to .github/ (the 9 fork guards are unchanged).

Performance

There is one atomic store at start-up. No hot path is touched.

Not in this PR (found while reading)

  • LoadChunked (chunked_load.go) and Load (store.go) read hasResolvedPath() twice for each chunk. They read it once when building the SELECT and once for every row's scan targets. If the flag flips between the two reads, which the healer can do, the column count and the scan targets differ. The rows of that chunk are then dropped with scan error. This PR removes the flip for resolved_path. The same pattern exists for raw_hex, scope_name and route_mask. A fix would be one snapshot of the flags per chunk.
  • The other optional-column flags that AssertReady guarantees could be set the same way: scope_name, last_seen, route_mask, default_scope, configured_scope and default_scope_confirmed_at.

Not verified

  • No staging deploy. The race was never observed in production.
  • The CI result is not part of this description.

🤖 Generated with Claude Code

dborup and others added 2 commits October 3, 2026 10:23
…gate (#184)

Two cases, both starting with hasResolvedPath false at OpenDB because the
ingestor adds the column afterwards:

- the post-gate re-probe works: passes on master (waitForDBSchemaWith
  already calls detectSchema after AssertReady);
- the post-gate probe fails: red on master. detectSchema gives up
  silently, the flag stays false, and RunStartupLoad indexes the
  observation without byPathHop.

Adds DB.schemaProbeHook, a nil-in-production test seam that makes a
detectSchema pass fail the way a failed PRAGMA table_info does.

Relates to #184

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

main.go forced the flag only after <-store.FirstChunkReady(), so it could
never help the first (newest) chunk. Until then the flag came from
detectSchema, which returns silently when its PRAGMA fails. If both the
OpenDB probe and the post-gate probe miss the column, the hot window is
loaded without resolved_path: byNode only, no byPathHop credit for its
relays until the next restart.

waitForDBSchema now sets the flag as soon as dbschema.AssertReady has
passed, since AssertReady requires observations.resolved_path. The late
forceTrue in main.go is removed.

Relates to #184

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dborup
dborup marked this pull request as ready for review October 3, 2026 09:05
@dborup
dborup merged commit ae59568 into master Oct 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants