Multi-process database support: absolute paths, open retry, cross-process change signal

[asiergmorato]

What

Three core changes that let a PowerSync database file be safely shared across processes (an app and its widgets or App Intents extensions). Extracted from #146 (the SwiftData DataStore) for separate review, as requested there.

This PR has no SwiftData dependency: it only touches the PowerSync target. #146 will rebase on top of it.

Changes

1. Absolute dbFilename paths. PowerSyncDatabase(dbFilename:) treats a path starting with / as absolute and uses it as-is, so the database can live in an App Group container. close(deleteDatabase: true) deletes the files at that location. Plain filenames keep the existing behavior (default databases directory).

2. Concurrent-open retry. pragma journal_mode = WAL needs a moment of exclusive access and reports SQLITE_BUSY/SQLITE_BUSY_RECOVERY without consulting the busy handler, so busy_timeout cannot cover the open path. Two processes opening the same file concurrently (an app launching while its extension opens the same file) failed ~80% of the time in a two-process harness, even for read-only consumers (the pool always opens a writer first). Opening and configuring a connection now retries with backoff. Reproduced in-process with a system-SQLite connection holding a reserved lock, which is the same file-level contention two processes produce.

3. Cross-process change signal. Update hooks are per-pool, so watch was structurally blind to writes made by other processes. Each pool now posts a Darwin notification after every committed write (the mechanism Core Data uses for its remote-change notifications; name derived from the canonical file path) and, on receipt, re-emits tableUpdates with EXTERNAL_CHANGES_MARKER, which watch and the upload client treat as "unknown tables changed". In-memory databases skip the signal. Two PowerSyncDatabase instances over the same file in one process use independent pools, reproducing the cross-process behavior deterministically (this is how the test exercises it).

On locking

You mentioned an asynchronous locking scheme in your other SDKs to avoid using the write connection concurrently. The open retry here is a pragmatic first step (it backs off on SQLITE_BUSY during the WAL transition, where the busy handler does not apply), and it currently sleeps on a GCD thread rather than awaiting. I'm happy to align this with your existing scheme instead of keeping a parallel mechanism: where does it live? Same question for guarding concurrent writes across processes (today they serialize through busy_timeout).

Testing

Builds with -strict-concurrency=complete. 35 tests pass across the new suites (ConcurrentOpenTests, CrossProcessSignalTests, AbsolutePathTests) and the in-memory sync integration tests. Also validated with a real two-process harness (separate OS processes over one file): concurrent cold opens 5/5 clean, and the observing process sees writes made by the other process.

[simolus3]

I did some research on what Darwin APIs there are for IPC, and it looks like the good stuff is all inaccessible to apps. So I think synchronizing by retrying on sqlite_busy errors is actually fine. It might be worth exerimenting with this suggestion to manage exclusive file access through NSFileCoordinator, but that sounds like it could add a fair bit of overhead.

The recovery loop when we open databases looks good to me (modulo blocking a thread, but even that it likely fine). For write statements, relying on sqlite_busy initially sounds acceptable to me too.

[asiergmorato]

Thanks @simolus3, all addressed in the latest push.

• Async retry: moved the backoff loop off the blocking usleep to Task.sleep, so it no longer pins a thread and a database open is now cancellable (added a test for that). I kept the whole pool building inside one runBlocking call rather than splitting open-then-configure, because RawSqliteConnection is ~Copyable and can't cross the async boundary; the retry/cancellation benefit comes from awaiting between attempts, which this still gets.
• Signal only for .atPath: good catch, the default directory is in the app sandbox and not reachable by extensions, so I now only enable the change signal for absolute-path databases.
• EXTERNAL_CHANGES_MARKER: made it internal.
• Shared open helper and the isDirectory: hint: done.

On the article: I read through it. Most of it targets document-style access (iCloud Drive, file providers), and NSFileCoordinator specifically doesn't fit SQLite in WAL mode well: SQLite already coordinates concurrency with POSIX file locks, WAL spans three files (.sqlite, -wal, -shm) plus shared-memory mmap that a single-URL coordinator doesn't understand, and Core Data deliberately doesn't use file coordination for its SQLite store. So I think retrying on SQLITE_BUSY (as you concluded) is the right mechanism here. I did adopt the parts that apply to us: the isDirectory: hint to avoid a blocking stat, and all our database I/O already runs off the main thread on background GCD queues, which the async retry reinforces.

[simolus3]

This looks good to me apart from some minor comments now 👍 Thanks for your contribution!

[asiergmorato]

Thanks! All three applied in the latest push:

• Renamed the Darwin notification prefix to com.powersync.changes.
• Simplified the changelog entry to your suggested wording.
• Expanded the dbFilename doc to note that the database can be used concurrently from the app and its extensions, but only the main app should call connect (extensions read and write, syncing stays with the app).

[simolus3]

Thank you for your contribution! Before we release this, we'll need to update our documentation website to point out potential issues (like connect() across app extensions not being allowed, and problems that can arise when apps using different versions of the SDK share a database). We'll take care of that, and we'll also reach out about the Swift Data integration. That will probably take a few more days, thanks for your patience.

[asiergmorato]

Thank you for letting me contribute! 🤗