diff --git a/CONTEXT.md b/CONTEXT.md index a3bc287e0..38642225e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -12,6 +12,26 @@ _Avoid_: Device, controller, scooter A deleted Board's surviving row, marked by a deletion stamp. The Board leaves every Rider-facing list but stays resolvable by id, so Ride History can still name the Board that produced it. Its configuration is hard-deleted; its telemetry and Tune Profiles are not (ADR 0027). _Avoid_: Soft delete, archived Board +**Sync Action**: +An append-only local record that something was semantically removed, so the removal reaches the Vescape Account backup. A deleted row cannot carry a **Change Timestamp** saying it is gone, so the log is the only signal there is. Typed — `delete` is the only type today — and written from Rider-facing removal paths only, never from retention, migrations or a database trigger. +_Avoid_: Delete log, tombstone table, audit trail, change event + +**Sync Cursor**: +A phone-held, device-local position saying how far one table has been accepted by the server. It never crosses the wire — the server keeps no watermark — and it runs on a counter rather than a clock, so a device clock that steps backwards cannot make the upload scan skip a write. Advanced only after a response, in its own transaction, so the failure mode is always a harmless re-send. +_Avoid_: Watermark, sync token, last-synced timestamp, offset + +**Sync Batch**: +One upload: rows from one or more tables, sent in the order the server applies them so a Board-owned row never arrives before its Board. Capped by row count and by actual compact JSON bytes. Accepted whole or refused whole — nothing is half-applied, and nothing is skipped to make a batch fit. +_Avoid_: Sync payload, upload chunk, page, delta + +**Backup Status**: +Native's one answer to "what is my backup doing": signed out, up to date, syncing, waiting for Wi-Fi, offline, or paused with the reason that stopped it. Derived from the same state the uploader decides on, so a status line can never disagree with the uploader. JS renders it and derives none of its own; every paused reason also raises a notification, because a pause never clears through ordinary retry. +_Avoid_: Sync state, upload progress, connection status + +**Account Binding**: +The one **Vescape Account** a phone's local database belongs to, claimed by the first Account to sign in. It survives sign-out, so data recorded while signed out stays protected from retention for the same Account. A different Account cannot take over the database; it can only replace it, which the Rider has to confirm. +_Avoid_: Account link, owner id, current user + **Board Link**: The saved, probe-confirmed reachability details for a Board, including BLE peripheral id, selected Board Transport, and capabilities or firmware facts discovered for that transport. _Avoid_: Pairing, connection settings, device config @@ -360,6 +380,10 @@ _Avoid_: User, account, member, profile, friend An optional online identity that never gates the app's local, offline-first capabilities or ownership of local data. _Avoid_: Rider profile, User, Profile +**Device Token**: +A long-lived server-issued credential held by one app install that lets native call the Vescape server for a **Vescape Account's** own data without a signed-in JS runtime. +_Avoid_: API key, session token, sync token, auth token, refresh token + **Rider Presence**: A **Rider's** live shared snapshot within a **Group Ride**: location and heading from the phone **GPS Fix**, plus optional speed and **Battery SoC Estimate** when a **Board Session** is live. Ephemeral and server-relayed, never persisted on phone or server, suppressed while the Rider is inside a **Privacy Zone**. A Rider with no recent Rider Presence goes stale, then drops from the Group Ride. _Avoid_: Position update, presence ping, location share, group telemetry @@ -484,6 +508,9 @@ _Avoid_: Position update, presence ping, location share, group telemetry - A **Group Ride** contains zero or more **Riders** and exists only while at least one **Rider** is present; it owns no durable truth and is never written to **Ride History**. - A **Rider** may be in at most one **Group Ride** at a time and is identified independently of any **Board**. - A **Vescape Account** is independent of a **Rider** and may enable optional online services such as backup, sync, or paid entitlements, but is not required to use local Boards, Ride Recording, Ride History, or tuning. +- **Ride History** is owned by the **Vescape Account** and only labelled by a **Board**; deleting a Board hides it and drops its configuration but never removes the rides it produced, on the phone or on the server. +- A **Device Token** belongs to exactly one **Vescape Account** and one app install; it authorizes reading and writing that Account's data, never changing the Account itself, which requires a freshly signed-in JS runtime. +- A **Device Token** is revoked on sign-out and is not a **Group Ride** credential, which stays unauthenticated. - A **Rider Presence** belongs to one **Rider** in one **Group Ride**, derives location from a **GPS Fix** and optional speed/**Battery SoC Estimate** from a live **Board Session**, and is not produced while the Rider is inside a **Privacy Zone**. - A **Group Ride** requires only a phone **GPS Fix** to join; a **Board Session** is optional and only enriches a **Rider Presence**, never gates it. @@ -539,5 +566,6 @@ _Avoid_: Position update, presence ping, location share, group telemetry - "force update" was used to mean both denying server compatibility and locking app UI; resolved terms: use **Online Block** for denying **Online Capabilities** and **App Block** for the exceptional update-only UI state. - "version warning" was used for both an update prompt and denial of server features; resolved terms: use **Update Warning** for the non-blocking prompt and **Online Block** when **Online Capabilities** are denied. - "message" may mean version compatibility or general communication; resolved: compatibility belongs to the **Release Policy**, while a **Community Message** never changes capability availability. +- "device" in **Device Token** names the calling app install, not a **Board** and not the phone BLE peripheral; resolved: a **Device Token** identifies a caller, while records the app backs up carry no device or install identity of their own. - "posi switch" and "dual switch" refer to **Posi Sensor** mode in rider language; the firmware field name is an implementation detail. - "move board" may mean **Remote Tilt** or motor movement while disengaged; resolved term: use **Board Move** for deliberate app-driven movement of a disengaged Board. diff --git a/docs/adr/0005-ride-history-read-paths-stay-precomputed.md b/docs/adr/0005-ride-history-read-paths-stay-precomputed.md index d467b2ab5..6382da9ce 100644 --- a/docs/adr/0005-ride-history-read-paths-stay-precomputed.md +++ b/docs/adr/0005-ride-history-read-paths-stay-precomputed.md @@ -14,3 +14,7 @@ Ride History and profile screens are latency-sensitive. Normal reads must load p - Existing Ride History may keep older derived values until an explicit maintenance path exists. - Future recalculation of old summaries must be an intentional maintenance workflow, not part of normal reads. - Read paths must not mutate durable Ride History as a side effect unless that behavior is documented as maintenance. + +## Scope + +"Reconstruct" here means replaying raw **Telemetry Samples** to recompute derived values. It does not mean any join at all. Resolving a label or attribute from a small configuration table — a **Board** name from its id, a **Tune Profile** name from its id — is a bounded lookup, not a replay, and this ADR does not forbid it. ADR-0028 relies on that reading. diff --git a/docs/adr/0027-boards-are-tombstoned-never-deleted.md b/docs/adr/0027-boards-are-tombstoned-never-deleted.md index 2eb53d600..58af916ef 100644 --- a/docs/adr/0027-boards-are-tombstoned-never-deleted.md +++ b/docs/adr/0027-boards-are-tombstoned-never-deleted.md @@ -1,20 +1,26 @@ # Boards Are Tombstoned, Never Deleted -Deleting a **Board** sets `boards.deleted_at` instead of removing the row. The Board disappears from every Rider-facing list, its configuration (Board settings, Board warnings, **Alert Rules**, Last Known Board Config Values) is hard-deleted as before, and its **Ride History** is untouched — as it already was. +Deleting a **Board** sets `boards.deleted_at` instead of removing the row, on the phone and on the Vescape server alike. The Board disappears from every Rider-facing list, its configuration (Board settings, Board warnings, **Alert Rules**) and its decoded config caches (Last Known Board Config Values and any pending change notice) are hard-deleted as before, and its **Ride History** is untouched — as it already was locally, and now as it is on the server too. The reason is that **Ride History** outlives the Board that produced it. The app has always kept telemetry after a Board delete, but the `boards` row vanishing left those rides pointing at a Board id that resolves to nothing. History could only fall back to the `device_name` snapshotted on each row: a frozen label, not an identity. A tombstone keeps the row resolvable, so a deleted Board's rides still name it and still group by it. +The server makes the same row load-bearing for a different reason: it models telemetry as Board-owned through a composite foreign key with `ON DELETE CASCADE`, so a Board **Delete Action** would have wiped exactly the rides backup exists to preserve — and the phone could never have re-uploaded them, because the missing parent row makes the foreign key refuse the whole **Sync Batch**. A tombstone keeps the parent alive, so the foreign key holds and orphaned **Tune Profiles** a phone re-uploads after a Board delete land instead of wedging the batch. + ## Considered Options -- **Cascade** — deleting a Board deletes its Ride History too. Rejected outright: it deletes the thing worth keeping. +- **Cascade** — deleting a Board deletes its Ride History too. Rejected outright: it deletes the thing worth keeping, and contradicts the rule that local storage cleanup never removes anything from the backup. - **Leave the hard delete and lean on the snapshotted `device_name`.** Rejected because a name is not an identity: renames before the delete produce rides labelled inconsistently, and nothing links a ride back to the Board it came from. - **Move Board identity onto the history rows** (denormalize more at write time). Rejected as strictly more storage for strictly less: it still cannot answer "which rides came from this Board" after the Board is gone. +- **Drop the foreign key on the server's telemetry tables**, keeping `board_id` as unenforced text. Rejected because it also drops the "a Sync Batch naming an unknown Board is refused whole" guard, which is the server's protection against a half-applied batch. +- **Hard delete plus per-child Delete Actions.** Rejected because it makes one Rider intent into an unbounded list of actions, and still leaves telemetry without a parent. ## Consequences - `getBoards()` filters `deleted_at IS NULL`. `getBoard(id)` deliberately does not — **Ride History** must still be able to name a deleted Board. Callers that act on a Board rather than describe one (`buildSessionConfig`, `BoardConnectConfig.resolve`) check `deletedAt` and refuse. - An ordinary upsert never clears an existing tombstone, so deletion is terminal. Only the delete path stamps one, and deleting an already-tombstoned Board is a no-op. -- **Tune Profiles** are deliberately outside the cascade. Tuning work is expensive to recreate and survives its Board; removing one takes its own deletion. +- `boards.deleted_at` is nullable and part of the synced row, so a tombstone reaches the server as an ordinary upsert as well as through its **Sync Action**. The two say different things and are both needed: the row says the Board is deleted, the action says its configuration is gone. Keeping the cascade an explicit, replay-safe action is what stops a dumb upsert from quietly deleting rows in three other tables — the phone writes both in one transaction, stamped with the same ratcheted timestamp (#282). +- On the server the `ON DELETE CASCADE` behind the Board-owned configuration tables stops firing, because nothing is deleted anymore. The Sync Action handler deletes those children explicitly, which makes the server's cascade identical to `deleteBoardWithSettings` rather than merely similar. +- **Tune Profiles** are deliberately outside that cascade on both sides. Tuning work is expensive to recreate and survives its Board; removing one takes its own deletion. - Telemetry can carry a stable `board_id` instead of keying on the mutable BLE identifier, because the row it points at never disappears. That unblocks the identity half of the `device_name` question (#274); the label half stays governed by ADR-0005. - Tombstones accumulate. They are one small row per deleted Board, bounded by how many Boards a rider ever owned, so no pruning rule is warranted. -- The server half of this decision — tombstones crossing the wire, and the Board **Delete Action** that carries the configuration cascade — lands with Ride History backup (#276) and is out of scope here. +- Account deletion still removes everything, cascading from the server's own user row. A tombstone is a Board-level intent, not a retention policy. diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md index 7684a0807..a534441ee 100644 --- a/docs/agents/issue-tracker.md +++ b/docs/agents/issue-tracker.md @@ -84,6 +84,7 @@ Use one or more app-area labels for filtering: | `area:legal-mode` | `[Legal Mode]` | Legal Mode UI, jurisdiction speed defaults, speed-warning alerts, and legal board constraints | | `area:warnings` | `[Warnings]` | Board Warnings: app-authored condition detection, warning registry, rider-facing warnings | | `area:diagnostics` | `[Diagnostics]` | Debug Recordings, replay tooling, Diagnostic Events, dev-mode debugging surfaces | +| `area:sync` | `[Sync]` | Backup sync — native uploader, Sync Cursors, Sync Actions, Device Token, backup status | | `area:auth` | `[Auth]` | Clerk sessions, native Device Tokens, credential lifecycle, and endpoint caller policy | | `area:release` | `[Release]` | release automation, Play tracks, versioning, release notes, and GitHub Releases | | `area:assets` | `[Assets]` | hosted media, image normalization, upload state, and remote Asset references | diff --git a/modules/vescape-core/Package.swift b/modules/vescape-core/Package.swift index 6f35aa33b..633215a60 100644 --- a/modules/vescape-core/Package.swift +++ b/modules/vescape-core/Package.swift @@ -33,7 +33,9 @@ let expoOwnedSources: Set = [ /// Test-only helpers that are not themselves `XCTestCase` files, so the `*Tests.swift` rule misses /// them. They use `@testable import VescapeCore` and belong in the test target. let testSupportSources: Set = [ - "replay/ConfigReplayHarness.swift" + "replay/ConfigReplayHarness.swift", + "sync/FakeSyncServer.swift", + "sync/FakeSyncSource.swift", ] /// Symlinks into `shared/`. The pod bundles all of them through `resource_bundles`; SPM only needs diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt index 1e9b187a1..4533acc9b 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt @@ -7,6 +7,7 @@ import expo.modules.vescapecore.alerts.AlertCoordinator import expo.modules.vescapecore.appstatus.AppStatusCoordinator import expo.modules.vescapecore.weather.WeatherCoordinator import expo.modules.vescapecore.auth.NativeAuthCoordinator +import expo.modules.vescapecore.sync.SyncCoordinator import expo.modules.vescapecore.service.BoardProbeAutoStartGate import expo.modules.vescapecore.connection.BoardTransport import expo.modules.vescapecore.connection.BoardTransportDetector @@ -89,6 +90,9 @@ private fun Map.toAlertTestRule(): AlertRuleEntity? { repeatEverySeconds = normalizedAlertRepeatSeconds((this["repeatEverySeconds"] as? Number)?.toDouble()), beepCount = normalizedAlertBeepCount((this["beepCount"] as? Number)?.toInt()), source = null, + // Ephemeral: the preview rule is never persisted, so it has no last-write-wins timestamp to + // carry and never reaches the upload scan. + updatedAt = 0, ) } @@ -183,6 +187,7 @@ class VescapeCoreModule : Module() { "onNavigation", "onRouteProgress", "onWeather", + "onSyncStatus", ) // Native owns App Status truth; JS mirrors it. Push every successful refresh (late subscribers @@ -300,6 +305,26 @@ class VescapeCoreModule : Module() { CoroutineScope(Dispatchers.IO).launch { BoardWarningRegistry.get(context).emitSnapshot() } } OnStopObserving("onBoardWarnings") { stopObserving("onBoardWarnings") } + // Native owns backup state; JS mirrors it. Push every transition, and replay the current one on + // subscribe so a late listener never renders an empty status line. + // @parity /modules/vescape-core/ios/VescapeCoreModule.swift `sendSyncStatus` + // @parity /modules/vescape-core/src/index.ts `SyncStatusEvent` + SyncCoordinator.get(context).onStatusChanged = { status -> + if (shouldEmitToFrontend("onSyncStatus")) { + mainHandler.post { + if (shouldEmitToFrontend("onSyncStatus")) sendEvent("onSyncStatus", status) + } + } + } + + OnStartObserving("onSyncStatus") { + startObserving("onSyncStatus") + CoroutineScope(Dispatchers.IO).launch { + val status = SyncCoordinator.get(context).status().toMap() + mainHandler.post { sendEvent("onSyncStatus", status) } + } + } + OnStopObserving("onSyncStatus") { stopObserving("onSyncStatus") } OnStartObserving("onVescFaults") { startObserving("onVescFaults") @@ -364,6 +389,9 @@ class VescapeCoreModule : Module() { // Cold start: fetch App Status before JS asks. A foreground event arriving right after is // coalesced into this request. AppStatusCoordinator.get(context).refresh() + // The Device Token outlives the process, so a signed-in phone has to pick the uploader back + // up here: provisioning only happens once, and nothing else would start the loop again. + SyncCoordinator.get(context).resumeIfBound() } OnActivityEntersForeground { @@ -501,6 +529,19 @@ class VescapeCoreModule : Module() { Function("clearDeviceCredential") { NativeAuthCoordinator.get(context).clear() } + // The Rider confirmed the destructive Account change; native performs the ordered transition. + // @parity /modules/vescape-core/ios/VescapeCoreModule.swift `confirmSyncAccountReset` + AsyncFunction("confirmSyncAccountReset") Coroutine { + serverUrl: String, + deviceToken: String, + accountId: String, + -> + NativeAuthCoordinator.get(context).confirmAccountReset(serverUrl, deviceToken, accountId) + } + // @parity /modules/vescape-core/ios/VescapeCoreModule.swift `getSyncStatus` + AsyncFunction("getSyncStatus") Coroutine { -> + SyncCoordinator.get(context).status().toMap() + } // Stable Vescape route keeps the app decoupled from the final store destination. // @parity /modules/vescape-core/ios/VescapeCoreModule.swift `openAppUpdate` // @platform-diff Android uses the stable Android download route. diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/alerts/AlertEngine.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/alerts/AlertEngine.kt index 0e0fd6acf..524697281 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/alerts/AlertEngine.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/alerts/AlertEngine.kt @@ -181,6 +181,8 @@ internal fun withLegalModeOverlay( soundType = "preset:tick", createdAt = 0L, source = null, + // In-memory overlay: no row is ever persisted, so the sync cursor is meaningless here. + updatedAt = 0L, ) } diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/ApiResult.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/ApiResult.kt index 23f690d6b..d0fc8fd96 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/ApiResult.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/ApiResult.kt @@ -70,7 +70,16 @@ data class ApiRequest( /** * @parity /modules/vescape-core/ios/api/ApiResult.swift `ApiResponse` */ -data class ApiResponse(val status: Int, val body: String) +data class ApiResponse( + val status: Int, + val body: String, + /** + * Lowercased response headers. Only what a caller has to act on crosses this seam today: a `429` + * carries its delay in `Retry-After`, and guessing one instead would either hammer the server or + * stall a drain far longer than it asked for. + */ + val headers: Map = emptyMap(), +) /** * The single blocking HTTP seam. Production wires OkHttp; tests wire a fake and never reach the diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/VescapeApi.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/VescapeApi.kt index 7fe41797f..442b7334b 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/VescapeApi.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/VescapeApi.kt @@ -54,6 +54,38 @@ class VescapeApi( send(request, authenticated = token.isNotEmpty(), parse = parse) } + /** + * One call whose status code is the answer, not an error to classify. The uploader needs `409`, + * `413` and `429` kept apart — each has a different recovery — so it reads the raw exchange while + * still going through this class's credential, headers and 401 policy. + * + * Never retried here: `POST /api/sync` carries no create key, and the caller's own backoff is what + * decides when the same batch is offered again. + * + * @parity /modules/vescape-core/ios/api/VescapeApi.swift `exchange` + */ + suspend fun exchange( + method: HttpMethod, + path: String, + rawBody: String?, + auth: AuthMode = AuthMode.Required, + ): ApiResponse? = withContext(Dispatchers.IO) { + val token = token(auth) ?: return@withContext ApiResponse(401, "") + val request = ApiRequest( + method = method, + url = url(path, emptyMap()), + headers = headers(token.ifEmpty { null }, rawBody != null), + body = rawBody, + ) + val response = try { + transport.execute(request) + } catch (_: Exception) { + return@withContext null + } + if (response.status == 401 && token.isNotEmpty()) onUnauthorized() + response + } + /** * Resolved bearer token, empty when the call goes out anonymously, `null` when a required * credential is missing. A credential minted against another origin belongs to another @@ -188,7 +220,11 @@ object OkHttpApiTransport : ApiTransport { } builder.method(request.method.name, body) return client.newCall(builder.build()).execute().use { response -> - ApiResponse(response.code, response.body?.string().orEmpty()) + ApiResponse( + status = response.code, + body = response.body?.string().orEmpty(), + headers = response.headers.names().associate { it.lowercase() to response.header(it).orEmpty() }, + ) } } } diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/auth/NativeAuthCoordinator.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/auth/NativeAuthCoordinator.kt index 00e2ceb76..4780f2ac8 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/auth/NativeAuthCoordinator.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/auth/NativeAuthCoordinator.kt @@ -6,6 +6,7 @@ import expo.modules.vescapecore.api.AuthMode import expo.modules.vescapecore.api.HttpMethod import expo.modules.vescapecore.api.VescapeApi import expo.modules.vescapecore.appstatus.AppStatusCoordinator +import expo.modules.vescapecore.sync.SyncCoordinator import org.json.JSONObject /** @@ -51,8 +52,39 @@ class NativeAuthCoordinator(private val context: Context) { else -> throw IllegalStateException("Account verification failed ($result)") } + // The database is claimed before the credential is stored: a second Account must not be able to + // upload from a database full of the first Account's Boards, Ride History and locations. The + // Rider confirms the destructive reset, and only then does [confirmAccountReset] finish this. + if (!SyncCoordinator.get(context).bindAccount(accountId)) { + return stateMap() + mapOf("accountChangeRequiresReset" to true) + } + + store.write(DeviceCredential(origin, token, accountId, null)) + AppStatusCoordinator.get(context).refresh() + SyncCoordinator.get(context).start() + return stateMap() + } + + /** + * The Rider confirmed that all local app data is erased and cannot yet be restored. + * + * One ordered transition: stop the uploader, invalidate in-flight work, replace the app-data + * database, clear Sync Cursors and pending Sync Actions, bind the fresh database to the new + * Account, install the new Device Token, start the uploader. Cancelling never reaches here, so the + * old database and Account binding stay untouched. + */ + suspend fun confirmAccountReset( + serverUrl: String, + token: String, + accountId: String, + ): Map { + val origin = serverUrl.trimEnd('/') + SyncCoordinator.get(context).resetForAccount(accountId) + // The token is installed before the uploader starts: a loop running on the previous Account's + // credential against the new Account's database is exactly what this ordering exists to prevent. store.write(DeviceCredential(origin, token, accountId, null)) AppStatusCoordinator.get(context).refresh() + SyncCoordinator.get(context).start() return stateMap() } @@ -75,7 +107,12 @@ class NativeAuthCoordinator(private val context: Context) { store.clear() } - fun clear() = store.clear() + fun clear() { + store.clear() + // Signing out stops the uploader but keeps the Account binding, so data recorded while signed + // out stays protected from retention for the same Account. + SyncCoordinator.get(context).stop() + } companion object { private const val ACCOUNT_PATH = "/api/account" diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/recording/RecordingCoordinator.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/recording/RecordingCoordinator.kt index 425f24f91..5b0659047 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/recording/RecordingCoordinator.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/recording/RecordingCoordinator.kt @@ -3,6 +3,7 @@ package expo.modules.vescapecore.recording import expo.modules.vescapecore.protocol.LocationSnapshot import android.content.Context import expo.modules.vescapecore.service.SessionConfig +import expo.modules.vescapecore.sync.SyncCoordinator import expo.modules.vescapecore.telemetry.AppDataRepository import expo.modules.vescapecore.telemetry.AppSettings import expo.modules.vescapecore.telemetry.TelemetryCapture @@ -141,8 +142,15 @@ internal class RecordingCoordinator( recorder = null } + /** + * The three ways recording stops — the session finishing, failing, or the Rider switching it + * off. The flush has to land before the kick, or the uploader scans a ride missing its tail. + * + * @parity /modules/vescape-core/ios/recording/RecordingCoordinator.swift `flushTelemetryBlocking` + */ private fun flushTelemetryBlocking() { telemetryStore?.flushBlocking() + SyncCoordinator.get(context).notifyRecordingStopped() } private fun configuredTelemetryStore(): TelemetryRepository { diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncAccepted.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncAccepted.kt new file mode 100644 index 000000000..8f1f32a4b --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncAccepted.kt @@ -0,0 +1,74 @@ +package expo.modules.vescapecore.sync + +/** + * The `200` body: what the server took, per table. + * + * Validated exactly before any cursor moves. A missing table, an extra table, a non-integer count or + * a count that differs from what was submitted is a protocol failure — the server applies a batch + * whole, so anything else means the two sides disagree about what was stored, and advancing a cursor + * on that disagreement is unrecoverable. + * + * Parsed here rather than with the platform JSON so the rule runs in plain unit tests and behaves + * identically on both platforms. + * + * @parity /modules/vescape-core/ios/sync/SyncAccepted.swift + */ +object SyncAccepted { + /** Accepted counts by table, or null when the body is not exactly the expected response. */ + fun parse(body: String): Map? { + val counts = LinkedHashMap() + val scanner = Scanner(body) + if (!scanner.expect('{') || !scanner.expectKey("accepted") || !scanner.expect('{')) return null + if (scanner.peek() != '}') { + while (true) { + val name = scanner.string() ?: return null + val table = SyncTable.entries.firstOrNull { it.wire == name } ?: return null + if (counts.containsKey(table) || !scanner.expect(':')) return null + counts[table] = scanner.integer() ?: return null + if (scanner.expect(',')) continue + break + } + } + if (!scanner.expect('}') || !scanner.expect('}') || !scanner.atEnd()) return null + return if (counts.size == SyncTable.entries.size) counts else null + } + + /** True when the response accounts for exactly the rows submitted, table by table. */ + fun matches(submitted: Map, accepted: Map): Boolean = + SyncTable.entries.all { accepted[it] == (submitted[it] ?: 0) } + + private class Scanner(private val source: String) { + private var index = 0 + + fun atEnd(): Boolean = skipSpace().let { index >= source.length } + + fun peek(): Char? = skipSpace().let { source.getOrNull(index) } + + fun expect(char: Char): Boolean { + if (peek() != char) return false + index += 1 + return true + } + + fun expectKey(name: String): Boolean = string() == name && expect(':') + + fun string(): String? { + if (!expect('"')) return null + val end = source.indexOf('"', index) + // Counts and table names carry no escapes; a body that needs them is not this response. + if (end < 0) return null + return source.substring(index, end).also { index = end + 1 } + } + + fun integer(): Int? { + skipSpace() + val start = index + while (index < source.length && source[index].isDigit()) index += 1 + return if (index == start) null else source.substring(start, index).toIntOrNull() + } + + private fun skipSpace() { + while (index < source.length && source[index].isWhitespace()) index += 1 + } + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncBatchBuilder.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncBatchBuilder.kt new file mode 100644 index 000000000..8b0f05635 --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncBatchBuilder.kt @@ -0,0 +1,128 @@ +package expo.modules.vescapecore.sync + +/** + * One row waiting to be uploaded: its cursor position and the compact JSON the server will read. + * + * The JSON is encoded once, by the wire layer, so the builder measures the bytes that will actually + * be sent rather than estimating from an object graph. + * + * @parity /modules/vescape-core/ios/sync/SyncBatchBuilder.swift `SyncPendingRow` + */ +data class SyncPendingRow(val cursor: Long, val json: String) { + val byteCount: Int = json.toByteArray(Charsets.UTF_8).size +} + +/** One table's pending rows, in cursor order. */ +data class SyncPendingTable(val table: SyncTable, val rows: List) + +/** + * What the builder made of the pending rows. + * + * @parity /modules/vescape-core/ios/sync/SyncBatchBuilder.swift `SyncBatchBuild` + */ +sealed interface SyncBatchBuild { + /** Nothing pending. */ + object Empty : SyncBatchBuild + + /** + * A batch and the cursor advance set describing exactly the rows in it. Cursors are committed only + * after the server accepts, and only these positions move. + */ + data class Ready( + val body: String, + val counts: Map, + val advances: Map, + val rowCount: Int, + val byteCount: Int, + ) : SyncBatchBuild + + /** + * One row cannot fit a batch of its own. Never skipped and never quarantined: the engine pauses + * with the row retained, because dropping it would silently lose data a Rider believes is backed + * up. + */ + data class RowTooLarge(val table: SyncTable, val cursor: Long, val byteCount: Int) : SyncBatchBuild +} + +/** + * Fills a Sync Batch from per-table pending rows. + * + * Pure: no database, no clock, no network. It walks [SyncTable] declaration order — the order the + * server applies a batch in — and stops at whichever cap comes first. Ordering by backlog size would + * produce a batch whose children arrive before their parents, which the server refuses whole. + * + * @parity /modules/vescape-core/ios/sync/SyncBatchBuilder.swift `SyncBatchBuilder` + */ +object SyncBatchBuilder { + fun build( + pending: List, + rowCap: Int = MAX_SYNC_BATCH_ROWS, + byteCap: Int = MAX_SYNC_BATCH_BYTES, + ): SyncBatchBuild { + val ordered = pending + .filter { it.rows.isNotEmpty() } + .sortedBy { it.table.ordinal } + if (ordered.isEmpty()) return SyncBatchBuild.Empty + + val body = StringBuilder("{") + val counts = LinkedHashMap() + val advances = LinkedHashMap() + var rowCount = 0 + // `{}`; every other cost below is added as the exact bytes appended. + var byteCount = 2 + + for (group in ordered) { + if (rowCount >= rowCap) break + // `,"appSettings":[]` — the separating comma only once a table is already open. + val header = (if (counts.isEmpty()) "" else ",") + "\"" + group.table.wire + "\":[" + val tableOverhead = header.length + 1 + if (byteCount + tableOverhead > byteCap) break + + var opened = false + var truncated = false + for (row in group.rows) { + if (rowCount >= rowCap) break + val rowCost = row.byteCount + if (opened) 1 else 0 + val overhead = if (opened) 0 else tableOverhead + if (byteCount + overhead + rowCost > byteCap) { + // A row no empty batch could carry is a permanent local protocol error, not a cap hit. + if (counts.isEmpty() && !opened && 2 + tableOverhead + row.byteCount > byteCap) { + return SyncBatchBuild.RowTooLarge(group.table, row.cursor, row.byteCount) + } + truncated = true + break + } + + if (!opened) { + body.append(header) + byteCount += tableOverhead + counts[group.table] = 0 + opened = true + } else { + body.append(',') + } + body.append(row.json) + byteCount += rowCost + rowCount += 1 + counts[group.table] = counts.getValue(group.table) + 1 + advances[group.table] = row.cursor + } + if (opened) body.append(']') + // A table cut short by the byte cap may still hold a parent — a Board whose settings, alerts + // or Tune Profiles sit further down this same batch. Carrying on would send the child ahead of + // it, and the server refuses that whole batch on the foreign key. The rest waits for the next + // batch, which starts where this one stopped. + if (truncated) break + } + + if (counts.isEmpty()) return SyncBatchBuild.Empty + body.append('}') + return SyncBatchBuild.Ready( + body = body.toString(), + counts = counts, + advances = advances, + rowCount = rowCount, + byteCount = byteCount, + ) + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt new file mode 100644 index 000000000..e33f6512a --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt @@ -0,0 +1,466 @@ +package expo.modules.vescapecore.sync + +import android.content.Context +import android.net.ConnectivityManager +import android.net.Network +import android.net.NetworkCapabilities +import android.util.Log +import expo.modules.vescapecore.api.HttpMethod +import expo.modules.vescapecore.api.VescapeApi +import expo.modules.vescapecore.appstatus.AppStatusCoordinator +import expo.modules.vescapecore.auth.DeviceCredentialStore +import expo.modules.vescapecore.telemetry.AppDataRepository +import expo.modules.vescapecore.telemetry.DatabaseBackupManager +import expo.modules.vescapecore.telemetry.TelemetryDatabase +import expo.modules.vescapecore.telemetry.TelemetryRepository +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Job +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.delay +import kotlinx.coroutines.isActive +import kotlinx.coroutines.launch +import kotlinx.coroutines.sync.Mutex +import kotlinx.coroutines.sync.withLock + +private const val TAG = "SyncCoordinator" + +/** What JS renders. Native owns every transition; JS only asks and shows. */ +data class SyncStatus( + val accountId: String?, + val pendingRows: Int, + val activity: SyncActivity, + val pause: SyncPauseReason?, + val lastUploadAtMs: Long?, +) { + fun toMap(): Map = mapOf( + "accountId" to accountId, + "pendingRows" to pendingRows, + "activity" to activity.slug, + "pause" to pause?.slug, + "lastUploadAtMs" to lastUploadAtMs, + ) +} + +/** + * The uploader's lifecycle: the loop, the kicks, and the Account binding it runs under. + * + * Runs inside the window the app already keeps alive — the foreground service during a Board Session + * or GPS, the existing background modes on iOS. Deliberately no `WorkManager`: a ride that ends + * offline on a phone that is never reopened waits for the next app open or the next ride. + * + * @parity /modules/vescape-core/ios/sync/SyncCoordinator.swift + */ +class SyncCoordinator private constructor(private val context: Context) { + /** Resolved per call: an Account reset replaces the whole database file under this object. */ + private val dao get() = TelemetryDatabase.get(context).telemetryDao() + private val credentials = DeviceCredentialStore(context) + private val scope = CoroutineScope(SupervisorJob()) + + /** Bumped by an Account reset; a response captured under an older value cannot commit. */ + @Volatile private var generation = 0L + + @Volatile private var lastSamplePersistedAtMs = 0L + @Volatile private var lastUploadAtMs: Long? = null + @Volatile private var wifiOnly = false + + /** The master switch. Off by default: nothing uploads until the Rider turns backup on. */ + @Volatile private var enabled = false + + /** Failure keys already recorded this process, so a wedged batch writes one event, not a stream. */ + private val recordedFailures = HashSet() + + private var loop: Job? = null + + /** Kicks in flight, so [stop] leaves nothing running against a database about to be replaced. */ + private val kicks = java.util.concurrent.CopyOnWriteArrayList() + + /** Serializes passes against each other and against [resetForAccount]. */ + private val passLock = Mutex() + + /** + * Last reachability the callback below reported, so a kick fires on the transition rather than on + * every capability change a network publishes while it is already up. + * + * Starts true: a phone that has been online all along must not read its first callback as a + * regain, which would kick on nothing. + */ + @Volatile private var online = true + + /** + * Connectivity regained is one of the immediate kicks, next to the ride ending and sign-in. Without + * it a phone that comes back online waits out the idle interval with a full backlog in hand. + * + * Registered for the process lifetime, not per pass: the callback is what makes the regain + * observable at all, and [kick] is a no-op whenever the switch is off or the loop is not running. + * + * @parity /modules/vescape-core/ios/sync/SyncCoordinator.swift `pathUpdateHandler` + */ + private val networkCallback = object : ConnectivityManager.NetworkCallback() { + override fun onCapabilitiesChanged(network: Network, capabilities: NetworkCapabilities) { + val reachable = capabilities.hasCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET) + val regained = reachable && !online + online = reachable + if (regained) kick() + } + + override fun onLost(network: Network) { + online = false + } + } + + private val store = SyncStore( + database = { dao }, + generation = { generation }, + onPermanentFailure = ::recordPermanentFailure, + ) + + private val engine = SyncEngine( + source = store, + transport = ::post, + environment = ::environment, + ) + + init { + runCatching { + context.getSystemService(ConnectivityManager::class.java) + ?.registerDefaultNetworkCallback(networkCallback) + } + } + + val pauseReason: SyncPauseReason? get() = engine.pauseReason + + /** + * Wired by the module: every status transition, pushed to JS. Native owns the state; JS renders it + * and never derives one of its own. + */ + @Volatile var onStatusChanged: ((Map) -> Unit)? = null + + /** Last map handed out, so an unchanged status emits nothing and raises no second notification. */ + @Volatile private var publishedStatus: Map? = null + + /** Recording persisted samples: the ride cadence follows sample production, not session presence. */ + fun notifySamplesPersisted(atMs: Long = System.currentTimeMillis()) { + lastSamplePersistedAtMs = atMs + } + + /** + * The ride ended and its last samples are on disk. Called after the final flush, so the kick scans + * a complete ride rather than one missing its tail. + * + * This is the moment with the largest fresh backlog and the moment a Rider is most likely to open + * the app and look at the status line, which is why it does not wait for the next tick. + * + * @parity /modules/vescape-core/ios/sync/SyncCoordinator.swift `notifyRecordingStopped` + */ + fun notifyRecordingStopped() { + kick() + } + + /** + * A Rider changed something durable and small — a Favorite pinned, renamed or unpinned. One row, + * created by hand, and the Rider is looking at the screen that says whether it is backed up, so a + * five-minute wait reads as the backup not working. + * + * Deliberately not wired to telemetry writes: those arrive at 2 Hz and already have the ride + * cadence. This is for edits a Rider makes, which are rare and individually visible. + * + * @parity /modules/vescape-core/ios/sync/SyncCoordinator.swift `notifyRiderEdit` + */ + fun notifyRiderEdit() { + kick() + } + + /** + * The master switch, from the App Setting native owns. Off stops the loop outright — no scan, no + * request, no backoff, no pause notification — rather than leaving a loop that decides to do + * nothing every five minutes. + */ + fun setEnabled(value: Boolean) { + if (enabled == value) return + enabled = value + if (value) { + start() + } else { + stop() + // A switched-off uploader asks the Rider for nothing: an earlier pause is no longer theirs to + // act on until they turn backup back on. + SyncNotifier.get(context).update(null) + } + scope.launch { publishStatus() } + } + + /** + * The "Back up over Wi-Fi only" App Setting, pushed by [AppDataRepository] on every write and read + * back on launch. Native reads the setting itself — JS never carries the switch to the uploader. + */ + fun setWifiOnly(enabled: Boolean) { + if (wifiOnly == enabled) return + wifiOnly = enabled + kick() + scope.launch { publishStatus() } + } + + suspend fun status(): SyncStatus { + val environment = environment() + val pending = store.pendingCount() + val pause = engine.pauseReason + return SyncStatus( + accountId = dao.getBoundAccountId(), + pendingRows = pending, + activity = SyncPolicy.describe( + SyncState( + nowMs = System.currentTimeMillis(), + pendingRows = pending, + ridingSamples = environment.ridingSamples, + enabled = environment.enabled, + online = environment.online, + wifiOnly = environment.wifiOnly, + onWifi = environment.onWifi, + credentialReady = environment.credentialReady, + onlineBlocked = environment.onlineBlocked, + pause = pause, + // Backoff is invisible to the Rider: a batch waiting to be retried is still syncing. + retryAtMs = 0, + ), + ), + pause = pause, + lastUploadAtMs = lastUploadAtMs, + ) + } + + /** + * Emit the current status when it differs from the last one, and raise the notification a pause + * needs: a permanent failure does not resolve through ordinary retry, so a backup that stopped + * weeks ago must not wait for the Rider to open the social sheet. + */ + private suspend fun publishStatus() { + val status = runCatching { status() }.getOrNull() ?: return + val map = status.toMap() + if (map == publishedStatus) return + val previousPause = publishedStatus?.get("pause") as? String + publishedStatus = map + if (status.pause?.slug != previousPause) SyncNotifier.get(context).update(status.pause) + onStatusChanged?.invoke(map) + } + + /** + * Pick the uploader back up on a cold launch: the credential outlives the process, so a phone that + * was signed in stays signed in, and nothing else would ever start the loop again. Binding the + * stored Account is a no-op when this database already belongs to it, and cannot claim a database + * that belongs to another one. + */ + fun resumeIfBound() { + scope.launch { + // The switch is a durable App Setting, so the uploader restores it before the first pass of + // this process — otherwise a cold launch on mobile data would upload once before JS loaded. + val settings = runCatching { AppDataRepository.get(context).getTypedSettings() }.getOrNull() + wifiOnly = settings?.syncWifiOnly ?: false + enabled = settings?.syncEnabled ?: false + val credential = credentials.read() + // Binding still happens with the switch off — it is what makes this database's Account known, + // and `start()` below is the only thing the switch gates. + if (credential != null && bindAccount(credential.accountId) && enabled) start() + publishStatus() + } + } + + fun start() { + if (!enabled) return + if (loop?.isActive == true) return + loop = scope.launch { + while (isActive) { + val waitMs = try { + pass() + } catch (e: Exception) { + Log.w(TAG, "Sync pass failed: ${e.message}") + SyncPolicy.IDLE_INTERVAL_MS + } + publishStatus() + delay(waitMs) + } + } + } + + /** Stops the loop and every kick in flight, so nothing is left running over a replaced database. */ + fun stop() { + loop?.cancel() + loop = null + kicks.forEach { it.cancel() } + kicks.clear() + } + + /** Connectivity regained, ride ended, sign-in: send now rather than waiting for the next tick. */ + fun kick() { + if (!enabled) return + if (loop?.isActive != true) return start() + val job = scope.launch { + runCatching { pass() } + publishStatus() + } + kicks.add(job) + job.invokeOnCompletion { kicks.remove(job) } + } + + /** + * One pass, draining while the server keeps accepting: a `200` with rows still pending sends again + * straight away, so a long backlog drains instead of trickling. + * + * Serialized against every other pass and against an Account reset: the whole scan → send → + * commit sequence holds the lock, so a reset can never land between reading a previous Account's + * rows and checkpointing them onto the fresh database. + */ + private suspend fun pass(): Long = passLock.withLock { + var drains = 0 + while (drains < MAX_DRAIN_STEPS) { + when (val outcome = engine.runOnce()) { + is SyncPass.Sent -> { + lastUploadAtMs = System.currentTimeMillis() + if (!outcome.morePending) return interval() + drains += 1 + } + // Nothing was accepted, but the next attempt differs — a narrowed byte target. + SyncPass.Retry -> drains += 1 + is SyncPass.Waiting -> + return (outcome.untilMs - System.currentTimeMillis()).coerceIn(0, SyncPolicy.BACKOFF_MAX_MS) + is SyncPass.Paused -> return SyncPolicy.IDLE_INTERVAL_MS + SyncPass.Idle -> return interval() + } + } + // A drain that never finishes yields rather than spinning; the next tick resumes it. + return SyncPolicy.RIDE_INTERVAL_MS + } + + private fun interval(): Long = + if (samplesProducing()) SyncPolicy.RIDE_INTERVAL_MS else SyncPolicy.IDLE_INTERVAL_MS + + private fun samplesProducing(): Boolean = + System.currentTimeMillis() - lastSamplePersistedAtMs < SAMPLE_ACTIVITY_WINDOW_MS + + private fun environment(): SyncEnvironment { + val capabilities = runCatching { + val manager = context.getSystemService(ConnectivityManager::class.java) + manager?.getNetworkCapabilities(manager.activeNetwork) + }.getOrNull() + return SyncEnvironment( + ridingSamples = samplesProducing(), + enabled = enabled, + online = capabilities?.hasCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET) == true, + wifiOnly = wifiOnly, + onWifi = capabilities?.hasTransport(NetworkCapabilities.TRANSPORT_WIFI) == true, + credentialReady = credentials.read() != null, + onlineBlocked = AppStatusCoordinator.get(context).onlineBlocked, + ) + } + + /** + * The Sync endpoints are Online Capabilities behind the App Status gate, and they authenticate with + * the shared Device Token, so the whole call goes through [VescapeApi]. + */ + private suspend fun post(body: String): SyncResponse { + val api = VescapeApi.forOrigin(context, AppStatusCoordinator.serverBaseUrl(context)) + val response = api.exchange(HttpMethod.POST, SYNC_PATH, body) + ?: return SyncResponse.Transient("network") + return when { + response.status == 200 -> SyncResponse.Accepted(response.body) + response.status == 401 -> SyncResponse.Unauthorized + response.status == 413 -> SyncResponse.TooLarge + response.status == 429 -> SyncResponse.RateLimited(retryAfterMs(response.headers)) + response.status >= 500 -> SyncResponse.Transient("http ${response.status}") + response.status >= 400 -> SyncResponse.Invalid(response.status, errorSlug(response.body)) + // A `2xx` that is not the accepted map is a protocol failure, not a success to interpret. + else -> SyncResponse.Invalid(response.status, "unexpected-success") + } + } + + /** The server's own delay in seconds, or the first backoff step when it named none. */ + private fun retryAfterMs(headers: Map): Long = + headers["retry-after"]?.trim()?.toLongOrNull()?.times(1_000L) ?: SyncPolicy.BACKOFF_START_MS + + private fun errorSlug(body: String): String = + Regex("\"error\"\\s*:\\s*\"([^\"]+)\"").find(body)?.groupValues?.get(1) ?: "invalid-request" + + // Account binding — the Device Token exchange returns a stable server Account id, and the first + // Account claims this database. + + /** + * Claim the local database for [accountId] when it is unbound or already belongs to it. + * + * False means a different Account: cursors are deliberately not reset over the existing rows, + * because that would upload the previous Account's Boards, Ride History, locations and settings to + * the new one. The Rider has to confirm the destructive reset first. + */ + suspend fun bindAccount(accountId: String): Boolean { + val bound = dao.bindAccount(accountId) + if (bound) { + engine.resume() + kick() + } + return bound + } + + /** + * The Account change transition, in the one order that cannot leak data between Accounts: stop the + * loop, invalidate in-flight work, replace the database, clear cursors and pending actions, bind + * the new Account, then start again. + * + * The wipe is local maintenance and emits no Sync Actions to either Account — replacing the file + * removes the log with everything else. + */ + suspend fun resetForAccount(accountId: String) { + stop() + // Held across the whole transition: a pass that started before `stop()` finishes its scan, send + // and commit against the old database before the file is replaced, and none can start midway. + passLock.withLock { + // Every in-flight response now belongs to a previous Account and can no longer commit. + generation += 1 + recordedFailures.clear() + DatabaseBackupManager.replaceWithFreshDatabase(context) + check(dao.bindAccount(accountId)) { "Fresh database did not accept the new Account" } + engine.resume() + lastUploadAtMs = null + } + // Deliberately not started here: the caller installs the new Device Token first, so the loop + // never runs with the previous Account's credential against the new Account's database. + publishStatus() + } + + /** + * One coalesced Diagnostic Event per failure class, table and cursor. Metadata only: an error + * code, a table, a cursor and the app version — never row contents, coordinates, the Device Token, + * the server body or an opaque database error. + */ + private fun recordPermanentFailure(reason: SyncPauseReason, detail: String) { + val key = "${reason.slug}:$detail" + synchronized(recordedFailures) { + if (!recordedFailures.add(key)) return + } + TelemetryRepository.get(context).recordDiagnosticEvent( + "sync_upload_paused", + mapOf( + "operation" to "sync", + "phase" to reason.slug, + "message" to "Sync upload paused", + "sync_failure" to reason.slug, + "sync_detail" to detail, + "app_version" to AppStatusCoordinator.get(context).appVersion, + ), + ) + } + + companion object { + internal const val SYNC_PATH = "/api/sync" + + /** Samples persisted this recently mean a ride is producing, Idle Pause included. */ + private const val SAMPLE_ACTIVITY_WINDOW_MS = 60_000L + + /** A drain is a burst, not a loop that can never yield to the rest of the process. */ + private const val MAX_DRAIN_STEPS = 50 + + @Volatile private var instance: SyncCoordinator? = null + + fun get(context: Context): SyncCoordinator = + instance ?: synchronized(this) { + instance ?: SyncCoordinator(context.applicationContext).also { instance = it } + } + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncEngine.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncEngine.kt new file mode 100644 index 000000000..3c861ca80 --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncEngine.kt @@ -0,0 +1,231 @@ +package expo.modules.vescapecore.sync + +/** What the transport made of one `POST /api/sync`. */ +sealed interface SyncResponse { + /** `2xx`. The body still has to be exactly the accepted map before anything is committed. */ + data class Accepted(val body: String) : SyncResponse + + /** `400`, `409`, `422` or any other unknown `4xx`: wrong request, not a bad moment. */ + data class Invalid(val status: Int, val error: String) : SyncResponse + + /** `401`: the Device Token is dead. Only sign-in resolves it. */ + object Unauthorized : SyncResponse + + /** `413`: over the wire byte bound. Retried with a smaller target, never with fewer rows dropped. */ + object TooLarge : SyncResponse + + /** `429`, with the server's own delay. */ + data class RateLimited(val retryAfterMs: Long) : SyncResponse + + /** `5xx`, a network error or a timeout — the batch may or may not have been applied. */ + data class Transient(val reason: String) : SyncResponse +} + +/** @parity /modules/vescape-core/ios/sync/SyncEngine.swift `SyncTransport` */ +fun interface SyncTransport { + suspend fun send(body: String): SyncResponse +} + +/** + * The database side of the uploader: what is pending, and where the cursors are. + * + * @parity /modules/vescape-core/ios/sync/SyncEngine.swift `SyncSource` + */ +interface SyncSource { + /** Pending rows per table, already encoded, capped at [rowLimit] rows in total. */ + suspend fun pending(rowLimit: Int): List + + /** Rows waiting across every table. Cheap enough to ask on every tick. */ + suspend fun pendingCount(): Int + + /** + * Commit the advance set in its own transaction, after the response. Never alongside the rows: a + * cursor advanced past rows the server did not take is unrecoverable, whereas a cursor left behind + * is a re-send the server upserts idempotently. Always fail toward re-sending. + * + * Throws rather than swallowing a write failure: an uncommitted cursor leaves the same rows + * pending, and a caller that believed the checkpoint landed would resend them without pause. + */ + suspend fun commit(advances: Map) + + /** + * Bumped by an Account change. Captured before a request and re-read before the commit, so a + * response belonging to the previous Account becomes a no-op instead of advancing a cursor over + * the fresh database. + */ + fun generation(): Long + + /** One coalesced, metadata-only Diagnostic Event for a permanent failure. */ + suspend fun recordPermanentFailure(reason: SyncPauseReason, detail: String) +} + +/** Environment the policy reads. Owned by the caller, so the engine keeps no platform types. */ +data class SyncEnvironment( + val ridingSamples: Boolean, + /** The Rider's master switch, read from the App Setting native owns. */ + val enabled: Boolean, + val online: Boolean, + val wifiOnly: Boolean, + val onWifi: Boolean, + val credentialReady: Boolean, + val onlineBlocked: Boolean, +) + +/** What one pass did, for the loop and for tests. */ +sealed interface SyncPass { + object Idle : SyncPass + + /** Nothing was accepted, but the next attempt differs from this one — a narrowed byte target. */ + object Retry : SyncPass + + data class Sent(val rowCount: Int, val morePending: Boolean) : SyncPass + data class Waiting(val untilMs: Long) : SyncPass + data class Paused(val reason: SyncPauseReason) : SyncPass +} + +/** + * The uploader: scan forward from each Sync Cursor, send a small batch, advance only what the server + * accepted. + * + * Owns transport policy, backoff and the permanent pause; the two interesting decisions — which rows + * go in a batch, and whether to send at all — live in [SyncBatchBuilder] and [SyncPolicy], which are + * pure. Drives no timer of its own: [SyncCoordinator] owns the loop and the kicks. + * + * @parity /modules/vescape-core/ios/sync/SyncEngine.swift `SyncEngine` + */ +class SyncEngine( + private val source: SyncSource, + private val transport: SyncTransport, + private val environment: () -> SyncEnvironment, + private val clock: () -> Long = System::currentTimeMillis, +) { + private var retryAtMs = 0L + private var backoffMs = 0L + private var byteTarget = MAX_SYNC_BATCH_BYTES + private var pause: SyncPauseReason? = null + + val pauseReason: SyncPauseReason? get() = pause + + /** Clears a pause. Sign-in and an Account reset are the only things that may. */ + fun resume() { + pause = null + retryAtMs = 0 + backoffMs = 0 + byteTarget = MAX_SYNC_BATCH_BYTES + } + + /** + * One pass: decide, send, commit. A `200` with rows still pending returns `morePending`, so the + * loop sends again immediately rather than trickling a long backlog one tick at a time. + */ + suspend fun runOnce(): SyncPass { + val env = environment() + val decision = SyncPolicy.decide( + SyncState( + nowMs = clock(), + pendingRows = source.pendingCount(), + ridingSamples = env.ridingSamples, + enabled = env.enabled, + online = env.online, + wifiOnly = env.wifiOnly, + onWifi = env.onWifi, + credentialReady = env.credentialReady, + onlineBlocked = env.onlineBlocked, + pause = pause, + retryAtMs = retryAtMs, + ), + ) + return when (decision) { + is SyncDecision.Paused -> SyncPass.Paused(decision.reason) + is SyncDecision.Wait -> SyncPass.Waiting(decision.atMs) + SyncDecision.SendNow -> send() + } + } + + private suspend fun send(): SyncPass { + // Captured before the rows are read, not after: an Account reset between the scan and the + // request would otherwise leave a batch of the previous Account's rows looking current, and its + // cursor advance would land on the fresh database. + val generation = source.generation() + val pending = try { + source.pending(MAX_SYNC_BATCH_ROWS) + } catch (e: SyncProtocolException) { + return pauseWith(SyncPauseReason.PROTOCOL, "${e.table.wire}.${e.field}") + } + + return when (val built = SyncBatchBuilder.build(pending, MAX_SYNC_BATCH_ROWS, byteTarget)) { + SyncBatchBuild.Empty -> SyncPass.Idle + is SyncBatchBuild.RowTooLarge -> + pauseWith(SyncPauseReason.ROW_TOO_LARGE, "${built.table.wire}@${built.cursor}") + is SyncBatchBuild.Ready -> deliver(built, generation) + } + } + + private suspend fun deliver(batch: SyncBatchBuild.Ready, generation: Long): SyncPass { + val response = transport.send(batch.body) + // A response that outlived its Account cannot touch the fresh database it would land in. + if (source.generation() != generation) return SyncPass.Idle + + return when (response) { + is SyncResponse.Accepted -> accept(batch, response.body) + SyncResponse.Unauthorized -> pauseWith(SyncPauseReason.AUTHENTICATION, "401") + is SyncResponse.Invalid -> + pauseWith(SyncPauseReason.PROTOCOL, "${response.status}:${response.error}") + SyncResponse.TooLarge -> shrink(batch) + is SyncResponse.RateLimited -> backOff(maxOf(response.retryAfterMs, 0L)) + is SyncResponse.Transient -> backOff(SyncPolicy.nextBackoffMs(backoffMs).also { backoffMs = it }) + } + } + + private suspend fun accept(batch: SyncBatchBuild.Ready, body: String): SyncPass { + val accepted = SyncAccepted.parse(body) + if (accepted == null || !SyncAccepted.matches(batch.counts, accepted)) { + return pauseWith(SyncPauseReason.PROTOCOL, "acceptedMismatch") + } + try { + source.commit(batch.advances) + } catch (e: Exception) { + // The server took the rows but the checkpoint did not land. Backing off re-sends the identical + // batch, which the server upserts idempotently — reporting success here would spin instead, + // because the same rows are still pending. + backoffMs = SyncPolicy.nextBackoffMs(backoffMs) + return backOff(backoffMs) + } + backoffMs = 0 + retryAtMs = 0 + byteTarget = MAX_SYNC_BATCH_BYTES + return SyncPass.Sent(batch.rowCount, morePending = source.pendingCount() > 0) + } + + /** + * `413` narrows the byte target instead of dropping anything. Once the target can no longer hold + * even one row, that row is a permanent local protocol error — it is retained, not skipped. + */ + private suspend fun shrink(batch: SyncBatchBuild.Ready): SyncPass { + val table = batch.counts.keys.first() + val detail = "${table.wire}@${batch.advances.getValue(table)}" + if (batch.rowCount <= 1) return pauseWith(SyncPauseReason.ROW_TOO_LARGE, detail) + // Already as small as a batch gets: halving again would resend the same bytes forever, so the + // disagreement about the wire limit is treated as what it is — permanent, with the rows kept. + if (byteTarget <= MIN_BYTE_TARGET) return pauseWith(SyncPauseReason.ROW_TOO_LARGE, detail) + + byteTarget = maxOf(byteTarget / 2, MIN_BYTE_TARGET) + return SyncPass.Retry + } + + private fun backOff(delayMs: Long): SyncPass { + retryAtMs = clock() + delayMs + return SyncPass.Waiting(retryAtMs) + } + + private suspend fun pauseWith(reason: SyncPauseReason, detail: String): SyncPass { + pause = reason + source.recordPermanentFailure(reason, detail) + return SyncPass.Paused(reason) + } + + private companion object { + /** Below this a batch cannot hold a realistic row, so shrinking further only hides the real fault. */ + const val MIN_BYTE_TARGET = 16 * 1024 + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncJson.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncJson.kt new file mode 100644 index 000000000..61287e62e --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncJson.kt @@ -0,0 +1,135 @@ +package expo.modules.vescapecore.sync + +/** + * A row the server could never store. Permanent for this phone: retrying the same bytes cannot make + * it succeed, so the engine pauses with the row retained rather than skipping it. + * + * The message names the table and the field only — never the value, which may be a coordinate, a + * Rider's text or a token. + * + * @parity /modules/vescape-core/ios/sync/SyncJson.swift `SyncProtocolError` + */ +class SyncProtocolException(val table: SyncTable, val field: String, val problem: String) : + IllegalStateException("${table.wire}.$field $problem") + +/** + * A compact JSON object writer that validates as it writes. + * + * Deliberately not `org.json`: this has to produce the exact bytes measured against the wire byte + * cap, in a stable field order, and run in plain JVM tests where the platform's JSON is a stub. The + * bounds it enforces are the server's own (`vescape-server` `src/sync/protocol.ts`), applied before + * transport so a wedged batch is impossible rather than merely unlikely. + * + * Nullable columns are written as explicit nulls: "cleared" and "not mentioned" are different + * intents, and a missing key cannot express the first. + * + * @parity /modules/vescape-core/ios/sync/SyncJson.swift `SyncRowWriter` + * @parity /modules/vescape-server/src/sync/protocol.ts + */ +class SyncRowWriter(private val table: SyncTable) { + private val out = StringBuilder("{") + + fun build(): String = out.append('}').toString() + + /** An identifier the phone chose: a Board id, a settings key, an event name. Never empty. */ + fun keyText(field: String, value: String): SyncRowWriter = apply { + if (value.isEmpty()) fail(field, "must not be empty") + boundedText(field, value) + } + + fun nullableKeyText(field: String, value: String?): SyncRowWriter = apply { + if (value == null) raw(field, "null") else keyText(field, value) + } + + /** + * A key column the phone derives rather than names, so it may legitimately be empty — a sanitizer + * writes `""` as the device id of a sample captured with no Board connected. + */ + fun derivedKeyText(field: String, value: String?): SyncRowWriter = apply { + if (value == null) raw(field, "null") else boundedText(field, value) + } + + /** Text the server stores opaquely and hands back unchanged. Uncapped, like the server's. */ + fun text(field: String, value: String?): SyncRowWriter = apply { + if (value == null) raw(field, "null") else raw(field, quote(value)) + } + + fun bool(field: String, value: Boolean): SyncRowWriter = raw(field, if (value) "true" else "false") + + /** Epoch ms, or a duration in ms: non-negative and inside the JSON-safe integer range. */ + fun timestamp(field: String, value: Long?): SyncRowWriter = bounded(field, value, 0, SYNC_SAFE_INT_MAX) + + fun int32(field: String, value: Int?): SyncRowWriter = + bounded(field, value?.toLong(), SYNC_INT32_MIN, SYNC_INT32_MAX) + + fun count(field: String, value: Int?): SyncRowWriter = + bounded(field, value?.toLong(), 0, SYNC_INT32_MAX) + + /** A 64-bit column that is not a timestamp — an odometer reading. */ + fun int64(field: String, value: Long?): SyncRowWriter = + bounded(field, value, -SYNC_SAFE_INT_MAX, SYNC_SAFE_INT_MAX) + + /** A real number. Neither infinity nor NaN is expressible in JSON. */ + fun number(field: String, value: Double?): SyncRowWriter = apply { + if (value == null) { + raw(field, "null") + return@apply + } + if (!value.isFinite()) fail(field, "must be finite") + val whole = value.toLong() + raw(field, if (value == whole.toDouble()) whole.toString() else value.toString()) + } + + /** + * A measurement the firmware reported, where non-finite means the reading is unusable rather + * than that the row is malformed. + * + * [number] refuses infinity and NaN, which is right for a value a Rider authored — one there is + * a bug worth stopping for. A decoded Board sample is the opposite case: the app did not choose + * the value, it received it, and one bad float would pause every table's backup on a permanent + * protocol error that no retry can clear. These columns are all nullable precisely because a + * field the firmware did not send is absent, so an unusable one is absent too. + * + * @parity /modules/vescape-core/ios/sync/SyncJson.swift `reading` + */ + fun reading(field: String, value: Double?): SyncRowWriter = + number(field, if (value != null && value.isFinite()) value else null) + + private fun bounded(field: String, value: Long?, min: Long, max: Long): SyncRowWriter = apply { + if (value == null) { + raw(field, "null") + return@apply + } + if (value < min || value > max) fail(field, "is out of bounds") + raw(field, value.toString()) + } + + private fun boundedText(field: String, value: String) { + if (value.length > MAX_SYNC_KEY_LENGTH) fail(field, "exceeds $MAX_SYNC_KEY_LENGTH characters") + raw(field, quote(value)) + } + + private fun raw(field: String, encoded: String): SyncRowWriter = apply { + if (out.length > 1) out.append(',') + out.append(quote(field)).append(':').append(encoded) + } + + private fun fail(field: String, problem: String): Nothing = + throw SyncProtocolException(table, field, problem) + + private fun quote(value: String): String { + val quoted = StringBuilder(value.length + 2).append('"') + for (char in value) { + when { + char == '"' -> quoted.append("\\\"") + char == '\\' -> quoted.append("\\\\") + char == '\n' -> quoted.append("\\n") + char == '\r' -> quoted.append("\\r") + char == '\t' -> quoted.append("\\t") + char < ' ' -> quoted.append("\\u%04x".format(char.code)) + else -> quoted.append(char) + } + } + return quoted.append('"').toString() + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncNotifier.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncNotifier.kt new file mode 100644 index 000000000..a0b348d11 --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncNotifier.kt @@ -0,0 +1,88 @@ +package expo.modules.vescapecore.sync + +import android.app.NotificationChannel +import android.app.NotificationManager +import android.app.PendingIntent +import android.content.Context +import android.content.Intent +import androidx.core.app.NotificationCompat +import expo.modules.vescapecore.R + +/** + * The one notification backup raises: it has stopped, and only the Rider can restart it. + * + * Deliberately narrow — ordinary retries, offline stretches and a metered connection say nothing. + * A [SyncPauseReason] does not resolve on its own, and a backup that has silently stopped for weeks + * is the failure this feature can least afford, so each reason gets one actionable notification and + * is cleared again the moment the pause lifts. + * + * @parity /modules/vescape-core/ios/sync/SyncNotifier.swift + */ +internal class SyncNotifier private constructor(private val context: Context) { + private var channelReady = false + + /** Show the notification for [reason], replacing any previous one, or clear it when null. */ + fun update(reason: SyncPauseReason?) { + val manager = context.getSystemService(NotificationManager::class.java) ?: return + if (reason == null) { + manager.cancel(NOTIFICATION_ID) + return + } + ensureChannel(manager) + manager.notify(NOTIFICATION_ID, build(reason)) + } + + private fun ensureChannel(manager: NotificationManager) { + if (channelReady) return + manager.createNotificationChannel( + NotificationChannel(CHANNEL_ID, "Backup", NotificationManager.IMPORTANCE_DEFAULT).apply { + description = "Tells you when ride backup has stopped and needs you" + }, + ) + channelReady = true + } + + private fun build(reason: SyncPauseReason) = NotificationCompat.Builder(context, CHANNEL_ID) + .setContentTitle("Backup paused") + .setContentText(text(reason)) + .setStyle(NotificationCompat.BigTextStyle().bigText(text(reason))) + .setSmallIcon(R.drawable.ic_vesc_notification) + .setContentIntent(openApp()) + .setCategory(NotificationCompat.CATEGORY_ERROR) + .setAutoCancel(true) + .build() + + private fun openApp(): PendingIntent { + val intent = context.packageManager.getLaunchIntentForPackage(context.packageName) ?: Intent() + return PendingIntent.getActivity( + context, + REQUEST_OPEN, + intent, + PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT, + ) + } + + internal companion object { + private const val CHANNEL_ID = "vescape_backup" + private const val NOTIFICATION_ID = 4271 + private const val REQUEST_OPEN = 1 + + /** + * What the Rider has to do, in the same three shapes the account widget names. + * + * @parity /modules/vescape-core/ios/sync/SyncNotifier.swift `text` + */ + fun text(reason: SyncPauseReason): String = when (reason) { + SyncPauseReason.AUTHENTICATION -> "Sign in again to keep backing up your rides." + SyncPauseReason.PROTOCOL -> "Update Vescape to keep backing up your rides." + SyncPauseReason.ROW_TOO_LARGE -> "Backup hit an error. Check the event log in settings." + } + + @Volatile private var instance: SyncNotifier? = null + + fun get(context: Context): SyncNotifier = + instance ?: synchronized(this) { + instance ?: SyncNotifier(context.applicationContext).also { instance = it } + } + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt new file mode 100644 index 000000000..3d8f4170d --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt @@ -0,0 +1,129 @@ +package expo.modules.vescapecore.sync + +/** How the uploader ran out of road. A paused engine is not woken by ordinary timer kicks. */ +enum class SyncPauseReason(val slug: String) { + /** No Device Token, or the server rejected the one we hold. Sign-in is the only way out. */ + AUTHENTICATION("authentication"), + + /** The server refused this batch on its contents, or answered `2xx` with something unreadable. */ + PROTOCOL("protocol"), + + /** A single row cannot fit inside the wire byte cap. Retained, never skipped. */ + ROW_TOO_LARGE("rowTooLarge"), +} + +/** + * The backup state the Rider is shown. Derived from the same [SyncState] the loop decides on, so the + * status line can never disagree with what the uploader is actually doing. + * + * @parity /modules/vescape-core/ios/sync/SyncPolicy.swift `SyncActivity` + * @parity /modules/vescape-core/src/index.ts `SyncActivity` + */ +enum class SyncActivity(val slug: String) { + /** The master switch is off. Nothing is scanned, sent, retried or reported. */ + DISABLED("disabled"), + + /** No credential: backup has never been turned on, or the Rider signed out. */ + SIGNED_OUT("signedOut"), + UP_TO_DATE("upToDate"), + SYNCING("syncing"), + WAITING_FOR_WIFI("waitingForWifi"), + OFFLINE("offline"), + + /** Stopped on a permanent failure; [SyncStatus.pause] names which one. */ + PAUSED("paused"), +} + +/** What the loop should do next. */ +sealed interface SyncDecision { + /** Send the next batch now. */ + object SendNow : SyncDecision + + /** Nothing to do until [atMs]; the loop re-decides then or when a kick lands. */ + data class Wait(val atMs: Long) : SyncDecision + + /** Stopped until the named condition changes. Timer and connectivity kicks do not bypass it. */ + data class Paused(val reason: SyncPauseReason) : SyncDecision +} + +/** + * Everything the decision depends on, read once by the caller so the decision itself stays pure. + * + * @parity /modules/vescape-core/ios/sync/SyncPolicy.swift `SyncState` + */ +data class SyncState( + val nowMs: Long, + /** Rows waiting across every table. Zero means idle, not finished. */ + val pendingRows: Int, + /** A Board Session is producing samples — Idle Pause halts production without ending the session. */ + val ridingSamples: Boolean, + /** The Rider's master switch. Off means the uploader does nothing at all. */ + val enabled: Boolean, + val online: Boolean, + /** Metered-connection setting; the uploader waits for Wi-Fi rather than failing. */ + val wifiOnly: Boolean, + val onWifi: Boolean, + val credentialReady: Boolean, + /** The App Status gate closed, like every other Online Capability. */ + val onlineBlocked: Boolean, + /** Set by a permanent failure; cleared only by sign-in or an Account reset. */ + val pause: SyncPauseReason?, + /** Backoff or `Retry-After` deadline; before it, nothing is sent. */ + val retryAtMs: Long, +) + +/** + * The one place that turns state into "send, wait, or stopped". + * + * Pure: no database, no clock, no network. The clock is [SyncState.nowMs] and the caller owns it. + * + * @parity /modules/vescape-core/ios/sync/SyncPolicy.swift `SyncPolicy` + */ +object SyncPolicy { + /** Cadence while a ride is producing samples: a crash loses at most this much. */ + const val RIDE_INTERVAL_MS = 30_000L + + /** Cadence when nothing is pending. Cheap, because it is a no-op. */ + const val IDLE_INTERVAL_MS = 5 * 60_000L + + const val BACKOFF_START_MS = 30_000L + const val BACKOFF_MAX_MS = 15 * 60_000L + + fun decide(state: SyncState): SyncDecision { + // The master switch is checked before everything, including a pause: switched off is not a + // broken uploader waiting to be resumed, it is one that is not running. + if (!state.enabled) return SyncDecision.Wait(state.nowMs + IDLE_INTERVAL_MS) + state.pause?.let { return SyncDecision.Paused(it) } + if (!state.credentialReady) return SyncDecision.Paused(SyncPauseReason.AUTHENTICATION) + + val interval = if (state.ridingSamples) RIDE_INTERVAL_MS else IDLE_INTERVAL_MS + if (state.pendingRows <= 0) return SyncDecision.Wait(state.nowMs + interval) + // Offline, metered, or gated: a pause in the loop, never a failure that moves backoff. + if (!state.online || state.onlineBlocked) return SyncDecision.Wait(state.nowMs + interval) + if (state.wifiOnly && !state.onWifi) return SyncDecision.Wait(state.nowMs + interval) + if (state.retryAtMs > state.nowMs) return SyncDecision.Wait(state.retryAtMs) + return SyncDecision.SendNow + } + + /** + * The same state, as the one line the Rider reads. + * + * Signed out wins over the pause it produces: a phone with no credential is not a broken backup, + * it is one that was never turned on. Everything below the pause is ordinary waiting. + */ + fun describe(state: SyncState): SyncActivity = when { + !state.enabled -> SyncActivity.DISABLED + !state.credentialReady -> SyncActivity.SIGNED_OUT + state.pause != null -> SyncActivity.PAUSED + state.pendingRows <= 0 -> SyncActivity.UP_TO_DATE + !state.online || state.onlineBlocked -> SyncActivity.OFFLINE + state.wifiOnly && !state.onWifi -> SyncActivity.WAITING_FOR_WIFI + else -> SyncActivity.SYNCING + } + + /** Next backoff step: doubling from [BACKOFF_START_MS], capped, and reset to 0 on success. */ + fun nextBackoffMs(previousMs: Long): Long = when { + previousMs <= 0L -> BACKOFF_START_MS + else -> minOf(previousMs * 2, BACKOFF_MAX_MS) + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncStore.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncStore.kt new file mode 100644 index 000000000..ca87382b8 --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncStore.kt @@ -0,0 +1,117 @@ +package expo.modules.vescapecore.sync + +import expo.modules.vescapecore.telemetry.TelemetryDao + +/** + * The database side of the uploader: the forward scan, the cursor commit and the failure record. + * + * Encoding happens here rather than in the engine, so the pure batch builder measures the exact + * bytes that will be sent. Rows are read in [SyncTable] order and the scan stops once the row limit + * is reached — a table further down waits for the next batch, which is what keeps parents ahead of + * children. + * + * @parity /modules/vescape-core/ios/sync/SyncStore.swift `SyncStore` + */ +class SyncStore( + /** Resolved per call: an Account reset replaces the whole database under this object. */ + private val database: () -> TelemetryDao, + private val generation: () -> Long, + private val onPermanentFailure: (SyncPauseReason, String) -> Unit, +) : SyncSource { + + override suspend fun pending(rowLimit: Int): List { + val tables = ArrayList(SyncTable.entries.size) + var budget = rowLimit + for (table in SyncTable.entries) { + if (budget <= 0) break + val rows = read(table, database().cursorOf(table.cursorKey), budget) + if (rows.isEmpty()) continue + tables += SyncPendingTable(table, rows) + budget -= rows.size + } + return tables + } + + override suspend fun pendingCount(): Int { + var total = 0 + for (table in SyncTable.entries) { + val cursor = database().cursorOf(table.cursorKey) + total += when (table) { + SyncTable.APP_SETTINGS -> database().countAppSettingsAfter(cursor) + SyncTable.BOARDS -> database().countBoardsAfter(cursor) + SyncTable.BOARD_SETTINGS -> database().countBoardSettingsAfter(cursor) + SyncTable.BOARD_WARNINGS -> database().countBoardWarningsAfter(cursor) + SyncTable.ALERTS -> database().countAlertsAfter(cursor) + SyncTable.TUNE_PROFILES -> database().countTuneProfilesAfter(cursor) + SyncTable.TUNE_HISTORY_ENTRIES -> database().countTuneHistoryEntriesAfter(cursor) + SyncTable.PRIVACY_ZONES -> database().countPrivacyZonesAfter(cursor) + SyncTable.TELEMETRY_MARKERS -> database().countTelemetryMarkersAfter(cursor) + SyncTable.METRIC_EXCLUSION_RANGES -> database().countExclusionRangesAfter(cursor) + SyncTable.DIAGNOSTIC_EVENTS -> database().countDiagnosticEventsAfter(cursor) + SyncTable.TELEMETRY_FRAMES -> database().countTelemetryFramesAfter(cursor) + SyncTable.TELEMETRY_MINUTE_BUCKETS -> database().countMinuteBucketsAfter(cursor) + SyncTable.FAVORITES -> database().countFavoritesAfter(cursor) + SyncTable.VESC_FAULT_OCCURRENCES -> database().countVescFaultOccurrencesAfter(cursor) + SyncTable.VESC_FAULT_CAPTURES -> database().countVescFaultCapturesAfter(cursor) + SyncTable.VESC_FAULT_CAPTURE_SAMPLES -> database().countVescFaultCaptureSamplesAfter(cursor) + SyncTable.DELETE_ACTIONS -> database().countSyncActionsAfter(cursor) + } + } + return total + } + + /** + * Cursors move only here, only after the server accepted, and each in its own statement. The + * accepted Sync Action cursor is also what prunes the log, so pruning can never outrun it. + */ + override suspend fun commit(advances: Map) { + for ((table, cursor) in advances) database().commitSyncCursor(table.cursorKey, cursor) + if (advances.containsKey(SyncTable.DELETE_ACTIONS)) database().pruneUploadedSyncActions() + } + + override fun generation(): Long = generation.invoke() + + override suspend fun recordPermanentFailure(reason: SyncPauseReason, detail: String) { + onPermanentFailure(reason, detail) + } + + private suspend fun read(table: SyncTable, cursor: Long, limit: Int): List = + when (table) { + SyncTable.APP_SETTINGS -> + database().getAppSettingsAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.appSetting(it)) } + SyncTable.BOARDS -> + database().getBoardsAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.board(it)) } + SyncTable.BOARD_SETTINGS -> + database().getBoardSettingsAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.boardSetting(it)) } + SyncTable.BOARD_WARNINGS -> + database().getBoardWarningsAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.boardWarning(it)) } + SyncTable.ALERTS -> + database().getAlertsAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.alert(it)) } + SyncTable.TUNE_PROFILES -> + database().getTuneProfilesAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.tuneProfile(it)) } + SyncTable.TUNE_HISTORY_ENTRIES -> + database().getTuneHistoryEntriesAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.tuneHistoryEntry(it)) } + SyncTable.PRIVACY_ZONES -> + database().getPrivacyZonesAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.privacyZone(it)) } + SyncTable.TELEMETRY_MARKERS -> + database().getTelemetryMarkersAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.telemetryMarker(it)) } + SyncTable.METRIC_EXCLUSION_RANGES -> + database().getExclusionRangesAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.metricExclusionRange(it)) } + SyncTable.DIAGNOSTIC_EVENTS -> + database().getDiagnosticEventsAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.diagnosticEvent(it)) } + SyncTable.TELEMETRY_FRAMES -> + database().getTelemetryFramesAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.telemetryFrame(it)) } + SyncTable.TELEMETRY_MINUTE_BUCKETS -> + database().getMinuteBucketsAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.telemetryMinuteBucket(it)) } + SyncTable.FAVORITES -> + database().getFavoritesAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.favorite(it)) } + SyncTable.VESC_FAULT_OCCURRENCES -> + database().getVescFaultOccurrencesAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.vescFaultOccurrence(it)) } + SyncTable.VESC_FAULT_CAPTURES -> + database().getVescFaultCapturesAfter(cursor, limit).map { SyncPendingRow(it.syncSeq, SyncWire.vescFaultCapture(it)) } + SyncTable.VESC_FAULT_CAPTURE_SAMPLES -> + database().getVescFaultCaptureSamplesAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.vescFaultCaptureSample(it)) } + SyncTable.DELETE_ACTIONS -> + database().getSyncActionsAfter(cursor, limit).map { SyncPendingRow(it.id, SyncWire.deleteAction(it)) } + } +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt new file mode 100644 index 000000000..e24adc937 --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt @@ -0,0 +1,85 @@ +package expo.modules.vescapecore.sync + +/** + * Every table a Sync Batch can carry, in the order the server writes them: a Board-owned row + * references its Board, so a batch carrying both has to put the Board first or the foreign key + * refuses the whole batch. Delete Actions come last, so an action is judged against the Change + * Timestamp the same batch just wrote. + * + * The batch builder walks this order and nothing else — never the size of a table's backlog, which + * would produce a batch the server cannot apply. + * + * [cursorColumn] is what the scan runs on: an `AUTOINCREMENT` key for append-only tables, `sync_seq` + * for mutable ones. Both are device-local counters that never cross the wire. + * + * @parity /modules/vescape-core/ios/sync/SyncTables.swift `SyncTable` + */ +enum class SyncTable(val wire: String, val table: String, val cursorColumn: String) { + APP_SETTINGS("appSettings", "app_settings", SYNC_SEQ_COLUMN), + BOARDS("boards", "boards", SYNC_SEQ_COLUMN), + BOARD_SETTINGS("boardSettings", "board_settings", SYNC_SEQ_COLUMN), + BOARD_WARNINGS("boardWarnings", "board_warnings", SYNC_SEQ_COLUMN), + ALERTS("alerts", "alerts", SYNC_SEQ_COLUMN), + TUNE_PROFILES("tuneProfiles", "tune_profiles", SYNC_SEQ_COLUMN), + TUNE_HISTORY_ENTRIES("tuneHistoryEntries", "tune_history_entries", ROW_ID_COLUMN), + PRIVACY_ZONES("privacyZones", "privacy_zones", SYNC_SEQ_COLUMN), + TELEMETRY_MARKERS("telemetryMarkers", "telemetry_markers", ROW_ID_COLUMN), + METRIC_EXCLUSION_RANGES("metricExclusionRanges", "metric_exclusion_ranges", ROW_ID_COLUMN), + DIAGNOSTIC_EVENTS("diagnosticEvents", "diagnostic_events", ROW_ID_COLUMN), + TELEMETRY_FRAMES("telemetryFrames", "telemetry_frames", ROW_ID_COLUMN), + TELEMETRY_MINUTE_BUCKETS("telemetryMinuteBuckets", "telemetry_minute_buckets", SYNC_SEQ_COLUMN), + FAVORITES("favorites", "favorites", SYNC_SEQ_COLUMN), + + // Board-owned, so after `boards`; the Capture and its samples reference the Occurrence, so after + // it in turn. This chain is the one place the ordering rule bites twice inside one batch. + VESC_FAULT_OCCURRENCES("vescFaultOccurrences", "vesc_fault_occurrences", SYNC_SEQ_COLUMN), + VESC_FAULT_CAPTURES("vescFaultCaptures", "vesc_fault_captures", SYNC_SEQ_COLUMN), + VESC_FAULT_CAPTURE_SAMPLES("vescFaultCaptureSamples", "vesc_fault_capture_samples", ROW_ID_COLUMN), + DELETE_ACTIONS("deleteActions", "sync_actions", ROW_ID_COLUMN), + ; + + /** + * `sync_sequences` key holding how far this table has been accepted. Distinct from the write + * counters keyed on the bare table name, which hand out `sync_seq` positions. + * + * Sync Actions keep the key #282 already shipped, so the log's prune keeps reading the same row + * the uploader commits. + */ + val cursorKey: String + get() = if (this == DELETE_ACTIONS) { + expo.modules.vescapecore.telemetry.SYNC_ACTIONS_UPLOADED_CURSOR + } else { + "$SYNC_CURSOR_PREFIX$table" + } +} + +internal const val SYNC_SEQ_COLUMN = "sync_seq" +internal const val ROW_ID_COLUMN = "id" +internal const val SYNC_CURSOR_PREFIX = "sync_cursor_" + +/** + * Rows accepted in one Sync Batch, total across every table. + * @parity /modules/vescape-core/ios/sync/SyncTables.swift `maxSyncBatchRows` + */ +const val MAX_SYNC_BATCH_ROWS = 1_000 + +/** + * Actual compact UTF-8 JSON bytes accepted by `POST /api/sync`. Measured on the encoded request, not + * estimated from object sizes — the server refuses on the byte count it actually receives. + * @parity /modules/vescape-core/ios/sync/SyncTables.swift `maxSyncBatchBytes` + */ +const val MAX_SYNC_BATCH_BYTES = 1024 * 1024 + +/** + * Longest text one column of a server key may hold. Mirrored from the server so a row that cannot be + * stored is refused here instead of wedging a batch. + * @parity /modules/vescape-core/ios/sync/SyncTables.swift `maxSyncKeyLength` + */ +const val MAX_SYNC_KEY_LENGTH = 128 + +/** Bounds of the Postgres `integer` columns the app's 32-bit values land in. */ +internal const val SYNC_INT32_MIN = -2_147_483_648L +internal const val SYNC_INT32_MAX = 2_147_483_647L + +/** `Number.MAX_SAFE_INTEGER`: past it `JSON.parse` rounds, so neither side could agree on the value. */ +internal const val SYNC_SAFE_INT_MAX = 9_007_199_254_740_991L diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncWire.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncWire.kt new file mode 100644 index 000000000..c9ddc882d --- /dev/null +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncWire.kt @@ -0,0 +1,335 @@ +package expo.modules.vescapecore.sync + +import expo.modules.vescapecore.telemetry.AlertRuleEntity +import expo.modules.vescapecore.telemetry.AppSettingEntity +import expo.modules.vescapecore.telemetry.BoardEntity +import expo.modules.vescapecore.telemetry.BoardSettingEntity +import expo.modules.vescapecore.telemetry.BoardWarningEntity +import expo.modules.vescapecore.telemetry.DiagnosticEventEntity +import expo.modules.vescapecore.telemetry.FavoriteEntity +import expo.modules.vescapecore.telemetry.MetricExclusionRangeEntity +import expo.modules.vescapecore.telemetry.PrivacyZoneEntity +import expo.modules.vescapecore.telemetry.SyncActionEntity +import expo.modules.vescapecore.telemetry.TelemetryFrameEntity +import expo.modules.vescapecore.telemetry.TelemetryMarkerEntity +import expo.modules.vescapecore.telemetry.TelemetryMinuteBucketEntity +import expo.modules.vescapecore.telemetry.TuneHistoryEntryEntity +import expo.modules.vescapecore.telemetry.TuneProfileEntity +import expo.modules.vescapecore.telemetry.VescFaultCaptureEntity +import expo.modules.vescapecore.telemetry.VescFaultCaptureSampleEntity +import expo.modules.vescapecore.telemetry.VescFaultOccurrenceEntity + +/** + * Local rows as the server reads them. + * + * Every encoder is strongly typed and validates before transport, so a batch is refused here — with + * the row retained and one metadata-only Diagnostic Event — rather than wedging against the server. + * The field sets mirror `vescape-server` `src/sync/protocol.ts`; a column the server does not declare + * is not sent, because an unknown field rejects the whole batch. + * + * @parity /modules/vescape-core/ios/sync/SyncWire.swift + * @parity /modules/vescape-server/src/sync/protocol.ts + */ +object SyncWire { + fun appSetting(row: AppSettingEntity): String = SyncRowWriter(SyncTable.APP_SETTINGS) + .keyText("key", row.key) + .text("valueJson", row.valueJson) + .timestamp("updatedAt", row.updatedAt) + .build() + + /** + * `transport` is iOS-only; Android keeps it in board settings and sends null, exactly as the + * server's own comment describes. + */ + fun board(row: BoardEntity): String = SyncRowWriter(SyncTable.BOARDS) + .keyText("id", row.id) + .text("name", row.name) + .text("bleId", row.bleId) + .text("transport", null) + .timestamp("createdAt", row.createdAt) + .timestamp("updatedAt", row.updatedAt) + .build() + + fun boardSetting(row: BoardSettingEntity): String = SyncRowWriter(SyncTable.BOARD_SETTINGS) + .keyText("boardId", row.boardId) + .keyText("key", row.key) + .text("valueJson", row.valueJson) + .timestamp("updatedAt", row.updatedAt) + .build() + + fun boardWarning(row: BoardWarningEntity): String = SyncRowWriter(SyncTable.BOARD_WARNINGS) + .keyText("boardId", row.boardId) + .keyText("kind", row.kind) + .text("severity", row.severity) + .timestamp("firstDetectedAt", row.firstDetectedAt) + .timestamp("lastDetectedAt", row.lastDetectedAt) + .text("payloadJson", row.payloadJson) + .timestamp("updatedAt", row.updatedAt) + .build() + + /** + * The threshold numbers travel with the kind that says how to read them: a config-relative rule + * restored as a bare number looks configured and fires at the wrong point, which is worse than + * losing it. + */ + fun alert(row: AlertRuleEntity): String = SyncRowWriter(SyncTable.ALERTS) + .keyText("boardId", row.boardId) + .keyText("id", row.id) + .keyText("controlId", row.controlId) + .number("threshold", row.threshold) + .number("thresholdMax", row.thresholdMax) + .keyText("thresholdKind", row.thresholdKind) + .nullableKeyText("configFieldId", row.configFieldId) + .number("thresholdOffset", row.thresholdOffset) + .number("thresholdMaxOffset", row.thresholdMaxOffset) + .bool("enabled", row.enabled) + .text("soundType", row.soundType) + .timestamp("repeatEverySeconds", row.repeatEverySeconds) + .count("beepCount", row.beepCount) + .text("source", row.source) + .timestamp("createdAt", row.createdAt) + .timestamp("updatedAt", row.updatedAt) + .build() + + fun tuneProfile(row: TuneProfileEntity): String = SyncRowWriter(SyncTable.TUNE_PROFILES) + .keyText("id", row.id) + .keyText("boardId", row.boardId) + // May legitimately be empty: the app defaults an unknown Refloat package version to `''`. + .derivedKeyText("refloatBaseVersion", row.refloatBaseVersion) + .text("name", row.name) + .text("icon", row.icon) + .text("color", row.color) + .text("fieldsJson", row.fieldsJson) + .timestamp("createdAt", row.createdAt) + .timestamp("updatedAt", row.updatedAt) + .build() + + /** Carries no id: the local one restarts on a fresh install, so identity is `(profileId, createdAt)`. */ + fun tuneHistoryEntry(row: TuneHistoryEntryEntity): String = + SyncRowWriter(SyncTable.TUNE_HISTORY_ENTRIES) + .keyText("profileId", row.profileId) + .text("fieldsJson", row.fieldsJson) + .timestamp("createdAt", row.createdAt) + .build() + + fun privacyZone(row: PrivacyZoneEntity): String = SyncRowWriter(SyncTable.PRIVACY_ZONES) + .keyText("id", row.id) + .text("preset", row.preset) + .text("name", row.name) + .bool("enabled", row.enabled) + .int32("centerLatitudeE7", row.centerLatitudeE7) + .int32("centerLongitudeE7", row.centerLongitudeE7) + .int32("radiusMeters", row.radiusMeters) + .timestamp("createdAt", row.createdAt) + .timestamp("updatedAt", row.updatedAt) + .build() + + fun telemetryMarker(row: TelemetryMarkerEntity): String = SyncRowWriter(SyncTable.TELEMETRY_MARKERS) + .timestamp("occurredAtMs", row.occurredAtMs) + .timestamp("elapsedRealtimeMs", row.elapsedRealtimeMs) + .keyText("type", row.type) + .derivedKeyText("boardId", row.boardId) + .text("message", row.message) + .timestamp("gapMs", row.gapMs) + .build() + + fun metricExclusionRange(row: MetricExclusionRangeEntity): String = + SyncRowWriter(SyncTable.METRIC_EXCLUSION_RANGES) + .derivedKeyText("boardId", row.boardId) + .text("reason", row.reason) + .timestamp("startMs", row.startMs) + .timestamp("endMs", row.endMs) + .count("sampleCount", row.sampleCount) + .build() + + fun diagnosticEvent(row: DiagnosticEventEntity): String = SyncRowWriter(SyncTable.DIAGNOSTIC_EVENTS) + .timestamp("occurredAtMs", row.occurredAtMs) + .timestamp("elapsedRealtimeMs", row.elapsedRealtimeMs) + .keyText("eventName", row.eventName) + .derivedKeyText("operation", row.operation) + .derivedKeyText("phase", row.phase) + .derivedKeyText("boardId", row.boardId) + .text("message", row.message) + .text("propertiesJson", row.propertiesJson) + .build() + + /** + * A Telemetry Sample as recorded: still delta-encoded, carrying the Changed Masks. The local row + * id and the per-row device columns never cross the wire — the Board reference replaces them + * (ADR-0028) and a restored phone's full re-upload has to be an idempotent no-op. + * + * A frame that names no Board cannot be encoded; [SyncSource] never offers one. + */ + fun telemetryFrame(row: TelemetryFrameEntity): String = SyncRowWriter(SyncTable.TELEMETRY_FRAMES) + .keyText( + "boardId", + row.boardId + ?: throw SyncProtocolException(SyncTable.TELEMETRY_FRAMES, "boardId", "must name a Board"), + ) + .timestamp("capturedAtMs", row.capturedAtMs) + .timestamp("elapsedRealtimeMs", row.elapsedRealtimeMs) + .int32("canId", row.canId) + .count("flags", row.flags) + .count("changedMask1", row.changedMask1) + .count("changedMask2", row.changedMask2) + .int32("speedCentiKmh", row.speedCentiKmh) + .int32("batteryVoltageMv", row.batteryVoltageMv) + .int32("motorCurrentMa", row.motorCurrentMa) + .int32("batteryCurrentMa", row.batteryCurrentMa) + .int32("dutyPermille", row.dutyPermille) + .int32("pitchCentiDeg", row.pitchCentiDeg) + .int32("rollCentiDeg", row.rollCentiDeg) + .int32("balancePitchCentiDeg", row.balancePitchCentiDeg) + .int32("balanceCurrentMa", row.balanceCurrentMa) + .int32("erpm", row.erpm) + .int32("state", row.state) + .int32("switchState", row.switchState) + .int32("adc1Milli", row.adc1Milli) + .int32("adc2Milli", row.adc2Milli) + .int64("odometerCm", row.odometerCm) + .int32("tempMosfetDeciC", row.tempMosfetDeciC) + .int32("tempMotorDeciC", row.tempMotorDeciC) + .int32("latitudeE7", row.latitudeE7) + .int32("longitudeE7", row.longitudeE7) + .int32("gpsSpeedCentiMps", row.gpsSpeedCentiMps) + .int32("bearingCentiDeg", row.bearingCentiDeg) + .int32("accuracyCm", row.accuracyCm) + .int32("altitudeCm", row.altitudeCm) + .timestamp("locationTimestampMs", row.locationTimestampMs) + .build() + + fun telemetryMinuteBucket(row: TelemetryMinuteBucketEntity): String = + SyncRowWriter(SyncTable.TELEMETRY_MINUTE_BUCKETS) + .keyText("boardId", row.boardId) + .timestamp("bucketStartMs", row.bucketStartMs) + .timestamp("updatedAt", row.updatedAt) + .count("sampleCount", row.sampleCount) + .timestamp("firstSampleAtMs", row.firstSampleAtMs) + .timestamp("lastSampleAtMs", row.lastSampleAtMs) + .int64("sumAbsSpeedCentiKmh", row.sumAbsSpeedCentiKmh) + .count("movingSpeedSampleCount", row.movingSpeedSampleCount) + .int64("sumMovingAbsSpeedCentiKmh", row.sumMovingAbsSpeedCentiKmh) + .int32("maxAbsSpeedCentiKmh", row.maxAbsSpeedCentiKmh) + .int32("minBatteryVoltageMv", row.minBatteryVoltageMv) + .int32("maxMotorCurrentAbsMa", row.maxMotorCurrentAbsMa) + .int32("maxBatteryCurrentAbsMa", row.maxBatteryCurrentAbsMa) + .int64("batteryUsedWhMilli", row.batteryUsedWhMilli) + .int64("batteryRegenWhMilli", row.batteryRegenWhMilli) + .int32("maxDutyAbsPermille", row.maxDutyAbsPermille) + .int64("firstOdometerCm", row.firstOdometerCm) + .int64("lastOdometerCm", row.lastOdometerCm) + .count("gpsPointCount", row.gpsPointCount) + .count("preciseGpsPointCount", row.preciseGpsPointCount) + .int64("gpsDistanceCm", row.gpsDistanceCm) + .int32("maxGpsSpeedCentiMps", row.maxGpsSpeedCentiMps) + .int32("maxTempMosfetDeciC", row.maxTempMosfetDeciC) + .int32("maxTempMotorDeciC", row.maxTempMotorDeciC) + .int32("firstLatitudeE7", row.firstLatitudeE7) + .int32("firstLongitudeE7", row.firstLongitudeE7) + .timestamp("firstMovingAtMs", row.firstMovingAtMs) + .timestamp("lastMovingAtMs", row.lastMovingAtMs) + .build() + + /** The Board name is resolved on read rather than snapshotted, so none crosses the wire. */ + fun favorite(row: FavoriteEntity): String = SyncRowWriter(SyncTable.FAVORITES) + .keyText("id", row.id) + .nullableKeyText("boardId", row.boardId) + .text("name", row.name) + .timestamp("startMs", row.startMs) + .timestamp("endMs", row.endMs) + .timestamp("createdAt", row.createdAt) + .timestamp("updatedAt", row.updatedAt) + .count("sampleCount", row.sampleCount) + .count("gpsPointCount", row.gpsPointCount) + .int64("distanceCm", row.distanceCm) + .timestamp("movingDurationMs", row.movingDurationMs) + .int32("avgSpeedCentiKmh", row.avgSpeedCentiKmh) + .int32("maxSpeedCentiKmh", row.maxSpeedCentiKmh) + .int64("batteryUsedWhMilli", row.batteryUsedWhMilli) + .build() + + /** + * One VESC Fault Occurrence: firmware-authored live evidence, keyed on the app's own minted id + * because the same code activating twice is two occurrences, never one row keyed on the code. + * + * `updatedAt` is carried in its own right rather than derived from `lastObservedAtMs`: a Rider + * dismissing an occurrence changes the row without the fault being observed again. + */ + fun vescFaultOccurrence(row: VescFaultOccurrenceEntity): String = + SyncRowWriter(SyncTable.VESC_FAULT_OCCURRENCES) + .keyText("id", row.id) + .keyText("boardId", row.boardId) + .int32("code", row.code) + .timestamp("occurredAtMs", row.occurredAtMs) + .timestamp("lastObservedAtMs", row.lastObservedAtMs) + .timestamp("clearedAtMs", row.clearedAtMs) + .bool("dismissed", row.dismissed) + .timestamp("updatedAt", row.updatedAt) + .build() + + /** + * One VESC Fault Capture. Keyed by the Occurrence — one Occurrence has at most one Capture — so + * the parent reference and the identity are the same field. Immutable, so it carries no change + * timestamp and a re-send is a no-op rather than an upsert. + */ + fun vescFaultCapture(row: VescFaultCaptureEntity): String = + SyncRowWriter(SyncTable.VESC_FAULT_CAPTURES) + .keyText("occurrenceId", row.occurrenceId) + .keyText("boardId", row.boardId) + .timestamp("startedAtMs", row.startedAtMs) + .timestamp("openedAtMs", row.openedAtMs) + .count("sampleCount", row.sampleCount) + .build() + + /** + * One decoded sample inside a Capture, identified by its Occurrence and its capture time. The + * local autoincrement id never crosses the wire — it restarts on a fresh install, exactly as a + * Tune History entry's does. + * + * Every value is nullable: a sample carries what that Board Session actually reported, and a + * field the firmware did not send is absent rather than zero. + */ + fun vescFaultCaptureSample(row: VescFaultCaptureSampleEntity): String = + SyncRowWriter(SyncTable.VESC_FAULT_CAPTURE_SAMPLES) + .keyText("occurrenceId", row.occurrenceId) + .timestamp("capturedAtMs", row.capturedAtMs) + .reading("speed", row.speed) + .reading("dutyCycle", row.dutyCycle) + .reading("erpm", row.erpm) + .reading("batteryVoltage", row.batteryVoltage) + .reading("batteryCurrent", row.batteryCurrent) + .reading("motorCurrent", row.motorCurrent) + .reading("tempMosfet", row.tempMosfet) + .reading("tempMotor", row.tempMotor) + .reading("pitch", row.pitch) + .reading("roll", row.roll) + .reading("balancePitch", row.balancePitch) + .reading("adc1", row.adc1) + .reading("adc2", row.adc2) + .int32("state", row.state) + .build() + + /** + * One Sync Action, flat: the target, the identity within that target's scope, and when the Rider + * removed it. The log's own `board_id`/`key` pair expands into the identity fields the server + * declares for that target, so an action reads like the row it names. + */ + fun deleteAction(row: SyncActionEntity): String { + val writer = SyncRowWriter(SyncTable.DELETE_ACTIONS).keyText("target", row.target) + when (row.target) { + "appSetting" -> writer.keyText("key", row.key) + "board" -> writer.keyText("id", row.id()) + "boardSetting" -> writer.keyText("boardId", row.board()).keyText("key", row.key) + "boardWarning" -> writer.keyText("boardId", row.board()).keyText("kind", row.key) + "alert" -> writer.keyText("boardId", row.board()).keyText("id", row.key) + "tuneProfile", "privacyZone", "favorite" -> writer.keyText("id", row.key) + else -> throw SyncProtocolException(SyncTable.DELETE_ACTIONS, "target", "is not a known target") + } + return writer.timestamp("deletedAt", row.deletedAt).build() + } + + private fun SyncActionEntity.id(): String = key + + private fun SyncActionEntity.board(): String = boardId + ?: throw SyncProtocolException(SyncTable.DELETE_ACTIONS, "boardId", "is missing for $target") +} diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/AppDataRepository.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/AppDataRepository.kt index 83da5f063..3f16930db 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/AppDataRepository.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/AppDataRepository.kt @@ -10,6 +10,7 @@ import expo.modules.vescapecore.diagnostics.DiagnosticReporter import expo.modules.vescapecore.service.CoreForegroundService import expo.modules.vescapecore.connection.BoardTransport +import expo.modules.vescapecore.sync.SyncCoordinator import android.content.Context import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.withContext @@ -212,6 +213,10 @@ class AppDataRepository private constructor(private val context: Context) { dao.deleteBoardWithSettings(id, System.currentTimeMillis()) dao.deleteBoardConfigValues(id) dao.deleteBoardConfigChangeNotice(id) + // Motor Config Values are keyed by Board too, and were the one decode cache this path forgot. + // Nothing reads them once the Board is a tombstone, and the next link re-reads them from the + // controller anyway, so leaving them behind only accumulated rows no one could reach. + dao.deleteMotorConfigValues(id) notifyDataChanged(AppDataScope.BOARDS) } @@ -402,7 +407,7 @@ class AppDataRepository private constructor(private val context: Context) { suspend fun setAlertRuleEnabled(boardId: String, id: String, enabled: Boolean): Unit = withContext(Dispatchers.IO) { - dao.setAlertRuleEnabled(boardId, id, enabled) + dao.setAlertRuleEnabled(boardId, id, enabled, System.currentTimeMillis()) } suspend fun deleteAlertRule(boardId: String, id: String): Unit = withContext(Dispatchers.IO) { @@ -473,6 +478,9 @@ class AppDataRepository private constructor(private val context: Context) { DEFAULT_RIDE_SPLIT_GAP_MINUTES, ::validRideSplitGapMinutes, ), + syncEnabled = req("syncEnabled", false) { it as? Boolean }, + syncWifiOnly = req("syncWifiOnly", false) { it as? Boolean }, + syncBackupChoiceMade = req("syncBackupChoiceMade", false) { it as? Boolean }, riderId = opt("riderId") { it as? String }, riderName = opt("riderName") { it as? String }, riderColor = opt("riderColor") { it as? String }, @@ -552,6 +560,8 @@ class AppDataRepository private constructor(private val context: Context) { validAutoCloseDelayMinutes(value) ?: return@withContext "rideSplitGapMinutes" -> validRideSplitGapMinutes(value) ?: return@withContext + "syncEnabled", "syncWifiOnly", "syncBackupChoiceMade" -> + value as? Boolean ?: return@withContext "riderId", "riderName", "riderColor" -> value as? String // Legal Policy is native-owned. JS can request refresh through the dedicated intent. "legalPolicy" -> return@withContext @@ -602,6 +612,9 @@ class AppDataRepository private constructor(private val context: Context) { "autoCloseEnabled" -> d.autoCloseEnabled "autoCloseDelayMinutes" -> d.autoCloseDelayMinutes "rideSplitGapMinutes" -> d.rideSplitGapMinutes + "syncEnabled" -> d.syncEnabled + "syncWifiOnly" -> d.syncWifiOnly + "syncBackupChoiceMade" -> d.syncBackupChoiceMade "riderId" -> d.riderId "riderName" -> d.riderName "riderColor" -> d.riderColor @@ -621,6 +634,12 @@ class AppDataRepository private constructor(private val context: Context) { ), ) } + // The uploader reads the Wi-Fi switch from native truth, not from a JS call, so a write from any + // source — the settings row, the one-time choice, a restored backup — reaches it the same way. + when (normalizedKey) { + "syncEnabled" -> SyncCoordinator.get(context).setEnabled(coerced as? Boolean ?: false) + "syncWifiOnly" -> SyncCoordinator.get(context).setWifiOnly(coerced as? Boolean ?: false) + } notifyDataChanged(AppDataScope.SETTINGS) } @@ -945,6 +964,7 @@ fun BoardEntity.toMap(settings: List): Map { "matchBoardConfig" to values["matchBoardConfig"], "legalMode" to (values["legalMode"] ?: mapOf("enabled" to false)), "link" to link, + "updatedAt" to updatedAt, "deletedAt" to deletedAt, ) } @@ -985,6 +1005,9 @@ fun AppSettings.toMap(): Map = mapOf( "autoCloseEnabled" to autoCloseEnabled, "autoCloseDelayMinutes" to autoCloseDelayMinutes, "rideSplitGapMinutes" to rideSplitGapMinutes, + "syncEnabled" to syncEnabled, + "syncWifiOnly" to syncWifiOnly, + "syncBackupChoiceMade" to syncBackupChoiceMade, "riderId" to riderId, "riderName" to riderName, "riderColor" to riderColor, @@ -1030,6 +1053,7 @@ fun AlertRuleEntity.toMap(): Map = mapOf( "repeatEverySeconds" to repeatEverySeconds, "beepCount" to beepCount, "source" to source, + "updatedAt" to updatedAt, ) fun TuneProfileEntity.toMap(): Map = mapOf( @@ -1182,11 +1206,17 @@ private fun Map.normalizedBoardLink(): Map? { ) } -internal fun Map.toBoardEntity(): BoardEntity = BoardEntity( +/** + * Native stamps [BoardEntity.updatedAt] itself rather than trusting the bridge value: it is a sync + * cursor, so it must come from the device clock that already writes `created_at` and must move on + * every upsert, including partial edits that leave `createdAt` untouched. + */ +internal fun Map.toBoardEntity(now: Long = System.currentTimeMillis()): BoardEntity = BoardEntity( id = getString("id"), name = getString("name"), bleId = normalizedBoardLink()?.get("bleId") as? String, createdAt = getLong("createdAt"), + updatedAt = now, ) internal fun Map.toBoardSettingEntities(boardId: String): Pair, List> { @@ -1336,7 +1366,10 @@ private fun parseLegacyMapString(value: String): Map? { }.toMap() } -private fun Map.toAlertRuleEntity(): AlertRuleEntity = AlertRuleEntity( +/** Native stamps [AlertRuleEntity.updatedAt]; see [toBoardEntity] for why the bridge value is ignored. */ +internal fun Map.toAlertRuleEntity( + now: Long = System.currentTimeMillis(), +): AlertRuleEntity = AlertRuleEntity( boardId = getString("boardId"), id = getString("id"), controlId = getString("controlId"), @@ -1352,6 +1385,7 @@ private fun Map.toAlertRuleEntity(): AlertRuleEntity = AlertRuleEn repeatEverySeconds = normalizedAlertRepeatSeconds(getDoubleOrNull("repeatEverySeconds")), beepCount = normalizedAlertBeepCount((get("beepCount") as? Number)?.toInt()), source = get("source") as? String, + updatedAt = now, ) private fun Map.getString(key: String): String = diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/DatabaseBackupManager.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/DatabaseBackupManager.kt index ff3ca8adb..eff53de87 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/DatabaseBackupManager.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/DatabaseBackupManager.kt @@ -95,6 +95,30 @@ object DatabaseBackupManager { } } + /** + * Replace the app-data database with an empty one, taking the Sync Cursors, the pending Sync + * Actions and the Account binding with it (#284). + * + * Deleting the file rather than clearing tables is what makes the Account change safe: nothing can + * survive with a cursor position or a binding that belonged to the previous Account. The wipe is + * local maintenance and emits no Sync Actions to either Account — the log is part of what goes. + * + * @parity /modules/vescape-core/ios/telemetry/DatabaseBackupManager.swift `replaceWithFreshDatabase` + */ + fun replaceWithFreshDatabase(context: Context) { + val appContext = context.applicationContext + resetRepositoriesAndCloseDatabase() + + val dbFile = appContext.getDatabasePath(TELEMETRY_DATABASE_NAME) + // Checked rather than best-effort: a delete that quietly failed would reopen the previous + // Account's database, which the caller is about to hand a different Account's Device Token. + check(!dbFile.exists() || dbFile.delete()) { "Could not remove the existing database" } + sidecarFiles(dbFile).forEach { it.delete() } + + // Opening rebuilds the schema from the entities, so the new database starts unbound. + TelemetryDatabase.get(appContext).openHelper.readableDatabase.query("SELECT 1").close() + } + private fun extractBackup(context: Context, uriString: String, restoredDb: File): JSONObject { var manifest: JSONObject? = null val uri = Uri.parse(uriString) diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryBucketBuilder.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryBucketBuilder.kt index 558129a1f..abf2c1c63 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryBucketBuilder.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryBucketBuilder.kt @@ -168,7 +168,14 @@ private class MutableBucket( } } - fun toEntity(): TelemetryMinuteBucketEntity = TelemetryMinuteBucketEntity( + /** + * [now] is the incremental-sync cursor stamped on the row, so every append or rebuild that + * reaches the database is visible to cursor sync. + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDao.swift `upsertBucket` + */ + fun toEntity(now: Long = System.currentTimeMillis()): TelemetryMinuteBucketEntity = TelemetryMinuteBucketEntity( + updatedAt = now, bucketStartMs = bucketStartMs, boardId = boardId, sampleCount = sampleCount, diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt index 23fd02440..3b5fa5c3b 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt @@ -61,13 +61,44 @@ interface TelemetryDao { suspend fun getEnabledPrivacyZones(): List @Insert(onConflict = OnConflictStrategy.REPLACE) - suspend fun upsertPrivacyZone(zone: PrivacyZoneEntity) + suspend fun insertPrivacyZoneRow(zone: PrivacyZoneEntity) - @Query("UPDATE privacy_zones SET enabled = :enabled, updated_at = :updatedAt WHERE id = :id") - suspend fun setPrivacyZoneEnabled(id: String, enabled: Boolean, updatedAt: Long) + @Query("SELECT updated_at FROM privacy_zones WHERE id = :id") + suspend fun getPrivacyZoneUpdatedAt(id: String): Long? + + /** Stamps both sync columns; see [upsertBoard]. */ + @Transaction + suspend fun upsertPrivacyZone(zone: PrivacyZoneEntity) { + insertPrivacyZoneRow( + zone.copy( + updatedAt = ratchetUpdatedAt(getPrivacyZoneUpdatedAt(zone.id), zone.updatedAt), + syncSeq = nextSyncSeq(SYNC_SEQ_PRIVACY_ZONES), + ), + ) + } + + /** Targeted toggle that bypasses the upsert, so it moves both columns itself; see + * [setAlertRuleEnabledRow]. */ + @Query( + "UPDATE privacy_zones SET enabled = :enabled, updated_at = MAX(updated_at + 1, :updatedAt), " + + "sync_seq = :syncSeq WHERE id = :id", + ) + suspend fun setPrivacyZoneEnabledRow(id: String, enabled: Boolean, updatedAt: Long, syncSeq: Long) + + @Transaction + suspend fun setPrivacyZoneEnabled(id: String, enabled: Boolean, updatedAt: Long) { + setPrivacyZoneEnabledRow(id, enabled, updatedAt, nextSyncSeq(SYNC_SEQ_PRIVACY_ZONES)) + } @Query("DELETE FROM privacy_zones WHERE id = :id") - suspend fun deletePrivacyZone(id: String) + suspend fun deletePrivacyZoneRow(id: String) + + /** Semantic removal: the Rider deleted the zone, so the server has to lose it too. */ + @Transaction + suspend fun deletePrivacyZone(id: String) { + appendDeleteAction(DeleteTarget.PRIVACY_ZONE, null, id, getPrivacyZoneUpdatedAt(id)) + deletePrivacyZoneRow(id) + } @Insert @@ -94,15 +125,324 @@ interface TelemetryDao { @Transaction suspend fun upsertBuckets(buckets: Collection) { for (bucket in buckets) { - val existing = getBucket(bucket.bucketStartMs, bucket.boardId) + // A merge rewrites a row the scan may already have passed, so the seq moves on both branches. + val next = bucket.copy(syncSeq = nextSyncSeq(SYNC_SEQ_MINUTE_BUCKETS)) + val existing = getBucket(next.bucketStartMs, next.boardId) if (existing == null) { - insertBucket(bucket) + insertBucket(next) } else { - updateBucket(existing.merge(bucket)) + updateBucket(existing.merge(next)) } } } + @Query("INSERT OR IGNORE INTO sync_sequences (name, last_value) VALUES (:name, 0)") + suspend fun seedSyncSequence(name: String) + + @Query("UPDATE sync_sequences SET last_value = last_value + 1 WHERE name = :name") + suspend fun bumpSyncSequence(name: String) + + @Query("SELECT last_value FROM sync_sequences WHERE name = :name") + suspend fun getSyncSequence(name: String): Long? + + /** + * Hands out the next Sync Cursor position for [name]. Bump-then-read rather than read-then-bump so + * two writes racing inside the same database can never be handed the same number; both statements + * run in the caller's transaction. + * + * Seeds the row first because a fresh install builds the schema from the entities and never runs + * the migration that inserts it. + */ + @Transaction + suspend fun nextSyncSeq(name: String): Long { + seedSyncSequence(name) + bumpSyncSequence(name) + return getSyncSequence(name) ?: 0L + } + + // Sync Actions — the append-only log of semantic removals (#282). Every write below runs inside + // the caller's transaction, so an action and the delete it describes commit together or not at all. + // @parity /modules/vescape-core/ios/telemetry/SyncActionLog.swift + + @Insert + suspend fun insertSyncAction(action: SyncActionEntity): Long + + /** + * Record that [target] identified by [boardId]/[key] was semantically removed. + * + * [rowUpdatedAt] is the removed row's own last-write-wins timestamp, read before the delete: the + * action is stamped `max(now, rowUpdatedAt)` so a rewound device clock cannot produce an action + * the server reads as older than the row it names — that action would be dropped as a no-op, and + * the phone could not self-heal by re-sending, because the row is gone. + * + * A null [rowUpdatedAt] means there was no row to remove, so no intent to record either. + */ + @Transaction + suspend fun appendDeleteAction( + target: DeleteTarget, + boardId: String?, + key: String, + rowUpdatedAt: Long?, + now: Long = System.currentTimeMillis(), + ) { + if (rowUpdatedAt == null) return + insertSyncAction( + SyncActionEntity( + target = target.wire, + boardId = boardId, + key = key, + deletedAt = maxOf(now, rowUpdatedAt), + ), + ) + } + + /** The next page of actions to upload, in cursor order. */ + @Query("SELECT * FROM sync_actions WHERE id > :afterId ORDER BY id ASC LIMIT :limit") + suspend fun getSyncActionsAfter(afterId: Long, limit: Int): List + + @Query( + "INSERT OR REPLACE INTO sync_sequences (name, last_value) VALUES (:name, " + + "MAX(:value, COALESCE((SELECT last_value FROM sync_sequences WHERE name = :name), 0)))", + ) + suspend fun commitSyncActionCursorRow(name: String, value: Long) + + /** + * Checkpoint the highest action cursor the server has accepted. Its own transaction, committed + * before [pruneUploadedSyncActions] runs: a crash between the two leaves rows that will be sent + * again — harmless, since applying an action twice is a no-op — whereas pruning first would drop + * an action nobody has accepted. Never moves backwards, so an out-of-order commit cannot un-accept + * what an earlier upload already checkpointed. + */ + @Transaction + suspend fun commitSyncActionCursor(throughId: Long) = + commitSyncActionCursorRow(SYNC_ACTIONS_UPLOADED_CURSOR, throughId) + + @Query("DELETE FROM sync_actions WHERE id <= :throughId") + suspend fun deleteSyncActionsThrough(throughId: Long): Int + + /** + * Drop what the server has already accepted. Gated on the committed cursor rather than a caller's + * number, so pruning structurally cannot outrun the checkpoint. + */ + @Transaction + suspend fun pruneUploadedSyncActions(): Int { + val accepted = getSyncSequence(SYNC_ACTIONS_UPLOADED_CURSOR) ?: return 0 + return deleteSyncActionsThrough(accepted) + } + + // Sync Cursors — the uploader's forward scan (#284). Mutable tables scan on `sync_seq`, + // append-only tables on their `AUTOINCREMENT` key; both are device-local counters that never + // cross the wire. + // @parity /modules/vescape-core/ios/sync/SyncStore.swift + + /** + * Checkpoint how far [name] has been accepted. Its own transaction, run after the response and + * never alongside the rows: a cursor advanced past rows the server did not take is unrecoverable, + * whereas a cursor left behind is a re-send the server upserts idempotently. Never moves + * backwards, so an out-of-order commit cannot un-accept an earlier one. + */ + @Transaction + suspend fun commitSyncCursor(name: String, throughValue: Long) = + commitSyncActionCursorRow(name, throughValue) + + @Query("SELECT * FROM app_settings WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getAppSettingsAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM boards WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getBoardsAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM board_settings WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getBoardSettingsAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM board_warnings WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getBoardWarningsAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM alerts WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getAlertsAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM tune_profiles WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getTuneProfilesAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM tune_history_entries WHERE id > :cursor ORDER BY id ASC LIMIT :limit") + suspend fun getTuneHistoryEntriesAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM privacy_zones WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getPrivacyZonesAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM telemetry_markers WHERE id > :cursor ORDER BY id ASC LIMIT :limit") + suspend fun getTelemetryMarkersAfter(cursor: Long, limit: Int): List + + /** + * A range whose Board is the unknown-Board sentinel is unowned in the same way as a frame: an + * unattributed range names no Board, so the server has nothing to hang it off — its composite + * foreign key refuses `''` and 409s the whole Sync Batch. The row is retained, so the same batch + * would retry forever; the scan skips it instead. + */ + @Query( + "SELECT * FROM metric_exclusion_ranges WHERE id > :cursor AND board_id != '' " + + "ORDER BY id ASC LIMIT :limit", + ) + suspend fun getExclusionRangesAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM diagnostic_events WHERE id > :cursor ORDER BY id ASC LIMIT :limit") + suspend fun getDiagnosticEventsAfter(cursor: Long, limit: Int): List + + /** + * Frames that name no Board cannot be uploaded — the server keys this table on the Board and has + * nowhere to put a sample that belongs to none (ADR-0028) — so the scan does not offer them and + * the cursor moves over them. They are unowned local rows, not rows a Rider is waiting to see + * backed up. + * + * The consequence is deliberate: a later owned frame carries the cursor past a skipped one, so + * cursor-gated retention prunes unowned telemetry on age alone, exactly as it did before the + * Account binding existed. Holding it forever would be the only alternative, because no future + * upload can ever accept it. + */ + @Query( + "SELECT * FROM telemetry_frames WHERE id > :cursor AND board_id IS NOT NULL " + + "ORDER BY id ASC LIMIT :limit", + ) + suspend fun getTelemetryFramesAfter(cursor: Long, limit: Int): List + + /** Buckets whose Board is the unknown-Board sentinel are unowned in the same way as a frame. */ + @Query( + "SELECT * FROM telemetry_minute_buckets WHERE sync_seq > :cursor AND board_id != '' " + + "ORDER BY sync_seq ASC LIMIT :limit", + ) + suspend fun getMinuteBucketsAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM favorites WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getFavoritesAfter(cursor: Long, limit: Int): List + + // VESC Fault Evidence. Never pruned — the one Rider-visible history that is permanent on the + // phone (ADR-0016) — so these scans have no retention counterpart to stay ahead of. + + @Query( + "SELECT * FROM vesc_fault_occurrences WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit", + ) + suspend fun getVescFaultOccurrencesAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM vesc_fault_captures WHERE sync_seq > :cursor ORDER BY sync_seq ASC LIMIT :limit") + suspend fun getVescFaultCapturesAfter(cursor: Long, limit: Int): List + + @Query("SELECT * FROM vesc_fault_capture_samples WHERE id > :cursor ORDER BY id ASC LIMIT :limit") + suspend fun getVescFaultCaptureSamplesAfter(cursor: Long, limit: Int): List + + @Query("SELECT COUNT(*) FROM app_settings WHERE sync_seq > :cursor") + suspend fun countAppSettingsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM boards WHERE sync_seq > :cursor") + suspend fun countBoardsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM board_settings WHERE sync_seq > :cursor") + suspend fun countBoardSettingsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM board_warnings WHERE sync_seq > :cursor") + suspend fun countBoardWarningsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM alerts WHERE sync_seq > :cursor") + suspend fun countAlertsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM tune_profiles WHERE sync_seq > :cursor") + suspend fun countTuneProfilesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM tune_history_entries WHERE id > :cursor") + suspend fun countTuneHistoryEntriesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM privacy_zones WHERE sync_seq > :cursor") + suspend fun countPrivacyZonesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM telemetry_markers WHERE id > :cursor") + suspend fun countTelemetryMarkersAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM metric_exclusion_ranges WHERE id > :cursor AND board_id != ''") + suspend fun countExclusionRangesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM diagnostic_events WHERE id > :cursor") + suspend fun countDiagnosticEventsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM telemetry_frames WHERE id > :cursor AND board_id IS NOT NULL") + suspend fun countTelemetryFramesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM telemetry_minute_buckets WHERE sync_seq > :cursor AND board_id != ''") + suspend fun countMinuteBucketsAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM favorites WHERE sync_seq > :cursor") + suspend fun countFavoritesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM vesc_fault_occurrences WHERE sync_seq > :cursor") + suspend fun countVescFaultOccurrencesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM vesc_fault_captures WHERE sync_seq > :cursor") + suspend fun countVescFaultCapturesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM vesc_fault_capture_samples WHERE id > :cursor") + suspend fun countVescFaultCaptureSamplesAfter(cursor: Long): Int + + @Query("SELECT COUNT(*) FROM sync_actions WHERE id > :cursor") + suspend fun countSyncActionsAfter(cursor: Long): Int + + // Account binding — which Vescape Account this local database belongs to (#284). One row, so a + // database replaced on an Account change starts unbound with no cursors and no actions. + + @Query("SELECT account_id FROM sync_binding WHERE id = 0") + suspend fun getBoundAccountId(): String? + + @Query("INSERT OR REPLACE INTO sync_binding (id, account_id, bound_at) VALUES (0, :accountId, :boundAt)") + suspend fun bindAccountRow(accountId: String, boundAt: Long) + + /** + * Claim this database for [accountId], or confirm it already belongs to it. Returns false when it + * belongs to a different Account: the caller has to replace the database first, because resetting + * the cursors over these rows would upload the previous Account's data to the new one. + */ + @Transaction + suspend fun bindAccount(accountId: String, now: Long = System.currentTimeMillis()): Boolean { + val bound = getBoundAccountId() + if (bound != null) return bound == accountId + bindAccountRow(accountId, now) + return true + } + + // Cursor-gated retention (#284). A retention cutoff is only a candidate cutoff: cleanup must not + // remove a row the uploader has not delivered. Each sweep reads its table cursor and deletes in + // one transaction, so racing an upload fails safe — before the cursor commit the rows are + // retained, after it the server has accepted them. A missing cursor is 0, protecting every row. + + @Query("DELETE FROM telemetry_frames WHERE captured_at_ms < :beforeMs AND id <= :cursor") + suspend fun deleteFramesBeforeUpTo(beforeMs: Long, cursor: Long): Int + + @Query("DELETE FROM telemetry_markers WHERE occurred_at_ms < :beforeMs AND id <= :cursor") + suspend fun deleteMarkersBeforeUpTo(beforeMs: Long, cursor: Long): Int + + @Query("DELETE FROM telemetry_minute_buckets WHERE bucket_start_ms < :beforeMs AND sync_seq <= :cursor") + suspend fun deleteBucketsBeforeUpTo(beforeMs: Long, cursor: Long): Int + + @Query("DELETE FROM diagnostic_events WHERE occurred_at_ms < :beforeMs AND id <= :cursor") + suspend fun deleteDiagnosticEventsBeforeUpTo(beforeMs: Long, cursor: Long): Int + + @Query("DELETE FROM metric_exclusion_ranges WHERE end_ms < :beforeMs AND id <= :cursor") + suspend fun deleteExclusionsBeforeUpTo(beforeMs: Long, cursor: Long): Int + + /** + * Age-only cleanup while the database has never been bound to an Account, and age plus the + * accepted Sync Cursor once it has. Emits no Sync Actions — a retention sweep is maintenance, and + * `DeleteTarget` has no case that could name a pruned table. + */ + @Transaction + suspend fun deleteBeforeGated(beforeMs: Long): Int { + if (getBoundAccountId() == null) return deleteBefore(beforeMs) + val frames = deleteFramesBeforeUpTo(beforeMs, cursorOf(SYNC_CURSOR_FRAMES)) + deleteMarkersBeforeUpTo(beforeMs, cursorOf(SYNC_CURSOR_MARKERS)) + deleteBucketsBeforeUpTo(beforeMs, cursorOf(SYNC_CURSOR_MINUTE_BUCKETS)) + deleteDiagnosticEventsBeforeUpTo(beforeMs, cursorOf(SYNC_CURSOR_DIAGNOSTIC_EVENTS)) + deleteExclusionsBeforeUpTo(beforeMs, cursorOf(SYNC_CURSOR_EXCLUSION_RANGES)) + return frames + } + + /** A table with no committed cursor has delivered nothing, so none of its rows may be pruned. */ + suspend fun cursorOf(name: String): Long = getSyncSequence(name) ?: 0L + @Transaction suspend fun insertBatch( frames: List, @@ -383,16 +723,29 @@ interface TelemetryDao { @Insert(onConflict = OnConflictStrategy.REPLACE) suspend fun insertBoardRow(board: BoardEntity) + @Query("SELECT updated_at FROM boards WHERE id = :id") + suspend fun getBoardUpdatedAt(id: String): Long? + @Query("SELECT deleted_at FROM boards WHERE id = :id") suspend fun getBoardDeletedAt(id: String): Long? /** + * Stamps both sync columns before the row lands: a fresh `sync_seq` so the upload scan sees this + * write, and a ratcheted `updated_at` so the server keeps it. Caller-supplied values for either + * are overwritten — see [SyncSequenceEntity] and [BoardEntity.updatedAt]. + * * An existing tombstone survives the write, so an ordinary upsert can never resurrect a deleted * Board — deletion is terminal (ADR 0027). Only [deleteBoardWithSettings] stamps a new one. */ @Transaction suspend fun upsertBoard(board: BoardEntity) { - insertBoardRow(board.copy(deletedAt = board.deletedAt ?: getBoardDeletedAt(board.id))) + insertBoardRow( + board.copy( + updatedAt = ratchetUpdatedAt(getBoardUpdatedAt(board.id), board.updatedAt), + syncSeq = nextSyncSeq(SYNC_SEQ_BOARDS), + deletedAt = board.deletedAt ?: getBoardDeletedAt(board.id), + ), + ) } /** @@ -403,6 +756,10 @@ interface TelemetryDao { @Query("SELECT id, name FROM boards") suspend fun getBoardNames(): List + /** The BLE identifier a Board currently claims, for the tables still keyed on it. */ + @Query("SELECT ble_id FROM boards WHERE id = :id LIMIT 1") + suspend fun getBoardBleId(id: String): String? + @Query("SELECT * FROM board_settings WHERE board_id = :boardId") suspend fun getBoardSettings(boardId: String): List @@ -410,10 +767,42 @@ interface TelemetryDao { suspend fun getBoardSettings(boardIds: List): List @Insert(onConflict = OnConflictStrategy.REPLACE) - suspend fun upsertBoardSetting(setting: BoardSettingEntity) + suspend fun insertBoardSettingRow(setting: BoardSettingEntity) + + @Query("SELECT updated_at FROM board_settings WHERE board_id = :boardId AND key = :key") + suspend fun getBoardSettingUpdatedAt(boardId: String, key: String): Long? + + /** Stamps both sync columns; see [upsertBoard]. */ + @Transaction + suspend fun upsertBoardSetting(setting: BoardSettingEntity) { + insertBoardSettingRow( + setting.copy( + updatedAt = ratchetUpdatedAt( + getBoardSettingUpdatedAt(setting.boardId, setting.key), + setting.updatedAt, + ), + syncSeq = nextSyncSeq(SYNC_SEQ_BOARD_SETTINGS), + ), + ) + } @Query("DELETE FROM board_settings WHERE board_id = :boardId AND key = :key") - suspend fun deleteBoardSetting(boardId: String, key: String) + suspend fun deleteBoardSettingRow(boardId: String, key: String) + + /** + * Semantic removal: a Board edit that drops a key is the Rider clearing that setting, so a restore + * must not resurrect the old value. + */ + @Transaction + suspend fun deleteBoardSetting(boardId: String, key: String) { + appendDeleteAction( + DeleteTarget.BOARD_SETTING, + boardId, + key, + getBoardSettingUpdatedAt(boardId, key), + ) + deleteBoardSettingRow(boardId, key) + } @Transaction suspend fun upsertBoardWithSettings(board: BoardEntity, settings: List, deletedKeys: List) { @@ -422,23 +811,36 @@ interface TelemetryDao { settings.forEach { upsertBoardSetting(it) } } + /** Parent-covered cascade: raw, because the Board's own action covers its configuration. */ @Query("DELETE FROM board_settings WHERE board_id = :boardId") - suspend fun deleteBoardSettings(boardId: String) + suspend fun deleteBoardSettingsRaw(boardId: String) /** * The Rider-facing delete: configuration goes, the Board row stays as a tombstone (ADR 0027). * Telemetry and Tune Profiles are untouched — both outlive the Board. * - * An unknown or already-tombstoned id is a no-op. + * The tombstone is an ordinary write, so it runs through [upsertBoard] and moves both sync + * columns like any other edit. An unknown or already-tombstoned id is a no-op. + * + * The tombstone syncs as an ordinary upsert *and* emits one Sync Action, because the two say + * different things: the row says the Board is deleted, the action says its configuration is gone. + * Keeping the cascade an explicit, replay-safe action is what stops a dumb upsert from quietly + * deleting rows in three other tables. The children are raw deletes — the Board's action covers + * them (#282). + * + * The action and the tombstone share one timestamp, the newly ratcheted `updated_at`, so the + * server judges both against the same moment. */ @Transaction suspend fun deleteBoardWithSettings(id: String, deletedAt: Long) { val board = getBoard(id)?.takeIf { it.deletedAt == null } ?: return - deleteBoardSettings(id) - deleteBoardWarnings(id) + val tombstonedAt = ratchetUpdatedAt(board.updatedAt, deletedAt) + deleteBoardSettingsRaw(id) + deleteBoardWarningsRaw(id) // Alert Rules are Board-owned (#254) — drop them with the Board so no orphan rows survive. - deleteAlertRules(id) - insertBoardRow(board.copy(deletedAt = deletedAt)) + deleteAlertRulesRaw(id) + appendDeleteAction(DeleteTarget.BOARD, null, id, tombstonedAt, tombstonedAt) + upsertBoard(board.copy(deletedAt = tombstonedAt, updatedAt = tombstonedAt)) } @Query("SELECT * FROM alerts WHERE board_id = :boardId ORDER BY created_at ASC") @@ -448,16 +850,65 @@ interface TelemetryDao { suspend fun getEnabledAlertRules(boardId: String): List @Insert(onConflict = OnConflictStrategy.REPLACE) - suspend fun upsertAlertRule(rule: AlertRuleEntity) + suspend fun insertAlertRuleRow(rule: AlertRuleEntity) - @Query("UPDATE alerts SET enabled = :enabled WHERE board_id = :boardId AND id = :id") - suspend fun setAlertRuleEnabled(boardId: String, id: String, enabled: Boolean) + @Query("SELECT updated_at FROM alerts WHERE board_id = :boardId AND id = :id") + suspend fun getAlertRuleUpdatedAt(boardId: String, id: String): Long? + + /** Stamps both sync columns; see [upsertBoard]. */ + @Transaction + suspend fun upsertAlertRule(rule: AlertRuleEntity) { + insertAlertRuleRow( + rule.copy( + updatedAt = ratchetUpdatedAt(getAlertRuleUpdatedAt(rule.boardId, rule.id), rule.updatedAt), + syncSeq = nextSyncSeq(SYNC_SEQ_ALERTS), + ), + ) + } + + /** + * Targeted toggle. Unlike the `@Insert` upserts it never round-trips an entity, so both sync + * columns have to move here explicitly — without them, toggling a rule leaves it invisible to the + * upload scan and the change never reaches the server. + * + * The `MAX(updated_at + 1, :updatedAt)` fold is the same ratchet [upsertBoard] applies, expressed + * in SQL because the row is already being read by the `WHERE`. + */ + @Query( + "UPDATE alerts SET enabled = :enabled, updated_at = MAX(updated_at + 1, :updatedAt), " + + "sync_seq = :syncSeq WHERE board_id = :boardId AND id = :id", + ) + suspend fun setAlertRuleEnabledRow( + boardId: String, + id: String, + enabled: Boolean, + updatedAt: Long, + syncSeq: Long, + ) + + /** @parity /modules/vescape-core/ios/telemetry/AppDataRepository.swift `setAlertRuleEnabled` */ + @Transaction + suspend fun setAlertRuleEnabled(boardId: String, id: String, enabled: Boolean, updatedAt: Long) { + setAlertRuleEnabledRow(boardId, id, enabled, updatedAt, nextSyncSeq(SYNC_SEQ_ALERTS)) + } @Query("DELETE FROM alerts WHERE board_id = :boardId AND id = :id") - suspend fun deleteAlertRule(boardId: String, id: String) + suspend fun deleteAlertRuleRow(boardId: String, id: String) + + /** + * Semantic removal, and the path preset regeneration takes too: JS regenerates a Board's preset + * rules by deleting the old ones and writing new ones, and the deleted ones have to disappear + * server-side as well. + */ + @Transaction + suspend fun deleteAlertRule(boardId: String, id: String) { + appendDeleteAction(DeleteTarget.ALERT, boardId, id, getAlertRuleUpdatedAt(boardId, id)) + deleteAlertRuleRow(boardId, id) + } + /** Parent-covered cascade: raw, because the Board's own action covers its Alert Rules. */ @Query("DELETE FROM alerts WHERE board_id = :boardId") - suspend fun deleteAlertRules(boardId: String) + suspend fun deleteAlertRulesRaw(boardId: String) @Query("SELECT * FROM app_settings") suspend fun getAllAppSettings(): List @@ -466,10 +917,46 @@ interface TelemetryDao { suspend fun getAppSetting(key: String): AppSettingEntity? @Insert(onConflict = OnConflictStrategy.REPLACE) - suspend fun upsertAppSetting(setting: AppSettingEntity) + suspend fun insertAppSettingRow(setting: AppSettingEntity) + + @Query("SELECT updated_at FROM app_settings WHERE key = :key") + suspend fun getAppSettingUpdatedAt(key: String): Long? + + /** + * Stamps both sync columns like [upsertBoard], except for the phone-local keys in + * [NOT_SYNCED_SETTING_KEYS]: those keep `sync_seq` at 0, which sits below every Sync Cursor, so + * the upload scan never picks the row up and the key stays on this phone (#277). + */ + @Transaction + suspend fun upsertAppSetting(setting: AppSettingEntity) { + val phoneLocal = setting.key in NOT_SYNCED_SETTING_KEYS + insertAppSettingRow( + setting.copy( + updatedAt = ratchetUpdatedAt(getAppSettingUpdatedAt(setting.key), setting.updatedAt), + syncSeq = if (phoneLocal) 0L else nextSyncSeq(SYNC_SEQ_APP_SETTINGS), + ), + ) + } @Query("DELETE FROM app_settings WHERE key = :key") - suspend fun deleteAppSetting(key: String) + suspend fun deleteAppSettingRow(key: String) + + /** + * Semantic removal. Every caller means the same thing — the stored override is gone: an edit back + * to the default, `legalPolicy` resolving to nothing, and the corrupt-value cleanup in + * [AppDataRepository.getTypedSettings], which is deliberately semantic so a restore cannot + * resurrect a value this phone already rejected. + * + * Phone-local keys never reach the server (they carry `sync_seq = 0`), so removing one records no + * action either — an action for a row the server never held would delete nothing and say nothing. + */ + @Transaction + suspend fun deleteAppSetting(key: String) { + if (key !in NOT_SYNCED_SETTING_KEYS) { + appendDeleteAction(DeleteTarget.APP_SETTING, null, key, getAppSettingUpdatedAt(key)) + } + deleteAppSettingRow(key) + } // Tune Profile / Tune History DAO. Transactional bodies below are mirrored in Swift. // @parity /modules/vescape-core/ios/telemetry/TuneProfileStore.swift @@ -480,28 +967,75 @@ interface TelemetryDao { suspend fun getTuneProfile(id: String): TuneProfileEntity? @Query("DELETE FROM tune_profiles WHERE id = :id") - suspend fun deleteTuneProfile(id: String) + suspend fun deleteTuneProfileRow(id: String) + /** Parent-covered cascade: raw, because the profile's own action covers its Tune History. */ @Query("DELETE FROM tune_history_entries WHERE profile_id = :profileId") - suspend fun deleteTuneHistoryForProfile(profileId: String) + suspend fun deleteTuneHistoryForProfileRaw(profileId: String) - @Query("UPDATE tune_profiles SET name = :name, icon = :icon, color = :color, updated_at = :updatedAt WHERE id = :profileId") - suspend fun updateProfileMetadata( + /** Targeted rename that bypasses the upsert, so it moves both columns itself; see + * [setAlertRuleEnabledRow]. */ + @Query( + "UPDATE tune_profiles SET name = :name, icon = :icon, color = :color, " + + "updated_at = MAX(updated_at + 1, :updatedAt), sync_seq = :syncSeq WHERE id = :profileId", + ) + suspend fun updateProfileMetadataRow( profileId: String, name: String, icon: String, color: String, updatedAt: Long, + syncSeq: Long, ): Int + @Transaction + suspend fun updateProfileMetadata( + profileId: String, + name: String, + icon: String, + color: String, + updatedAt: Long, + ): Int = updateProfileMetadataRow( + profileId, + name, + icon, + color, + updatedAt, + nextSyncSeq(SYNC_SEQ_TUNE_PROFILES), + ) + @Query("SELECT * FROM tune_history_entries WHERE id = :id LIMIT 1") suspend fun getTuneHistoryEntry(id: Long): TuneHistoryEntryEntity? @Insert(onConflict = OnConflictStrategy.REPLACE) - suspend fun upsertTuneProfile(profile: TuneProfileEntity) + suspend fun insertTuneProfileRow(profile: TuneProfileEntity) + + @Query("SELECT updated_at FROM tune_profiles WHERE id = :id") + suspend fun getTuneProfileUpdatedAt(id: String): Long? + + /** Stamps both sync columns; see [upsertBoard]. */ + @Transaction + suspend fun upsertTuneProfile(profile: TuneProfileEntity) { + insertTuneProfileRow( + profile.copy( + updatedAt = ratchetUpdatedAt(getTuneProfileUpdatedAt(profile.id), profile.updatedAt), + syncSeq = nextSyncSeq(SYNC_SEQ_TUNE_PROFILES), + ), + ) + } @Insert(onConflict = OnConflictStrategy.IGNORE) - suspend fun insertTuneProfile(profile: TuneProfileEntity): Long + suspend fun insertTuneProfileRowIfAbsent(profile: TuneProfileEntity): Long + + /** Stamps both sync columns; see [upsertBoard]. Returns -1 when the row already exists. */ + @Transaction + suspend fun insertTuneProfile(profile: TuneProfileEntity): Long = + insertTuneProfileRowIfAbsent( + profile.copy( + updatedAt = ratchetUpdatedAt(getTuneProfileUpdatedAt(profile.id), profile.updatedAt), + syncSeq = nextSyncSeq(SYNC_SEQ_TUNE_PROFILES), + ), + ) @Query("SELECT COUNT(*) FROM tune_profiles WHERE board_id = :boardId AND refloat_base_version = :refloatBaseVersion") suspend fun countTuneProfilesForBoard(boardId: String, refloatBaseVersion: String): Int @@ -515,8 +1049,22 @@ interface TelemetryDao { @Query("SELECT * FROM tune_history_entries WHERE profile_id = :profileId ORDER BY created_at DESC, id DESC") suspend fun getTuneHistoryEntries(profileId: String): List - @Query("UPDATE tune_profiles SET fields_json = :fieldsJson, updated_at = :updatedAt WHERE id = :profileId") - suspend fun updateProfileFields(profileId: String, fieldsJson: String, updatedAt: Long): Int + /** Targeted save that bypasses the upsert, so it moves both columns itself; see + * [setAlertRuleEnabledRow]. */ + @Query( + "UPDATE tune_profiles SET fields_json = :fieldsJson, " + + "updated_at = MAX(updated_at + 1, :updatedAt), sync_seq = :syncSeq WHERE id = :profileId", + ) + suspend fun updateProfileFieldsRow( + profileId: String, + fieldsJson: String, + updatedAt: Long, + syncSeq: Long, + ): Int + + @Transaction + suspend fun updateProfileFields(profileId: String, fieldsJson: String, updatedAt: Long): Int = + updateProfileFieldsRow(profileId, fieldsJson, updatedAt, nextSyncSeq(SYNC_SEQ_TUNE_PROFILES)) @Transaction suspend fun saveTuneProfile(profileId: String, fieldsJson: String, updatedAt: Long): TuneProfileEntity { @@ -538,8 +1086,9 @@ interface TelemetryDao { if (countTuneProfilesForBoard(profile.boardId, profile.refloatBaseVersion) <= 1) { throw IllegalStateException("Cannot delete the last profile for a board") } - deleteTuneHistoryForProfile(profileId) - deleteTuneProfile(profileId) + deleteTuneHistoryForProfileRaw(profileId) + appendDeleteAction(DeleteTarget.TUNE_PROFILE, null, profileId, profile.updatedAt) + deleteTuneProfileRow(profileId) } @Transaction @@ -584,13 +1133,69 @@ interface TelemetryDao { suspend fun getAllBoardWarnings(): List @Insert(onConflict = OnConflictStrategy.REPLACE) - suspend fun upsertBoardWarning(warning: BoardWarningEntity) + suspend fun insertBoardWarningRow(warning: BoardWarningEntity) + + @Query("SELECT updated_at FROM board_warnings WHERE board_id = :boardId AND kind = :kind") + suspend fun getBoardWarningUpdatedAt(boardId: String, kind: String): Long? + + /** + * Stamps both sync columns; see [upsertBoard]. The caller supplies detection times only — + * `updated_at` is authored here, from [BoardWarningEntity.lastDetectedAt] as the write clock. + */ + @Transaction + suspend fun upsertBoardWarning(warning: BoardWarningEntity) { + insertBoardWarningRow( + warning.copy( + updatedAt = ratchetUpdatedAt( + getBoardWarningUpdatedAt(warning.boardId, warning.kind), + warning.lastDetectedAt, + ), + syncSeq = nextSyncSeq(SYNC_SEQ_BOARD_WARNINGS), + ), + ) + } + + @Query("SELECT last_detected_at FROM board_warnings WHERE board_id = :boardId AND kind = :kind") + suspend fun getBoardWarningLastDetectedAt(boardId: String, kind: String): Long? + + @Query("SELECT kind FROM board_warnings WHERE board_id = :boardId") + suspend fun getBoardWarningKinds(boardId: String): List @Query("DELETE FROM board_warnings WHERE board_id = :boardId AND kind = :kind") - suspend fun deleteBoardWarning(boardId: String, kind: String): Int + suspend fun deleteBoardWarningRow(boardId: String, kind: String): Int + + /** + * Semantic removal, whether the Rider cleared the warning or a detector evaluated the kind with + * real data and found the condition gone — an automatic clear is still a durable state transition + * the server has to make (#282). + * + * Stamped from `last_detected_at` rather than `updated_at`: it is the warning's own change clock, + * and it is what the row's `updated_at` was written from. + */ + @Transaction + suspend fun deleteBoardWarning(boardId: String, kind: String): Int { + appendDeleteAction( + DeleteTarget.BOARD_WARNING, + boardId, + kind, + getBoardWarningLastDetectedAt(boardId, kind), + ) + return deleteBoardWarningRow(boardId, kind) + } @Query("DELETE FROM board_warnings WHERE board_id = :boardId") - suspend fun deleteBoardWarnings(boardId: String): Int + suspend fun deleteBoardWarningsRaw(boardId: String): Int + + /** + * The Rider cleared every warning on one Board: one action per removed row, because each row is a + * separate piece of current state. Distinct from the Board delete's cascade, which is raw. + */ + @Transaction + suspend fun deleteBoardWarnings(boardId: String): Int { + var removed = 0 + for (kind in getBoardWarningKinds(boardId)) removed += deleteBoardWarning(boardId, kind) + return removed + } // VESC Fault Occurrences — see VescFaultCoordinator for lifecycle rules. Deliberately absent from // `deleteBoardWithSettings`: fault evidence outlives the Board record. @@ -615,36 +1220,78 @@ interface TelemetryDao { @Insert(onConflict = OnConflictStrategy.IGNORE) suspend fun insertVescFault(fault: VescFaultOccurrenceEntity): Long + @Query("SELECT updated_at FROM vesc_fault_occurrences WHERE id = :id") + suspend fun getVescFaultUpdatedAt(id: String): Long? + @Query( - "UPDATE vesc_fault_occurrences SET last_observed_at = :lastObservedAt, cleared_at = :clearedAt WHERE id = :id", + "UPDATE vesc_fault_occurrences SET last_observed_at = :lastObservedAt, cleared_at = :clearedAt, " + + "updated_at = :updatedAt, sync_seq = :syncSeq WHERE id = :id", ) suspend fun updateVescFaultLifecycle( id: String, lastObservedAt: Long, clearedAt: Long?, + updatedAt: Long, + syncSeq: Long, ) /** * Insert-or-advance. Deliberately not a `REPLACE` upsert: that rewrites `dismissed` from the * caller's in-memory snapshot, so a stale heartbeat could un-dismiss what the rider just * acknowledged. Dismissal has its own statement. + * + * Stamps both sync columns like [upsertBoardWarning]: the caller supplies observation times only, + * so `updated_at` is authored here from [VescFaultOccurrenceEntity.lastObservedAtMs] as the write + * clock. Both branches write the row, so both take the one position handed out. */ @Transaction suspend fun upsertVescFault(fault: VescFaultOccurrenceEntity) { - if (insertVescFault(fault) == -1L) { - updateVescFaultLifecycle(fault.id, fault.lastObservedAtMs, fault.clearedAtMs) + val updatedAt = ratchetUpdatedAt(getVescFaultUpdatedAt(fault.id), fault.lastObservedAtMs) + val syncSeq = nextSyncSeq(SYNC_SEQ_VESC_FAULT_OCCURRENCES) + if (insertVescFault(fault.copy(updatedAt = updatedAt, syncSeq = syncSeq)) == -1L) { + updateVescFaultLifecycle(fault.id, fault.lastObservedAtMs, fault.clearedAtMs, updatedAt, syncSeq) } } - @Query("UPDATE vesc_fault_occurrences SET dismissed = :dismissed WHERE id = :id") - suspend fun setVescFaultDismissed(id: String, dismissed: Boolean): Int + @Query( + "UPDATE vesc_fault_occurrences SET dismissed = :dismissed, " + + "updated_at = MAX(updated_at + 1, :updatedAt), sync_seq = :syncSeq WHERE id = :id", + ) + suspend fun setVescFaultDismissedRow( + id: String, + dismissed: Boolean, + updatedAt: Long, + syncSeq: Long, + ): Int + + /** + * The Rider acknowledged an occurrence. A targeted `UPDATE` rather than an entity round-trip, so + * it has to move both sync columns in its own SQL — and it is the write `updated_at` exists for: + * nothing else about the row changes, so without the stamp the edit would never reach the server. + */ + @Transaction + suspend fun setVescFaultDismissed( + id: String, + dismissed: Boolean, + now: Long = System.currentTimeMillis(), + ): Int = setVescFaultDismissedRow(id, dismissed, now, nextSyncSeq(SYNC_SEQ_VESC_FAULT_OCCURRENCES)) // VESC Fault Captures — one self-contained window of decoded Board samples per occurrence. Append // only, no GPS, and outside every Ride History retention/pruning path. // @parity /modules/vescape-core/ios/faults/VescFaultCaptureStore.swift @Upsert - suspend fun upsertVescFaultCapture(capture: VescFaultCaptureEntity) + suspend fun upsertVescFaultCaptureRow(capture: VescFaultCaptureEntity) + + /** + * Stamps the Sync Cursor position. A Capture is immutable once written, so a rewrite is a + * re-statement of the same snapshot rather than an edit — it still takes a fresh position, + * because the scan may already have passed the row it replaces. + */ + @Transaction + suspend fun upsertVescFaultCapture(capture: VescFaultCaptureEntity) { + upsertVescFaultCaptureRow(capture.copy(syncSeq = nextSyncSeq(SYNC_SEQ_VESC_FAULT_CAPTURES))) + } @Query("SELECT * FROM vesc_fault_captures WHERE occurrence_id = :occurrenceId LIMIT 1") suspend fun getVescFaultCapture(occurrenceId: String): VescFaultCaptureEntity? @@ -738,14 +1385,34 @@ interface TelemetryDao { suspend fun getFavorites(): List @Insert - suspend fun insertFavorite(favorite: FavoriteEntity) + suspend fun insertFavoriteRow(favorite: FavoriteEntity) @Query("SELECT * FROM favorites WHERE id = :id") suspend fun getFavorite(id: String): FavoriteEntity? - /** Re-trim/rename one row in place so its identity and Favorite Media remain stable. */ + @Query("SELECT updated_at FROM favorites WHERE id = :id") + suspend fun getFavoriteUpdatedAt(id: String): Long? + @Update - suspend fun updateFavorite(favorite: FavoriteEntity): Int + suspend fun updateFavoriteRow(favorite: FavoriteEntity): Int + + /** Stamps both sync columns; see [upsertBoard]. */ + @Transaction + suspend fun insertFavorite(favorite: FavoriteEntity) { + insertFavoriteRow(favorite.copy(syncSeq = nextSyncSeq(SYNC_SEQ_FAVORITES))) + } + + /** + * Re-trim/rename one row in place so its identity and Favorite Media remain stable. Stamps both + * sync columns; see [upsertBoard]. + */ + @Transaction + suspend fun updateFavorite(favorite: FavoriteEntity): Int = updateFavoriteRow( + favorite.copy( + updatedAt = ratchetUpdatedAt(getFavoriteUpdatedAt(favorite.id), favorite.updatedAt), + syncSeq = nextSyncSeq(SYNC_SEQ_FAVORITES), + ), + ) @Query("DELETE FROM favorites WHERE id = :id") suspend fun deleteFavoriteRow(id: String): Int @@ -762,16 +1429,21 @@ interface TelemetryDao { @Query("DELETE FROM favorite_media WHERE id = :id") suspend fun deleteFavoriteMedia(id: String): Int + /** Parent-covered cascade: raw, because the Favorite's own action covers its manifest rows. */ @Query("DELETE FROM favorite_media WHERE favorite_id = :favoriteId") - suspend fun deleteFavoriteMediaForFavorite(favoriteId: String): Int + suspend fun deleteFavoriteMediaForFavoriteRaw(favoriteId: String): Int @Query("DELETE FROM favorite_media WHERE favorite_id NOT IN (SELECT id FROM favorites)") suspend fun deleteOrphanFavoriteMedia(): Int - /** Parent-covered raw cascade: media rows and Favorite disappear in one SQLite transaction. */ + /** + * Semantic removal of the Favorite, with its Favorite Media manifest rows as a parent-covered raw + * cascade — one action, not one per media row, matching the server's own cascade. + */ @Transaction suspend fun deleteFavorite(id: String): Int { - deleteFavoriteMediaForFavorite(id) + deleteFavoriteMediaForFavoriteRaw(id) + appendDeleteAction(DeleteTarget.FAVORITE, null, id, getFavoriteUpdatedAt(id)) return deleteFavoriteRow(id) } } @@ -837,9 +1509,25 @@ private fun TelemetryMinuteBucketEntity.merge(next: TelemetryMinuteBucketEntity) }, firstMovingAtMs = mergeNullableMin(firstMovingAtMs, next.firstMovingAtMs), lastMovingAtMs = mergeNullableMax(lastMovingAtMs, next.lastMovingAtMs), + // The merged row is being written now, so `next` normally carries the fresher stamp. The same + // ratchet as boards and alerts, for the same reason: the server guards this table with + // `WHERE stored.updated_at < EXCLUDED.updated_at` like every other mutable table, so a stamp + // frozen at the stored value would satisfy the scan and still be dropped server-side. + updatedAt = ratchetUpdatedAt(updatedAt, next.updatedAt), + syncSeq = next.syncSeq, ) } +/** + * The write-time fold behind [BoardEntity.updatedAt]: never below the value already stored, and + * strictly above it whenever the clock fails to be. + * + * `+ 1` rather than a plain `maxOf` because the server keeps the stored row unless the incoming + * stamp is strictly newer. Freezing at the old value would satisfy the scan and still lose the edit. + */ +internal fun ratchetUpdatedAt(previous: Long?, now: Long): Long = + if (previous == null) now else maxOf(previous + 1, now) + private fun mergeNullableSums(a: Int?, b: Int?): Int? { if (a == null && b == null) return null return (a ?: 0) + (b ?: 0) diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt index 04d1e6250..8bd9ccc25 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt @@ -14,7 +14,7 @@ import java.io.File internal const val TELEMETRY_DATABASE_NAME = "vescape.db" internal const val LEGACY_TELEMETRY_DATABASE_NAME = "telemetry.db" // @parity /modules/vescape-core/ios/telemetry/DatabaseBackupManager.swift `TELEMETRY_SCHEMA_VERSION` -internal const val TELEMETRY_DATABASE_VERSION = 42 +internal const val TELEMETRY_DATABASE_VERSION = 48 @Database( entities = [ @@ -31,6 +31,9 @@ internal const val TELEMETRY_DATABASE_VERSION = 42 DiagnosticEventEntity::class, PrivacyZoneEntity::class, BoardWarningEntity::class, + SyncSequenceEntity::class, + SyncActionEntity::class, + SyncBindingEntity::class, VescFaultOccurrenceEntity::class, VescFaultCaptureEntity::class, VescFaultCaptureSampleEntity::class, @@ -572,6 +575,199 @@ abstract class TelemetryDatabase : RoomDatabase() { } } + /** + * Change Timestamps for the three tables that had none (#275). `boards` and `alerts` carried + * `created_at` only and `telemetry_minute_buckets` carried nothing, so a rename, a toggle or a + * bucket still filling was invisible to an "everything changed since T" scan. Additive, and + * existing rows are backfilled rather than left at the `DEFAULT 0` a scan would re-send. + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDatabase.swift `v43_sync_cursors` + */ + internal val MIGRATION_42_43 = object : Migration(42, 43) { + override fun migrate(db: SupportSQLiteDatabase) { + if (!hasColumn(db, "boards", "updated_at")) { + db.execSQL("ALTER TABLE boards ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + db.execSQL("UPDATE boards SET updated_at = created_at") + } + db.execSQL("CREATE INDEX IF NOT EXISTS index_boards_updated_at ON boards(updated_at)") + + if (!hasColumn(db, "alerts", "updated_at")) { + db.execSQL("ALTER TABLE alerts ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + db.execSQL("UPDATE alerts SET updated_at = created_at") + } + db.execSQL("CREATE INDEX IF NOT EXISTS index_alerts_updated_at ON alerts(updated_at)") + + if (!hasColumn(db, "telemetry_minute_buckets", "updated_at")) { + db.execSQL( + "ALTER TABLE telemetry_minute_buckets ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0", + ) + db.execSQL("UPDATE telemetry_minute_buckets SET updated_at = last_sample_at_ms") + } + db.execSQL( + "CREATE INDEX IF NOT EXISTS index_telemetry_minute_buckets_updated_at " + + "ON telemetry_minute_buckets(updated_at)", + ) + } + } + + /** + * Splits the device-local Sync Cursor from the wall-clock last-write-wins timestamp (#275). + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDatabase.swift `v44_sync_seq` + */ + internal val MIGRATION_43_44 = object : Migration(43, 44) { + override fun migrate(db: SupportSQLiteDatabase) { + db.execSQL( + """ + CREATE TABLE IF NOT EXISTS sync_sequences ( + name TEXT NOT NULL PRIMARY KEY, + last_value INTEGER NOT NULL + ) + """.trimIndent(), + ) + for (table in SYNC_SEQ_TABLES_V44) { + if (!hasColumn(db, table, "sync_seq")) { + db.execSQL("ALTER TABLE $table ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0") + db.execSQL("UPDATE $table SET sync_seq = rowid") + } + db.execSQL("CREATE INDEX IF NOT EXISTS index_${table}_sync_seq ON $table(sync_seq)") + db.execSQL( + "INSERT OR REPLACE INTO sync_sequences (name, last_value) " + + "VALUES ('$table', (SELECT COALESCE(MAX(sync_seq), 0) FROM $table))", + ) + } + } + } + + + /** + * Sync Cursors for the six remaining mutable tables (#281). `board_warnings` also gains the + * wall-clock `updated_at` every other mutable table already carries, backfilled from its newest + * detection. + * + * Existing rows are backfilled from `rowid` — distinct and non-zero, so no two rows share a + * cursor position and none of them sit at the seed value — and each table's sequence is seeded + * past the highest value handed out. Every step is guarded, so a re-run is a no-op. + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDatabase.swift `v45_sync_seq_remaining` + */ + internal val MIGRATION_44_45 = object : Migration(44, 45) { + override fun migrate(db: SupportSQLiteDatabase) { + if (!hasColumn(db, "board_warnings", "updated_at")) { + db.execSQL("ALTER TABLE board_warnings ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + db.execSQL("UPDATE board_warnings SET updated_at = last_detected_at") + } + + for (table in SYNC_SEQ_TABLES_V45) { + if (!hasColumn(db, table, "sync_seq")) { + db.execSQL("ALTER TABLE $table ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0") + db.execSQL("UPDATE $table SET sync_seq = rowid") + } + db.execSQL("CREATE INDEX IF NOT EXISTS index_${table}_sync_seq ON $table(sync_seq)") + db.execSQL( + "INSERT OR REPLACE INTO sync_sequences (name, last_value) " + + "VALUES ('$table', (SELECT COALESCE(MAX(sync_seq), 0) FROM $table))", + ) + } + + // Phone-local keys are defined by their absence from the scan, so the backfill above has to + // be undone for them: an uploader would otherwise ship whatever this phone happened to hold + // at upgrade time, exactly once. See NOT_SYNCED_SETTING_KEYS. + val phoneLocal = NOT_SYNCED_SETTING_KEYS.joinToString(",") { "'$it'" } + db.execSQL("UPDATE app_settings SET sync_seq = 0 WHERE key IN ($phoneLocal)") + } + } + + /** + * The Sync Action log (#282): an append-only record of semantic removals, which no surviving row + * can express. Additive — a new table only — and guarded, so a re-run is a no-op. + * + * The log is keyed on its own `AUTOINCREMENT` cursor and carries no `sync_seq`: SQLite + * guarantees that key monotonic and never reused, so it already *is* the cursor. + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDatabase.swift `v46_sync_actions` + */ + internal val MIGRATION_45_46 = object : Migration(45, 46) { + override fun migrate(db: SupportSQLiteDatabase) { + db.execSQL( + """ + CREATE TABLE IF NOT EXISTS sync_actions ( + id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + type TEXT NOT NULL, + target TEXT NOT NULL, + board_id TEXT, + key TEXT NOT NULL, + deleted_at INTEGER NOT NULL + ) + """.trimIndent(), + ) + db.execSQL("CREATE INDEX IF NOT EXISTS index_sync_actions_target ON sync_actions(target)") + } + } + + + /** + * The Account binding (#284): which Vescape Account this local database belongs to. Additive and + * guarded, and deliberately left empty — an existing install is unbound until an Account signs + * in and claims it, which is also what keeps the current age-only retention behaviour until + * then. + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDatabase.swift `v47_sync_binding` + */ + internal val MIGRATION_46_47 = object : Migration(46, 47) { + override fun migrate(db: SupportSQLiteDatabase) { + db.execSQL( + """ + CREATE TABLE IF NOT EXISTS sync_binding ( + id INTEGER PRIMARY KEY NOT NULL, + account_id TEXT NOT NULL, + bound_at INTEGER NOT NULL + ) + """.trimIndent(), + ) + } + } + + /** + * VESC Fault Evidence joins the backup (#288). Occurrences and Captures are given the Sync + * Cursor every mutable table carries, and an Occurrence also gains the wall-clock `updated_at` + * the server compares two writes to the same row on. + * + * The stamp is backfilled from `last_observed_at` rather than left at the `DEFAULT 0`: an + * existing occurrence has a truthful moment it last changed, and a row reporting epoch zero + * loses every race against whatever the server already holds. + * + * A Capture needs no stamp — it is a past snapshot with no lifecycle — and + * `vesc_fault_capture_samples` needs nothing at all: it is append-only on an `AUTOINCREMENT` + * id, which already is its cursor. + * + * Every step is guarded, so a re-run is a no-op. + * + * @parity /modules/vescape-core/ios/telemetry/TelemetryDatabase.swift `v48_fault_sync` + */ + internal val MIGRATION_47_48 = object : Migration(47, 48) { + override fun migrate(db: SupportSQLiteDatabase) { + if (!hasColumn(db, "vesc_fault_occurrences", "updated_at")) { + db.execSQL( + "ALTER TABLE vesc_fault_occurrences ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0", + ) + db.execSQL("UPDATE vesc_fault_occurrences SET updated_at = last_observed_at") + } + + for (table in SYNC_SEQ_TABLES_V48) { + if (!hasColumn(db, table, "sync_seq")) { + db.execSQL("ALTER TABLE $table ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0") + db.execSQL("UPDATE $table SET sync_seq = rowid") + } + db.execSQL("CREATE INDEX IF NOT EXISTS index_${table}_sync_seq ON $table(sync_seq)") + db.execSQL( + "INSERT OR REPLACE INTO sync_sequences (name, last_value) " + + "VALUES ('$table', (SELECT COALESCE(MAX(sync_seq), 0) FROM $table))", + ) + } + } + } + /** * Telemetry whose `device_id` matches no Board would lose both its identity and its label: * either the Board was hard-deleted before tombstones existed (ADR 0027), or it was re-linked @@ -1412,6 +1608,12 @@ abstract class TelemetryDatabase : RoomDatabase() { MIGRATION_36_40, MIGRATION_40_41, MIGRATION_41_42, + MIGRATION_42_43, + MIGRATION_43_44, + MIGRATION_44_45, + MIGRATION_45_46, + MIGRATION_46_47, + MIGRATION_47_48, ) .fallbackToDestructiveMigration(true) .addCallback(object : Callback() { diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt index 84d83c0f9..31dd8f73a 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt @@ -110,7 +110,11 @@ data class TelemetryFrameEntity( @Entity( tableName = "telemetry_minute_buckets", primaryKeys = ["bucket_start_ms", "board_id"], - indices = [Index(value = ["bucket_start_ms"])], + indices = [ + Index(value = ["bucket_start_ms"]), + Index(value = ["updated_at"]), + Index(value = ["sync_seq"]), + ], ) data class TelemetryMinuteBucketEntity( @ColumnInfo(name = "bucket_start_ms") @@ -172,6 +176,19 @@ data class TelemetryMinuteBucketEntity( val firstMovingAtMs: Long? = null, @ColumnInfo(name = "last_moving_at_ms") val lastMovingAtMs: Long? = null, + /** + * Last-write-wins timestamp: wall-clock epoch ms of the last write to this bucket. Distinct from + * [lastSampleAtMs], which tracks the newest *sample* in the bucket — a merge that folds in older + * samples, or a bucket rebuild, changes the row without moving that. + * + * Not the Sync Cursor column; [syncSeq] is. This one crosses the wire and decides which of two + * writes to the same row the server keeps, so it stays a truthful wall clock. + */ + @ColumnInfo(name = "updated_at") + val updatedAt: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) @Entity( @@ -228,6 +245,8 @@ data class DiagnosticEventEntity( tableName = "boards", indices = [ Index(value = ["created_at"]), + Index(value = ["updated_at"]), + Index(value = ["sync_seq"]), ], ) data class BoardEntity( @@ -238,12 +257,28 @@ data class BoardEntity( val bleId: String?, @ColumnInfo(name = "created_at") val createdAt: Long, + /** + * Last-write-wins timestamp: epoch ms of the last write to this row, from the same clock as + * [createdAt]. Equal to [createdAt] on insert and bumped on every mutation. It crosses the wire + * and is what the server compares to decide which of two writes to this row it keeps, so it stays + * a truthful wall clock rather than a counter. + * + * Ratcheted to `max(previous + 1, now)` on write. A device clock that steps backwards would + * otherwise stamp an edit below the copy the server already holds, and the server's + * last-write-wins guard would silently drop it. Per row, so the inflation is bounded by the + * rewind and disappears once the wall clock passes it again. + */ + @ColumnInfo(name = "updated_at") + val updatedAt: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, /** * Tombstone stamp: epoch ms of the rider's delete, null while the Board is alive. A deleted Board - * keeps its row so Ride History can still name it; only the Board's configuration is - * hard-deleted (ADR-0027). + * keeps its row so Ride History can still name it and the server's Board-owned foreign keys hold; + * only the Board's configuration is hard-deleted (ADR-0027). * - * Written by the delete path only — an upsert from the bridge never authors it. + * Written by the delete path only — an upsert from the bridge never authors it, like [updatedAt]. */ @ColumnInfo(name = "deleted_at") val deletedAt: Long? = null, @@ -260,6 +295,7 @@ data class BoardNameRow( primaryKeys = ["board_id", "key"], indices = [ Index(value = ["board_id"]), + Index(value = ["sync_seq"]), ], ) data class BoardSettingEntity( @@ -268,8 +304,12 @@ data class BoardSettingEntity( val key: String, @ColumnInfo(name = "value_json") val valueJson: String, + /** Ratcheted last-write-wins timestamp; see [BoardEntity.updatedAt]. */ @ColumnInfo(name = "updated_at") val updatedAt: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) @Entity( @@ -280,6 +320,8 @@ data class BoardSettingEntity( Index(value = ["control_id"]), Index(value = ["enabled"]), Index(value = ["created_at"]), + Index(value = ["updated_at"]), + Index(value = ["sync_seq"]), ], ) data class AlertRuleEntity( @@ -314,6 +356,196 @@ data class AlertRuleEntity( * JS authors and regenerates preset rules; native only persists the string. */ val source: String?, + /** + * Last-write-wins timestamp, ratcheted on write exactly as [BoardEntity.updatedAt] is, and moved + * by every mutation including the targeted enable/disable update. + */ + @ColumnInfo(name = "updated_at") + val updatedAt: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, +) + +/** + * One counter per syncable table, handing out the strictly increasing `sync_seq` those tables stamp + * on every write. + * + * The Sync Cursor is the phone's own record of how far it has uploaded, and it never crosses the + * wire — the server stores no watermark and has no opinion about one. That is what lets the scan + * run on a counter instead of a clock: a device clock that steps backwards makes an + * `updated_at >= watermark` scan skip the write entirely, because the row lands below a cursor the + * phone already passed. A counter cannot regress, so the scan stays complete however the clock + * behaves. + * + * The counter lives in its own table rather than being derived as `MAX(sync_seq) + 1` per table: + * deleting the highest row would hand the same number out twice, and the second row would fall on + * the wrong side of a cursor already advanced past it. + */ +@Entity(tableName = "sync_sequences") +data class SyncSequenceEntity( + @PrimaryKey + val name: String, + @ColumnInfo(name = "last_value") + val lastValue: Long, +) + +/** Table names used as [SyncSequenceEntity] keys. */ +internal const val SYNC_SEQ_BOARDS = "boards" +internal const val SYNC_SEQ_ALERTS = "alerts" +internal const val SYNC_SEQ_MINUTE_BUCKETS = "telemetry_minute_buckets" +internal const val SYNC_SEQ_APP_SETTINGS = "app_settings" +internal const val SYNC_SEQ_BOARD_SETTINGS = "board_settings" +internal const val SYNC_SEQ_BOARD_WARNINGS = "board_warnings" +internal const val SYNC_SEQ_PRIVACY_ZONES = "privacy_zones" +internal const val SYNC_SEQ_TUNE_PROFILES = "tune_profiles" +internal const val SYNC_SEQ_FAVORITES = "favorites" +internal const val SYNC_SEQ_VESC_FAULT_OCCURRENCES = "vesc_fault_occurrences" +internal const val SYNC_SEQ_VESC_FAULT_CAPTURES = "vesc_fault_captures" + +/** + * The three tables the schema-44 migration gave a `sync_seq`, frozen at the set that existed then. + * A migration iterates the tables it actually shipped with, never the current [SYNC_SEQ_TABLES] — + * growing that list must not retroactively change what an older migration step does. + */ +internal val SYNC_SEQ_TABLES_V44 = listOf( + SYNC_SEQ_BOARDS, + SYNC_SEQ_ALERTS, + SYNC_SEQ_MINUTE_BUCKETS, +) + +/** The six remaining mutable tables, given a `sync_seq` at schema 45 (#281). */ +internal val SYNC_SEQ_TABLES_V45 = listOf( + SYNC_SEQ_APP_SETTINGS, + SYNC_SEQ_BOARD_SETTINGS, + SYNC_SEQ_BOARD_WARNINGS, + SYNC_SEQ_PRIVACY_ZONES, + SYNC_SEQ_TUNE_PROFILES, + SYNC_SEQ_FAVORITES, +) + +/** + * VESC Fault Evidence, given a `sync_seq` at schema 48. `vesc_fault_capture_samples` is absent for + * the usual reason: it is append-only and already keyed on an `AUTOINCREMENT` id. + */ +internal val SYNC_SEQ_TABLES_V48 = listOf( + SYNC_SEQ_VESC_FAULT_OCCURRENCES, + SYNC_SEQ_VESC_FAULT_CAPTURES, +) + +/** + * Every table carrying a `sync_seq`. Append-only tables are deliberately absent: they declare + * `INTEGER PRIMARY KEY AUTOINCREMENT`, which SQLite guarantees monotonic and never reused, so their + * key already *is* their cursor. + */ +internal val SYNC_SEQ_TABLES = SYNC_SEQ_TABLES_V44 + SYNC_SEQ_TABLES_V45 + SYNC_SEQ_TABLES_V48 + +/** + * What a [SyncActionEntity] can name — and, by omission, what it cannot. + * + * Every case is configuration or current state a Rider edits directly. Ride History is absent on + * purpose: Telemetry Samples, markers, minute buckets, exclusion ranges and diagnostic events are + * pruned on a retention rule, and an action naming one of those would make the server delete exactly + * the rides the backup exists to preserve. Leaving them unnameable makes that boundary structural + * rather than a rule someone has to remember (server ADR-0004). + * + * [table] is the local table the case removes from, so a test can assert no retained table is ever + * given a case. + * + * @parity /modules/vescape-core/ios/telemetry/SyncActionLog.swift `DeleteTarget` + * @parity /modules/vescape-core/src/index.ts `DeleteTarget` + */ +enum class DeleteTarget(val wire: String, val table: String) { + APP_SETTING("appSetting", "app_settings"), + BOARD("board", "boards"), + BOARD_SETTING("boardSetting", "board_settings"), + BOARD_WARNING("boardWarning", "board_warnings"), + ALERT("alert", "alerts"), + TUNE_PROFILE("tuneProfile", "tune_profiles"), + PRIVACY_ZONE("privacyZone", "privacy_zones"), + + FAVORITE("favorite", "favorites"), +} + +/** The only Sync Action type today. Named rather than implied so a later intent needs no second log. */ +internal const val SYNC_ACTION_TYPE_DELETE = "delete" + +/** + * One Sync Action: an append-only record that something was semantically removed. A deleted row + * cannot carry a Change Timestamp saying it is gone, so this log is the only signal the server can + * apply the same durable state transition from. + * + * Its cursor is [id] — `AUTOINCREMENT`, which SQLite guarantees monotonic and never reused — so the + * log needs no `sync_seq` of its own. The row is transport state, not durable truth: it is pruned + * once the server has accepted it. + * + * Written only from Rider-facing removal paths, never from a trigger or a retention sweep. Intent + * cannot be inferred from SQL alone, so there is no database trigger behind this table. + * + * @parity /modules/vescape-core/ios/telemetry/SyncActionLog.swift `createSyncActionsTable` + */ +@Entity( + tableName = "sync_actions", + indices = [Index(value = ["target"])], +) +data class SyncActionEntity( + @PrimaryKey(autoGenerate = true) + val id: Long = 0, + /** Always [SYNC_ACTION_TYPE_DELETE] today; see [DeleteTarget]. */ + val type: String = SYNC_ACTION_TYPE_DELETE, + /** [DeleteTarget.wire]. */ + val target: String, + /** Owning Board, or null when the target is not Board-owned. A Board names itself in [key]. */ + @ColumnInfo(name = "board_id") + val boardId: String?, + /** The removed row's identity within its scope: a settings key, a warning kind, a row id. */ + val key: String, + /** + * Epoch ms of the removal, stamped `max(now, row.updated_at)` from the row being removed. A plain + * `now` on a rewound clock produces an action the server treats as a no-op, and it cannot + * self-heal by re-sending because the row it would re-send is gone. + */ + @ColumnInfo(name = "deleted_at") + val deletedAt: Long, +) + +/** [SyncSequenceEntity] key holding the highest action cursor the server has accepted. */ +internal const val SYNC_ACTIONS_UPLOADED_CURSOR = "sync_actions_uploaded" + +/** + * [SyncSequenceEntity] keys holding how far each table has been accepted — the Sync Cursors the + * uploader commits and cursor-gated retention reads back. Prefixed so a cursor can never collide + * with the write counters, which are keyed on the bare table name. + * + * The five below are the retained tables; every other table's key is derived the same way from + * `SyncTable`, and a test pins the two spellings together. + */ +internal const val SYNC_CURSOR_PREFIX = "sync_cursor_" +internal const val SYNC_CURSOR_FRAMES = "sync_cursor_telemetry_frames" +internal const val SYNC_CURSOR_MARKERS = "sync_cursor_telemetry_markers" +internal const val SYNC_CURSOR_MINUTE_BUCKETS = "sync_cursor_telemetry_minute_buckets" +internal const val SYNC_CURSOR_DIAGNOSTIC_EVENTS = "sync_cursor_diagnostic_events" +internal const val SYNC_CURSOR_EXCLUSION_RANGES = "sync_cursor_metric_exclusion_ranges" + +/** + * Which Vescape Account this local database belongs to. One row, claimed by the first Account to + * sign in and never rewritten in place: a different Account replaces the whole database, because + * resetting the cursors over these rows would upload the previous Account's Boards, Ride History, + * locations and settings to the new one. + * + * Signing out does not clear the binding, so data recorded while signed out keeps its retention + * protection for the same Account. + * + * @parity /modules/vescape-core/ios/sync/SyncStore.swift `createSyncBindingTable` + */ +@Entity(tableName = "sync_binding") +data class SyncBindingEntity( + @PrimaryKey + val id: Int = 0, + @ColumnInfo(name = "account_id") + val accountId: String, + @ColumnInfo(name = "bound_at") + val boundAt: Long, ) @Entity( @@ -340,6 +572,9 @@ data class MetricExclusionRangeEntity( @Entity( tableName = "privacy_zones", + indices = [ + Index(value = ["sync_seq"]), + ], ) data class PrivacyZoneEntity( @PrimaryKey @@ -355,18 +590,78 @@ data class PrivacyZoneEntity( val radiusMeters: Int, @ColumnInfo(name = "created_at") val createdAt: Long, + /** Ratcheted last-write-wins timestamp; see [BoardEntity.updatedAt]. */ @ColumnInfo(name = "updated_at") val updatedAt: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) -@Entity(tableName = "app_settings") +@Entity( + tableName = "app_settings", + indices = [ + Index(value = ["sync_seq"]), + ], +) data class AppSettingEntity( @PrimaryKey val key: String, @ColumnInfo(name = "value_json") val valueJson: String, + /** Ratcheted last-write-wins timestamp; see [BoardEntity.updatedAt]. */ @ColumnInfo(name = "updated_at") val updatedAt: Long, + /** + * Device-local Sync Cursor position; see [SyncSequenceEntity]. Stays 0 — below every cursor, so + * invisible to the upload scan — for the keys in [NOT_SYNCED_SETTING_KEYS]. + */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, +) + +/** + * App settings that name *this phone* rather than the Rider, and so never leave it: restoring them + * onto a second phone would overwrite that phone's own identity or session state. Enforced at the + * write path — [TelemetryDao.upsertAppSetting] leaves their `sync_seq` at 0, which is below every + * Sync Cursor, so no upload scan ever sees the row. + * + * Rider Name and Rider Color live in `app_settings` by design, so that Group Ride keeps working + * signed-out; that placement is what makes them phone-local rather than Account-scoped. See #277. + * + * @parity /modules/vescape-core/ios/telemetry/AppDataRepository.swift `notSyncedSettingKeys` + */ +internal val NOT_SYNCED_SETTING_KEYS = setOf( + // Rider identity — a second phone in the same Group Ride must not become the same Rider. + "riderId", + "riderName", + "riderColor", + // Device/session state — names this phone's current session, not the Rider's configuration. + "selectedBoardId", + "lastGpsLatitude", + "lastGpsLongitude", + "directionPointLatitude", + "directionPointLongitude", + // Connection and companion behaviour — phone-side BLE and foreground policy. + "autoConnect", + "companionPresenceEnabled", + "companionPresenceCooldownMinutes", + "connectionSoundsEnabled", + "autoCloseEnabled", + "autoCloseDelayMinutes", + // Wear pairing — the watch is paired to one phone. + "wearMirrorIntervalMs", + "wearAutoLaunchOnConnect", + // The backup master switch is per phone, and deliberately does not travel through the mechanism + // it turns off: a restored snapshot must never be able to switch backup back on. + "syncEnabled", + // The backup choice is per phone: the expensive first upload belongs to the phone that holds the + // backlog, so a restore onto a second phone asks that Rider again rather than deciding for them. + "syncBackupChoiceMade", + // So is the data-plan choice, and for the same reason the other two are: it answers what this + // phone's connection costs, not what the Rider prefers. Travelling would let a restore onto a + // cellular-only phone inherit the other phone's answer and upload a ride over metered data. + "syncWifiOnly", ) /** @@ -412,6 +707,12 @@ data class AppSettings( val companionPresenceCooldownMinutes: Int = 60, val autoCloseEnabled: Boolean = false, val autoCloseDelayMinutes: Int = 15, + /** Backup master switch. Off by default: the uploader does nothing until the Rider turns it on. */ + val syncEnabled: Boolean = false, + /** Nothing uploads on a metered connection while this is on — mid-ride included. */ + val syncWifiOnly: Boolean = false, + /** The one-time backup choice has been offered on this phone and answered. */ + val syncBackupChoiceMade: Boolean = false, val riderId: String? = null, val riderName: String? = null, val riderColor: String? = null, @@ -424,6 +725,7 @@ data class AppSettings( indices = [ Index(value = ["board_id"]), Index(value = ["board_id", "refloat_base_version"]), + Index(value = ["sync_seq"]), ], ) data class TuneProfileEntity( @@ -440,8 +742,12 @@ data class TuneProfileEntity( val fieldsJson: String, @ColumnInfo(name = "created_at") val createdAt: Long, + /** Ratcheted last-write-wins timestamp; see [BoardEntity.updatedAt]. */ @ColumnInfo(name = "updated_at") val updatedAt: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) @Entity( @@ -474,6 +780,7 @@ data class TuneHistoryEntryEntity( primaryKeys = ["board_id", "kind"], indices = [ Index(value = ["board_id"]), + Index(value = ["sync_seq"]), ], ) data class BoardWarningEntity( @@ -487,6 +794,16 @@ data class BoardWarningEntity( val lastDetectedAt: Long, @ColumnInfo(name = "payload_json") val payloadJson: String, + /** + * Ratcheted last-write-wins timestamp; see [BoardEntity.updatedAt]. Distinct from + * [lastDetectedAt], which moves only when the detector fires — a severity or payload change + * rewrites the row without necessarily being a fresh detection. + */ + @ColumnInfo(name = "updated_at") + val updatedAt: Long = 0, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) /** @@ -504,6 +821,7 @@ data class BoardWarningEntity( indices = [ Index(value = ["start_ms", "end_ms"]), Index(value = ["board_id"]), + Index(value = ["sync_seq"]), ], ) data class FavoriteEntity( @@ -539,6 +857,9 @@ data class FavoriteEntity( val maxSpeedCentiKmh: Int, @ColumnInfo(name = "battery_used_wh_milli") val batteryUsedWhMilli: Long, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) { /** * Board name is resolved on read from `boards`, not snapshotted, so renames propagate. @@ -690,6 +1011,7 @@ data class BoardConfigChangeNoticeEntity( tableName = "vesc_fault_occurrences", indices = [ Index(value = ["board_id", "occurred_at"]), + Index(value = ["sync_seq"]), ], ) data class VescFaultOccurrenceEntity( @@ -710,6 +1032,17 @@ data class VescFaultOccurrenceEntity( val clearedAtMs: Long?, /** Rider acknowledged this occurrence: stays durable, stops driving the fault icon. */ val dismissed: Boolean, + /** + * Ratcheted last-write-wins timestamp; see [BoardEntity.updatedAt]. Distinct from + * [lastObservedAtMs], which moves only when the controller reported the code again — a Rider + * dismissing an occurrence rewrites the row without the fault ever being observed a second time, + * and that edit is precisely what a restore has to preserve. + */ + @ColumnInfo(name = "updated_at") + val updatedAt: Long = 0, + /** Device-local Sync Cursor position; see [SyncSequenceEntity]. */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) /** * Metadata for one VESC Fault Capture: the self-contained window of decoded Board samples a single @@ -725,6 +1058,7 @@ data class VescFaultOccurrenceEntity( tableName = "vesc_fault_captures", indices = [ Index(value = ["board_id"]), + Index(value = ["sync_seq"]), ], ) data class VescFaultCaptureEntity( @@ -742,6 +1076,13 @@ data class VescFaultCaptureEntity( /** Samples actually retained — the achieved Board Session rate, never a fabricated cadence. */ @ColumnInfo(name = "sample_count") val sampleCount: Int, + /** + * Device-local Sync Cursor position; see [SyncSequenceEntity]. A Capture carries no change + * timestamp — it is a past snapshot with no lifecycle — so the counter alone decides when the + * uploader has caught up with it. + */ + @ColumnInfo(name = "sync_seq") + val syncSeq: Long = 0, ) /** diff --git a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryRepository.kt b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryRepository.kt index 385a12d9e..123ef46e5 100644 --- a/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryRepository.kt +++ b/modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryRepository.kt @@ -4,6 +4,7 @@ import android.content.Context import android.os.SystemClock import android.util.Log import expo.modules.kotlin.jni.NativeArrayBuffer +import expo.modules.vescapecore.sync.SyncCoordinator import java.nio.ByteBuffer import java.nio.ByteOrder import java.util.UUID @@ -557,8 +558,13 @@ class TelemetryRepository private constructor(context: Context) { dao.clearDiagnosticEvents() } + /** + * Retention sweep. Age-only while this database has never been bound to an Account, and age plus + * the accepted Sync Cursor once it has — cleanup must not remove a row the uploader has not + * delivered (#284). + */ suspend fun deleteBefore(beforeMs: Long): Int = withContext(Dispatchers.IO) { - dao.deleteBefore(beforeMs) + dao.deleteBeforeGated(beforeMs) } suspend fun deleteRange(options: Map): Int = withContext(Dispatchers.IO) { @@ -583,7 +589,7 @@ class TelemetryRepository private constructor(context: Context) { */ suspend fun getFavorites(): List> = withContext(Dispatchers.IO) { favoriteMediaStore.reconcileAll() - val boardNames = dao.getBoards().associate { it.id to it.name } + val boardNames = boardNamesById() dao.getFavorites().map { favorite -> favorite.toMap(boardNames[favorite.boardId], favoriteRoutePoints(favorite)) } @@ -642,6 +648,7 @@ class TelemetryRepository private constructor(context: Context) { batteryUsedWhMilli = summary.batteryUsedWhMilli, ) dao.insertFavorite(favorite) + SyncCoordinator.get(appContext).notifyRiderEdit() favorite.toMap( boardId?.let { boardNamesById()[it] }, favoriteRoutePoints(favorite), @@ -681,8 +688,9 @@ class TelemetryRepository private constructor(context: Context) { batteryUsedWhMilli = summary.batteryUsedWhMilli, ) if (dao.updateFavorite(updated) == 0) return@withContext null + SyncCoordinator.get(appContext).notifyRiderEdit() updated.toMap( - dao.getBoards().firstOrNull { it.id == updated.boardId }?.name, + updated.boardId?.let { boardNamesById()[it] }, favoriteRoutePoints(updated), ) } @@ -694,7 +702,10 @@ class TelemetryRepository private constructor(context: Context) { */ suspend fun deleteFavorite(id: String): Boolean = withContext(Dispatchers.IO) { val deleted = dao.deleteFavorite(id) > 0 - if (deleted) favoriteMediaStore.deleteDirectory(id) + if (deleted) { + favoriteMediaStore.deleteDirectory(id) + SyncCoordinator.get(appContext).notifyRiderEdit() + } deleted } @@ -915,6 +926,9 @@ class TelemetryRepository private constructor(context: Context) { markers = markers, exclusions = sanitization.exclusions, ) + // Samples are actually being produced, which is what the uploader's ride cadence follows — + // Idle Pause halts production without ending the Board Session. + SyncCoordinator.get(appContext).notifySamplesPersisted() } catch (e: Exception) { Log.w(TAG, "Telemetry flush failed: ${e.message}") } diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/alerts/AlertEngineTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/alerts/AlertEngineTest.kt index afdbd5843..7208b9447 100644 --- a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/alerts/AlertEngineTest.kt +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/alerts/AlertEngineTest.kt @@ -39,6 +39,7 @@ class AlertEngineTest { repeatEverySeconds = repeatEverySeconds, beepCount = beepCount, source = null, + updatedAt = 0L, ) @Test fun `config relative duty resolves fraction and follows updates`() { diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncServer.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncServer.kt new file mode 100644 index 000000000..88d4aaf10 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncServer.kt @@ -0,0 +1,71 @@ +package expo.modules.vescapecore.sync + +import org.json.JSONObject + +/** + * A server that stores what it is sent and answers with the accepted map the real one would. + * + * It exists so a test can assert the only thing that matters end to end: every row the Rider owns + * reached the server. The engine's own return values cannot show that — a cursor advanced past a row + * that was never in a batch reports `Sent` and looks identical to a correct pass. + * + * Stores rows by identity and upserts, exactly like the real one, so a re-send after a lost + * checkpoint is a no-op rather than a duplicate. + * + * @parity /modules/vescape-core/ios/sync/FakeSyncServer.swift `FakeSyncServer` + */ +class FakeSyncServer : SyncTransport { + + /** Every row the server holds, by table and cursor. */ + val stored = mutableSetOf>() + + /** Bodies received, including the ones answered with a failure. */ + val received = mutableListOf() + + /** Rows written, counting re-sends — the cost of failing toward re-sending. */ + var writes = 0 + private set + + /** Queued failures, consumed one per request before the server stores anything. */ + val failures = ArrayDeque() + + /** Fires after the batch is stored but before the response is returned. */ + var afterStore: (() -> Unit)? = null + + /** + * Store the next batch and then answer as if the response never arrived. The engine cannot tell + * this apart from a batch that was never applied, which is exactly why it re-sends. + */ + var loseNextResponse = false + + private val byWire = SyncTable.entries.associateBy { it.wire } + + override suspend fun send(body: String): SyncResponse { + received += body + failures.removeFirstOrNull()?.let { return it } + + val batch = JSONObject(body) + val counts = LinkedHashMap() + for (wire in batch.keys()) { + val table = byWire.getValue(wire) + val rows = batch.getJSONArray(wire) + counts[table] = rows.length() + for (i in 0 until rows.length()) { + stored += table to rows.getJSONObject(i).getLong("c") + writes += 1 + } + } + afterStore?.invoke() + if (loseNextResponse) { + loseNextResponse = false + return SyncResponse.Transient("timeout") + } + return SyncResponse.Accepted(accepted(counts)) + } + + /** The server answers for every table it knows, not only the ones the batch carried. */ + private fun accepted(counts: Map): String { + val body = SyncTable.entries.joinToString(",") { "\"${it.wire}\":${counts[it] ?: 0}" } + return "{\"accepted\":{$body}}" + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncSource.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncSource.kt new file mode 100644 index 000000000..3b43c44b5 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncSource.kt @@ -0,0 +1,87 @@ +package expo.modules.vescapecore.sync + +/** + * A [SyncSource] that models the database the way [SyncStore] actually behaves: rows keyed by their + * cursor position, a scan that serves strictly `cursor > committed` in ascending order, and a commit + * that ratchets each table's cursor forward and never backwards. + * + * The point of modelling it rather than counting is that the loss bug is a cursor bug. A source that + * decrements a row counter accepts an advance set that names the wrong position — the exact mistake + * that makes a row unreachable forever — because the next scan is not derived from what was + * committed. Here it is, so a wrong advance shows up as a row nobody is ever offered again. + * + * @parity /modules/vescape-core/ios/sync/FakeSyncSource.swift `FakeSyncSource` + */ +class FakeSyncSource(seed: Map> = emptyMap()) : SyncSource { + + constructor(rows: Int) : this(mapOf(SyncTable.BOARDS to (1L..rows.toLong()).toList())) + + /** Cursor positions present per table, ascending — the rows on disk. */ + private val rows: Map> = + seed.mapValues { (_, positions) -> positions.sorted() } + + /** How far each table has been accepted. Absent means nothing delivered. */ + val cursors = mutableMapOf() + + /** Every advance set handed to [commit], in order. */ + val committed = mutableListOf>() + + /** Every row the scan handed out, in order, across every pass. Re-sends appear more than once. */ + val offered = mutableListOf>() + + var generation = 0L + val failures = mutableListOf>() + var encodeFailure: SyncProtocolException? = null + var commitFailure: Exception? = null + /** Caps one scan below the engine's own row limit, so a drain takes more than one pass. */ + var scanLimit = Int.MAX_VALUE + + /** + * Each row carries its own cursor on the wire, so a fake server can report back exactly which + * rows it stored. Without row identity in the body, "the server received everything" is not a + * claim a test can make. + */ + private fun rowJson(cursor: Long) = "{\"c\":$cursor}" + + /** Rows the scan will still offer. Zero means the backlog is drained. */ + val remaining: Int + get() = rows.entries.sumOf { (table, positions) -> positions.count { it > cursorOf(table) } } + + private fun cursorOf(table: SyncTable): Long = cursors[table] ?: 0L + + /** Mirrors [SyncStore.pending]: table order, one shared row budget, forward from each cursor. */ + override suspend fun pending(rowLimit: Int): List { + encodeFailure?.let { throw it } + val tables = mutableListOf() + var budget = minOf(rowLimit, scanLimit) + for (table in SyncTable.entries) { + if (budget <= 0) break + val positions = rows[table] ?: continue + val pending = positions.filter { it > cursorOf(table) }.take(budget) + if (pending.isEmpty()) continue + pending.forEach { offered += table to it } + tables += SyncPendingTable(table, pending.map { SyncPendingRow(it, rowJson(it)) }) + budget -= pending.size + } + return tables + } + + override suspend fun pendingCount(): Int = remaining + + /** Mirrors `commitSyncCursor`: `MAX(existing, incoming)`, so a cursor never moves backwards. */ + override suspend fun commit(advances: Map) { + commitFailure?.let { throw it } + committed += advances + for ((table, cursor) in advances) cursors[table] = maxOf(cursorOf(table), cursor) + } + + override fun generation(): Long = generation + + override suspend fun recordPermanentFailure(reason: SyncPauseReason, detail: String) { + failures += reason to detail + } + + /** Every row on disk, for the loss invariant. */ + fun allRows(): Set> = + rows.entries.flatMap { (table, positions) -> positions.map { table to it } }.toSet() +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncAcceptedTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncAcceptedTest.kt new file mode 100644 index 000000000..7104fc110 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncAcceptedTest.kt @@ -0,0 +1,57 @@ +package expo.modules.vescapecore.sync + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The `200` body is the last thing standing between an accepted batch and a cursor that can never be + * walked back, so it is validated exactly rather than trusted. + * + * @parity /modules/vescape-core/ios/sync/SyncAcceptedTests.swift + */ +class SyncAcceptedTest { + private fun body(counts: Map = emptyMap(), tables: List = SyncTable.entries): String { + val pairs = tables.joinToString(",") { "\"${it.wire}\":${counts[it] ?: 0}" } + return "{\"accepted\":{$pairs}}" + } + + @Test + fun `every table accounted for parses`() { + val parsed = SyncAccepted.parse(body(mapOf(SyncTable.BOARDS to 3))) + assertEquals(3, parsed?.get(SyncTable.BOARDS)) + assertEquals(0, parsed?.get(SyncTable.FAVORITES)) + } + + @Test + fun `a missing table, an extra table or a duplicate is refused`() { + assertNull(SyncAccepted.parse(body(tables = SyncTable.entries.drop(1)))) + assertNull(SyncAccepted.parse("{\"accepted\":{\"unknownTable\":0}}")) + assertNull(SyncAccepted.parse("{\"accepted\":{\"boards\":1,\"boards\":1}}")) + } + + @Test + fun `anything that is not this response is refused rather than half-read`() { + assertNull(SyncAccepted.parse("")) + assertNull(SyncAccepted.parse("{}")) + assertNull(SyncAccepted.parse("{\"ok\":true}")) + assertNull(SyncAccepted.parse(body() + "trailing")) + } + + @Test + fun `counts have to equal what was submitted, table by table`() { + val submitted = mapOf(SyncTable.BOARDS to 2) + assertTrue(SyncAccepted.matches(submitted, SyncAccepted.parse(body(submitted))!!)) + assertFalse( + SyncAccepted.matches(submitted, SyncAccepted.parse(body(mapOf(SyncTable.BOARDS to 1)))!!), + ) + assertFalse( + SyncAccepted.matches( + submitted, + SyncAccepted.parse(body(mapOf(SyncTable.BOARDS to 2, SyncTable.ALERTS to 1)))!!, + ), + ) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncBatchBuilderTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncBatchBuilderTest.kt new file mode 100644 index 000000000..f83a9c346 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncBatchBuilderTest.kt @@ -0,0 +1,145 @@ +package expo.modules.vescapecore.sync + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The batch builder is pure: no database, no clock, no network. What it has to get right is the + * order tables go out in, the two caps, and an advance set that describes exactly the rows sent. + * + * @parity /modules/vescape-core/ios/sync/SyncBatchBuilderTests.swift + */ +class SyncBatchBuilderTest { + private fun rows(count: Int, size: Int = 10, from: Long = 1): List = + (0 until count).map { SyncPendingRow(from + it, "\"" + "x".repeat(size) + "\"") } + + @Test + fun `walks server table order regardless of backlog size`() { + val built = SyncBatchBuilder.build( + listOf( + SyncPendingTable(SyncTable.TELEMETRY_FRAMES, rows(5)), + SyncPendingTable(SyncTable.BOARDS, rows(1)), + SyncPendingTable(SyncTable.APP_SETTINGS, rows(1)), + ), + ) as SyncBatchBuild.Ready + + assertEquals( + listOf(SyncTable.APP_SETTINGS, SyncTable.BOARDS, SyncTable.TELEMETRY_FRAMES), + built.counts.keys.toList(), + ) + assertTrue(built.body.indexOf("appSettings") < built.body.indexOf("boards")) + assertTrue(built.body.indexOf("boards") < built.body.indexOf("telemetryFrames")) + } + + @Test + fun `advance set names the last row actually included, per table`() { + val built = SyncBatchBuilder.build( + listOf( + SyncPendingTable(SyncTable.BOARDS, rows(2, from = 40)), + SyncPendingTable(SyncTable.FAVORITES, rows(3, from = 7)), + ), + rowCap = 4, + ) as SyncBatchBuild.Ready + + assertEquals(4, built.rowCount) + assertEquals(mapOf(SyncTable.BOARDS to 2, SyncTable.FAVORITES to 2), built.counts) + assertEquals(mapOf(SyncTable.BOARDS to 41L, SyncTable.FAVORITES to 8L), built.advances) + } + + @Test + fun `exactly-at and one-over the row cap behave the same way on every platform`() { + val atCap = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, rows(3))), + rowCap = 3, + ) as SyncBatchBuild.Ready + assertEquals(3, atCap.rowCount) + + val overCap = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, rows(4))), + rowCap = 3, + ) as SyncBatchBuild.Ready + assertEquals(3, overCap.rowCount) + assertEquals(3L, overCap.advances.getValue(SyncTable.BOARDS)) + } + + /** The cap is on the bytes actually sent, so the encoded body is what gets measured. */ + @Test + fun `byte cap counts the encoded body, boundary included`() { + val one = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, rows(2, size = 8))), + byteCap = Int.MAX_VALUE, + ) as SyncBatchBuild.Ready + assertEquals(one.body.toByteArray(Charsets.UTF_8).size, one.byteCount) + + val atCap = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, rows(2, size = 8))), + byteCap = one.byteCount, + ) as SyncBatchBuild.Ready + assertEquals(2, atCap.rowCount) + + val oneUnder = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, rows(2, size = 8))), + byteCap = one.byteCount - 1, + ) as SyncBatchBuild.Ready + assertEquals(1, oneUnder.rowCount) + assertEquals(oneUnder.body.toByteArray(Charsets.UTF_8).size, oneUnder.byteCount) + } + + /** Multi-byte characters count as their UTF-8 bytes, not as characters. */ + @Test + fun `measures utf-8 bytes rather than characters`() { + val row = SyncPendingRow(1, "\"ąęółśż\"") + val built = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, listOf(row))), + ) as SyncBatchBuild.Ready + assertEquals(built.body.toByteArray(Charsets.UTF_8).size, built.byteCount) + } + + @Test + fun `a row no empty batch could carry is a permanent error, not a silent skip`() { + val huge = SyncPendingRow(9, "\"" + "x".repeat(500) + "\"") + val built = SyncBatchBuilder.build( + listOf(SyncPendingTable(SyncTable.BOARDS, listOf(huge))), + byteCap = 100, + ) + assertEquals(SyncBatchBuild.RowTooLarge(SyncTable.BOARDS, 9, huge.byteCount), built) + } + + /** + * A Board left behind by the byte cap must not be followed by its Alert Rules in the same batch — + * the server writes them in this order and refuses the whole batch on the foreign key. + */ + @Test + fun `a table truncated by the byte cap ends the batch instead of sending children`() { + val full = SyncBatchBuilder.build( + listOf( + SyncPendingTable(SyncTable.BOARDS, rows(2, size = 40)), + SyncPendingTable(SyncTable.ALERTS, rows(1, size = 4)), + ), + byteCap = Int.MAX_VALUE, + ) as SyncBatchBuild.Ready + assertEquals(3, full.rowCount) + + val truncated = SyncBatchBuilder.build( + listOf( + SyncPendingTable(SyncTable.BOARDS, rows(2, size = 40)), + SyncPendingTable(SyncTable.ALERTS, rows(1, size = 4)), + ), + byteCap = full.byteCount - 20, + ) as SyncBatchBuild.Ready + + assertEquals(listOf(SyncTable.BOARDS), truncated.counts.keys.toList()) + assertEquals(1, truncated.counts.getValue(SyncTable.BOARDS)) + assertEquals(truncated.body.toByteArray(Charsets.UTF_8).size, truncated.byteCount) + } + + @Test + fun `nothing pending is idle, not an empty batch`() { + assertEquals(SyncBatchBuild.Empty, SyncBatchBuilder.build(emptyList())) + assertEquals( + SyncBatchBuild.Empty, + SyncBatchBuilder.build(listOf(SyncPendingTable(SyncTable.BOARDS, emptyList()))), + ) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncCursorContractTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncCursorContractTest.kt new file mode 100644 index 000000000..06ee5b4bd --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncCursorContractTest.kt @@ -0,0 +1,204 @@ +package expo.modules.vescapecore.sync + +import expo.modules.vescapecore.telemetry.SYNC_ACTIONS_UPLOADED_CURSOR +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import java.io.File + +/** + * The two contracts the uploader cannot express in code alone: the cursor key each table commits + * under, and the promise that retention deletes nothing the uploader has not delivered. + * + * Room keeps its SQL out of reach of a JVM test, so the retention half is asserted against the DAO + * source — the same technique the Sync Action classification test uses. + * + * @parity /modules/vescape-core/ios/sync/SyncCursorContractTests.swift + */ +class SyncCursorContractTest { + private fun daoSource(): String = + File("src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt").readText() + + private fun storeSource(): String = File("src/main/java/expo/modules/vescapecore/sync/SyncStore.kt").readText() + + /** The `@Query` text attached to a DAO method, whitespace-normalised across its string concatenation. */ + private fun query(method: String): String = + daoSource() + .substringBefore("suspend fun $method(") + .substringAfterLast("@Query(") + .replace("\" +", "") + .replace("\"", "") + .replace(Regex("\\s+"), " ") + .trim() + + /** The table each scan method serves, read from the store rather than restated here. */ + private fun scanMethods(): Map = + Regex("""SyncTable\.(\w+) ->\s*database\(\)\.(\w+)\(cursor, limit\)""") + .findAll(storeSource()) + .associate { SyncTable.valueOf(it.groupValues[1]) to it.groupValues[2] } + + /** + * The forward scan, asserted against the SQL that actually runs. Room keeps its queries out of + * reach of a JVM test, so this is the only place the comparison, the ordering and the limit are + * checked at all — and each of the three is a silent data-loss bug on its own: + * + * - `>=` instead of `>` re-sends the row at the cursor on every pass, forever + * - an unordered or descending scan hands out a row above one it skipped, and the commit then + * moves the cursor past the skipped row, which no later scan can reach + * - a missing `LIMIT` ignores the batch budget the byte cap is built on + */ + @Test + fun `every scan reads strictly forward from its cursor, in order, under a limit`() { + val methods = scanMethods() + assertEquals( + "every table needs a scan in SyncStore", + SyncTable.entries.toSet(), + methods.keys, + ) + for ((table, method) in methods) { + val sql = query(method) + val column = table.cursorColumn + // The bound parameter's name is the DAO's business; the comparison is the contract. + assertTrue( + "$method must read strictly past the cursor, not from it: $sql", + sql.contains(Regex("WHERE $column > :\\w+")), + ) + assertTrue("$method must order by $column ascending: $sql", sql.contains("ORDER BY $column ASC")) + assertTrue("$method must respect the batch budget: $sql", sql.contains("LIMIT :limit")) + assertTrue("$method must read $table's own table: $sql", sql.contains("FROM ${table.table} ")) + } + } + + /** + * The position reported for a row has to be the column the scan ordered on. Reporting a row id + * from a `sync_seq` scan commits a cursor in the wrong number space — every row below it in the + * other space becomes unreachable, with no error anywhere. + */ + @Test + fun `each row reports the cursor column its own scan ran on`() { + val field = mapOf(SYNC_SEQ_COLUMN to "it.syncSeq", ROW_ID_COLUMN to "it.id") + for ((table, method) in scanMethods()) { + val mapping = storeSource().substringAfter("database().$method(cursor, limit)").substringBefore("\n") + assertTrue( + "$method must report ${field.getValue(table.cursorColumn)} — it scans on ${table.cursorColumn}: $mapping", + mapping.contains("SyncPendingRow(${field.getValue(table.cursorColumn)},"), + ) + } + } + + /** A count that disagrees with the scan reports a drained backlog while rows are still waiting. */ + @Test + fun `every pending count matches its scan's own predicate`() { + val counts = Regex("""SyncTable\.(\w+) -> database\(\)\.(\w+)\(cursor\)""") + .findAll(storeSource()) + .associate { SyncTable.valueOf(it.groupValues[1]) to it.groupValues[2] } + assertEquals(SyncTable.entries.toSet(), counts.keys) + for ((table, method) in counts) { + val sql = query(method) + val scan = query(scanMethods().getValue(table)) + assertTrue( + "$method must count strictly past the cursor: $sql", + sql.contains(Regex("WHERE ${table.cursorColumn} > :\\w+")), + ) + // The unowned-telemetry exclusion is the one predicate that has to appear in both. + for (extra in listOf("board_id IS NOT NULL", "board_id != ''")) { + assertEquals( + "$method and its scan must agree about `$extra`", + scan.contains(extra), + sql.contains(extra), + ) + } + } + } + + /** + * A Metric Exclusion Range written by a sanitizer with no Board connected carries the + * unknown-Board sentinel, and the server's composite foreign key refuses it — a 409 that fails + * the whole Sync Batch. The row is retained, so without this filter the same batch retries + * forever and backup wedges permanently. An unattributed range names no Board, so there is + * nothing for the server to hang it off; it is an unowned local row, exactly like an unowned + * frame or bucket. + */ + @Test + fun `unowned rows are skipped by every scan the server keys on a Board`() { + for (table in listOf( + SyncTable.METRIC_EXCLUSION_RANGES, + SyncTable.TELEMETRY_MINUTE_BUCKETS, + )) { + val method = scanMethods().getValue(table) + assertTrue( + "$method must skip the unknown-Board sentinel: ${query(method)}", + query(method).contains("board_id != ''"), + ) + } + } + + /** The retained tables, and the column whose cursor decides what may be pruned. */ + private val gatedSweeps = mapOf( + "deleteFramesBeforeUpTo" to "id <= :cursor", + "deleteMarkersBeforeUpTo" to "id <= :cursor", + "deleteBucketsBeforeUpTo" to "sync_seq <= :cursor", + "deleteDiagnosticEventsBeforeUpTo" to "id <= :cursor", + "deleteExclusionsBeforeUpTo" to "id <= :cursor", + ) + + @Test + fun `every retained table prunes only up to its accepted cursor`() { + val source = daoSource() + for ((name, predicate) in gatedSweeps) { + val declaration = source.substringBefore("suspend fun $name") + val query = declaration.substringAfterLast("@Query(") + assertTrue("$name must gate on $predicate", query.contains(predicate)) + assertTrue("$name must still apply the age cutoff", query.contains("< :beforeMs")) + } + } + + /** + * A mutable bucket is protected by `sync_seq`, not by its row id: a bucket rewritten after an + * earlier version uploaded gets a fresh position and has to survive until that one is accepted. + */ + @Test + fun `minute buckets are gated on the counter their scan runs on`() { + assertEquals(SYNC_SEQ_COLUMN, SyncTable.TELEMETRY_MINUTE_BUCKETS.cursorColumn) + assertEquals(ROW_ID_COLUMN, SyncTable.TELEMETRY_FRAMES.cursorColumn) + } + + @Test + fun `cursor keys are namespaced away from the write counters`() { + val keys = SyncTable.entries.map { it.cursorKey } + assertEquals(keys.size, keys.toSet().size) + for (table in SyncTable.entries - SyncTable.DELETE_ACTIONS) { + assertEquals("$SYNC_CURSOR_PREFIX${table.table}", table.cursorKey) + } + // Sync Actions keep the key #282 shipped, so the log's prune reads what the uploader commits. + assertEquals(SYNC_ACTIONS_UPLOADED_CURSOR, SyncTable.DELETE_ACTIONS.cursorKey) + } + + /** Parents before children, and Delete Actions last: the order the server applies a batch in. */ + @Test + fun `table order matches the server's declared batch order`() { + assertEquals( + listOf( + "appSettings", + "boards", + "boardSettings", + "boardWarnings", + "alerts", + "tuneProfiles", + "tuneHistoryEntries", + "privacyZones", + "telemetryMarkers", + "metricExclusionRanges", + "diagnosticEvents", + "telemetryFrames", + "telemetryMinuteBuckets", + "favorites", + "vescFaultOccurrences", + "vescFaultCaptures", + "vescFaultCaptureSamples", + "deleteActions", + ), + SyncTable.entries.map { it.wire }, + ) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncDrainTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncDrainTest.kt new file mode 100644 index 000000000..b6caefc0d --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncDrainTest.kt @@ -0,0 +1,215 @@ +package expo.modules.vescapecore.sync + +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The invariant the whole uploader exists to hold: **a row the Rider owns is never left behind.** + * + * Every other sync test checks one decision in isolation. These run a backlog all the way to zero + * against a server that stores what it is sent and a source that scans forward from what was + * committed, then compare the two sets. That closes the loop the unit tests leave open — a cursor + * advanced past a row that never went in a batch is indistinguishable from a correct pass when you + * only look at the engine's return value. + * + * The direction of failure is asserted too: after a lost response or a lost checkpoint, rows may be + * re-sent (the server upserts them) but must never be skipped. + * + * @parity /modules/vescape-core/ios/sync/SyncDrainTests.swift + */ +class SyncDrainTest { + + private var now = 1_000L + + private fun engine(source: SyncSource, server: SyncTransport) = SyncEngine( + source = source, + transport = server, + environment = { + SyncEnvironment( + ridingSamples = false, + enabled = true, + online = true, + wifiOnly = false, + onWifi = false, + credentialReady = true, + onlineBlocked = false, + ) + }, + clock = { now }, + ) + + /** + * Runs passes until the backlog is drained, stepping the clock over any backoff so a transient + * failure costs a wait rather than ending the test. Bounded: a loop that stops making progress + * fails here rather than hanging. + */ + private suspend fun drain(engine: SyncEngine, source: FakeSyncSource, maxPasses: Int = 200): Int { + var passes = 0 + while (source.remaining > 0 && passes < maxPasses) { + when (val pass = engine.runOnce()) { + is SyncPass.Waiting -> now = maxOf(now, pass.untilMs) + 1 + is SyncPass.Paused -> return passes + else -> Unit + } + passes += 1 + } + return passes + } + + private fun backlog(vararg tables: Pair): FakeSyncSource = + FakeSyncSource(tables.associate { (table, count) -> table to (1L..count.toLong()).toList() }) + + @Test + fun `a full drain delivers every row exactly once`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 5, SyncTable.FAVORITES to 3) + val server = FakeSyncServer() + + drain(engine(source, server), source) + + assertEquals(source.allRows(), server.stored) + assertEquals(0, source.remaining) + // Nothing failed, so nothing had to be re-sent. + assertEquals(source.allRows().size, server.writes) + } + + /** The scan is the only thing that decides what goes next, so a small budget must not open a gap. */ + @Test + fun `a backlog larger than one scan still loses nothing`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 17, SyncTable.ALERTS to 11, SyncTable.FAVORITES to 4) + source.scanLimit = 3 + val server = FakeSyncServer() + + drain(engine(source, server), source) + + assertEquals(source.allRows(), server.stored) + assertEquals(source.allRows().size, server.writes) + } + + /** + * The response was lost, not the write. The engine cannot tell those apart, so it re-sends — and + * the rows must arrive, once, because the server upserts on identity. + */ + @Test + fun `a batch the server stored but never acknowledged is re-sent, not skipped`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 6) + source.scanLimit = 2 + val server = FakeSyncServer() + server.loseNextResponse = true + val engine = engine(source, server) + + engine.runOnce() + assertTrue("the rows are on the server", (SyncTable.BOARDS to 1L) in server.stored) + assertTrue("but nothing may be checkpointed", source.committed.isEmpty()) + + drain(engine, source) + + assertEquals(source.allRows(), server.stored) + assertEquals(0, source.remaining) + } + + /** The server took the rows; the checkpoint did not land. Re-sending is the only safe direction. */ + @Test + fun `a lost cursor commit re-sends the same rows and still drains`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 6) + source.scanLimit = 2 + val server = FakeSyncServer() + source.commitFailure = IllegalStateException("disk full") + + val engine = engine(source, server) + engine.runOnce() + assertTrue("nothing may be checkpointed", source.committed.isEmpty()) + assertEquals(6, source.remaining) + + source.commitFailure = null + drain(engine, source) + + assertEquals(source.allRows(), server.stored) + // The first batch went twice: failing toward a re-send is the whole design. + assertTrue("the lost batch must have been re-sent", server.writes > source.allRows().size) + } + + @Test + fun `a transient failure part-way through a drain loses nothing`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 9, SyncTable.PRIVACY_ZONES to 5) + source.scanLimit = 2 + val server = FakeSyncServer() + val engine = engine(source, server) + + engine.runOnce() + server.failures += SyncResponse.Transient("5xx") + server.failures += SyncResponse.Transient("5xx") + server.failures += SyncResponse.RateLimited(30_000) + + drain(engine, source) + + assertEquals(source.allRows(), server.stored) + assertEquals(0, source.remaining) + } + + /** + * A committed cursor is a promise that everything below it reached the server. Checked after every + * pass rather than at the end, because a mid-drain violation self-heals by the time the backlog is + * empty and would otherwise go unseen. + */ + @Test + fun `no cursor ever moves past a row the server does not hold`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 8, SyncTable.ALERTS to 6, SyncTable.FAVORITES to 5) + source.scanLimit = 3 + val server = FakeSyncServer() + server.failures += SyncResponse.Transient("5xx") + val engine = engine(source, server) + + var passes = 0 + while (source.remaining > 0 && passes < 100) { + when (val pass = engine.runOnce()) { + is SyncPass.Waiting -> now = maxOf(now, pass.untilMs) + 1 + is SyncPass.Paused -> break + else -> Unit + } + for ((table, cursor) in source.cursors) { + for (position in 1..cursor) { + assertTrue( + "$table cursor reached $cursor but the server never received $position", + (table to position) in server.stored, + ) + } + } + passes += 1 + } + + assertEquals(source.allRows(), server.stored) + } + + /** + * The Account changed while the request was in flight. The response belongs to the previous + * database, so nothing may be checkpointed — and every row stays pending for whoever owns it now. + */ + @Test + fun `a response that outlived its Account checkpoints nothing and strands no row`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 4) + val server = FakeSyncServer() + server.afterStore = { source.generation += 1 } + + assertEquals(SyncPass.Idle, engine(source, server).runOnce()) + + assertTrue(source.committed.isEmpty()) + assertTrue(source.cursors.isEmpty()) + assertEquals(4, source.remaining) + } + + /** A permanent pause must strand the batch in place: retained, not consumed. */ + @Test + fun `a refused batch leaves the whole backlog pending`() = runBlocking { + val source = backlog(SyncTable.BOARDS to 4, SyncTable.FAVORITES to 2) + val server = FakeSyncServer() + server.failures += SyncResponse.Invalid(409, "dependency-conflict") + + assertEquals(SyncPass.Paused(SyncPauseReason.PROTOCOL), engine(source, server).runOnce()) + + assertTrue(source.committed.isEmpty()) + assertEquals(6, source.remaining) + assertTrue(server.stored.isEmpty()) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncEngineTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncEngineTest.kt new file mode 100644 index 000000000..0665e4126 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncEngineTest.kt @@ -0,0 +1,235 @@ +package expo.modules.vescapecore.sync + +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The engine against a fake transport: the cases that decide whether a Rider's data survives — a + * wedged batch, a failure part-way through a drain, a dead token, and a response that outlived the + * Account it was sent for. + * + * @parity /modules/vescape-core/ios/sync/SyncEngineTests.swift + */ +class SyncEngineTest { + /** Two rows per scan, so a backlog of four takes two passes — the shape these cases were written against. */ + private fun FakeSource(rows: Int = 3) = FakeSyncSource(rows).also { it.scanLimit = 2 } + + private fun accepted(boards: Int): String { + val counts = SyncTable.entries.joinToString(",") { + "\"${it.wire}\":${if (it == SyncTable.BOARDS) boards else 0}" + } + return "{\"accepted\":{$counts}}" + } + + private fun engine( + source: SyncSource, + responses: MutableList, + sent: MutableList = mutableListOf(), + ) = SyncEngine( + source = source, + transport = { body -> + sent += body + responses.removeAt(0) + }, + environment = { + SyncEnvironment( + ridingSamples = false, + enabled = true, + online = true, + wifiOnly = false, + onWifi = false, + credentialReady = true, + onlineBlocked = false, + ) + }, + clock = { 1_000 }, + ) + + @Test + fun `a valid 200 advances only the rows it accounted for`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = engine(source, mutableListOf(SyncResponse.Accepted(accepted(2)))) + + assertEquals(SyncPass.Sent(2, morePending = false), engine.runOnce()) + assertEquals(listOf(mapOf(SyncTable.BOARDS to 2L)), source.committed) + } + + @Test + fun `a mismatched accepted count is a protocol failure and moves no cursor`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = engine(source, mutableListOf(SyncResponse.Accepted(accepted(1)))) + + assertEquals(SyncPass.Paused(SyncPauseReason.PROTOCOL), engine.runOnce()) + assertTrue(source.committed.isEmpty()) + assertEquals(listOf(SyncPauseReason.PROTOCOL to "acceptedMismatch"), source.failures) + } + + @Test + fun `a malformed success body never advances a cursor`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = engine(source, mutableListOf(SyncResponse.Accepted("not json"))) + + assertEquals(SyncPass.Paused(SyncPauseReason.PROTOCOL), engine.runOnce()) + assertTrue(source.committed.isEmpty()) + } + + @Test + fun `a refused batch leaves every cursor untouched and does not retry on a kick`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = engine( + source, + mutableListOf(SyncResponse.Invalid(409, "dependency-conflict")), + ) + + assertEquals(SyncPass.Paused(SyncPauseReason.PROTOCOL), engine.runOnce()) + assertTrue(source.committed.isEmpty()) + // No second response is queued, so a pass that sent again would fail the test outright. + assertEquals(SyncPass.Paused(SyncPauseReason.PROTOCOL), engine.runOnce()) + } + + @Test + fun `a failure part-way through a drain leaves cursors at the last accepted batch`() = runBlocking { + val source = FakeSource(rows = 4) + val engine = engine( + source, + mutableListOf( + SyncResponse.Accepted(accepted(2)), + SyncResponse.Transient("5xx"), + ), + ) + + assertEquals(SyncPass.Sent(2, morePending = true), engine.runOnce()) + val second = engine.runOnce() + assertTrue(second is SyncPass.Waiting) + assertEquals(listOf(mapOf(SyncTable.BOARDS to 2L)), source.committed) + } + + @Test + fun `a dead token stops the loop for sign-in`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = engine(source, mutableListOf(SyncResponse.Unauthorized)) + + assertEquals(SyncPass.Paused(SyncPauseReason.AUTHENTICATION), engine.runOnce()) + assertEquals(SyncPauseReason.AUTHENTICATION, engine.pauseReason) + assertTrue(source.committed.isEmpty()) + } + + @Test + fun `a response from the previous Account cannot advance a cursor`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = SyncEngine( + source = source, + transport = { + // The Account changed while this request was in flight. + source.generation += 1 + SyncResponse.Accepted(accepted(2)) + }, + environment = { + SyncEnvironment( + ridingSamples = false, + enabled = true, + online = true, + wifiOnly = false, + onWifi = false, + credentialReady = true, + onlineBlocked = false, + ) + }, + clock = { 1_000 }, + ) + + assertEquals(SyncPass.Idle, engine.runOnce()) + assertTrue(source.committed.isEmpty()) + } + + @Test + fun `a timeout after the server committed resends the identical batch`() = runBlocking { + val source = FakeSource(rows = 2) + val sent = mutableListOf() + val engine = engine( + source, + mutableListOf(SyncResponse.Transient("timeout"), SyncResponse.Accepted(accepted(2))), + sent, + ) + + engine.runOnce() + engine.resume() + engine.runOnce() + assertEquals(2, sent.size) + assertEquals(sent[0], sent[1]) + } + + @Test + fun `413 narrows the byte target and a single row that still fails pauses without being skipped`() = + runBlocking { + val source = FakeSource(rows = 1) + val engine = engine(source, mutableListOf(SyncResponse.TooLarge)) + + assertEquals(SyncPass.Paused(SyncPauseReason.ROW_TOO_LARGE), engine.runOnce()) + assertTrue(source.committed.isEmpty()) + assertEquals(1, source.remaining) + } + + /** A shrink accepted nothing, so it must not be reported as an upload. */ + @Test + fun `413 on a multi-row batch narrows the target and retries`() = runBlocking { + val source = FakeSource(rows = 4) + val engine = engine(source, mutableListOf(SyncResponse.TooLarge)) + + assertEquals(SyncPass.Retry, engine.runOnce()) + assertTrue(source.committed.isEmpty()) + assertEquals(4, source.remaining) + } + + /** Halving forever against a server that keeps refusing would be an unbounded request storm. */ + @Test + fun `413 at the smallest batch pauses instead of resending the same bytes forever`() = runBlocking { + val source = FakeSource(rows = 4) + val engine = engine(source, MutableList(10) { SyncResponse.TooLarge }) + + var passes = 0 + var outcome = engine.runOnce() + while (outcome == SyncPass.Retry && passes < 10) { + outcome = engine.runOnce() + passes += 1 + } + assertEquals(SyncPass.Paused(SyncPauseReason.ROW_TOO_LARGE), outcome) + assertTrue(source.committed.isEmpty()) + } + + /** The server took the rows; the checkpoint did not land. Resending is safe, claiming success is not. */ + @Test + fun `a failed cursor commit backs off instead of reporting an upload`() = runBlocking { + val source = FakeSource(rows = 2) + source.commitFailure = IllegalStateException("disk full") + val engine = engine(source, mutableListOf(SyncResponse.Accepted(accepted(2)))) + + val outcome = engine.runOnce() + assertTrue(outcome is SyncPass.Waiting) + assertTrue(source.committed.isEmpty()) + assertEquals(2, source.remaining) + } + + @Test + fun `429 waits for the server's own delay`() = runBlocking { + val source = FakeSource(rows = 2) + val engine = engine(source, mutableListOf(SyncResponse.RateLimited(90_000))) + + assertEquals(SyncPass.Waiting(91_000), engine.runOnce()) + assertNull(engine.pauseReason) + } + + @Test + fun `a row that cannot be encoded pauses with the row retained`() = runBlocking { + val source = FakeSource(rows = 2) + source.encodeFailure = SyncProtocolException(SyncTable.BOARDS, "id", "must not be empty") + val engine = engine(source, mutableListOf()) + + assertEquals(SyncPass.Paused(SyncPauseReason.PROTOCOL), engine.runOnce()) + assertEquals(listOf(SyncPauseReason.PROTOCOL to "boards.id"), source.failures) + assertEquals(2, source.remaining) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncPolicyTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncPolicyTest.kt new file mode 100644 index 000000000..72e2526e9 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncPolicyTest.kt @@ -0,0 +1,149 @@ +package expo.modules.vescapecore.sync + +import org.junit.Assert.assertEquals +import org.junit.Test + +/** + * The send/wait/paused decision, with no database, clock or network behind it. + * + * @parity /modules/vescape-core/ios/sync/SyncPolicyTests.swift + */ +class SyncPolicyTest { + private fun state( + pendingRows: Int = 1, + ridingSamples: Boolean = false, + enabled: Boolean = true, + online: Boolean = true, + wifiOnly: Boolean = false, + onWifi: Boolean = false, + credentialReady: Boolean = true, + onlineBlocked: Boolean = false, + pause: SyncPauseReason? = null, + retryAtMs: Long = 0, + ) = SyncState( + nowMs = 1_000, + pendingRows = pendingRows, + ridingSamples = ridingSamples, + enabled = enabled, + online = online, + wifiOnly = wifiOnly, + onWifi = onWifi, + credentialReady = credentialReady, + onlineBlocked = onlineBlocked, + pause = pause, + retryAtMs = retryAtMs, + ) + + @Test + fun `pending rows on a live connection send now`() { + assertEquals(SyncDecision.SendNow, SyncPolicy.decide(state())) + } + + @Test + fun `cadence follows sample production, not session presence`() { + assertEquals( + SyncDecision.Wait(1_000 + SyncPolicy.RIDE_INTERVAL_MS), + SyncPolicy.decide(state(pendingRows = 0, ridingSamples = true)), + ) + assertEquals( + SyncDecision.Wait(1_000 + SyncPolicy.IDLE_INTERVAL_MS), + SyncPolicy.decide(state(pendingRows = 0)), + ) + } + + /** Offline, metered and gated are pauses in the loop, never failures that move backoff. */ + @Test + fun `offline, wifi-only on cellular and a closed gate all wait`() { + val idle = SyncDecision.Wait(1_000 + SyncPolicy.IDLE_INTERVAL_MS) + assertEquals(idle, SyncPolicy.decide(state(online = false))) + assertEquals(idle, SyncPolicy.decide(state(wifiOnly = true, onWifi = false))) + assertEquals(idle, SyncPolicy.decide(state(onlineBlocked = true))) + assertEquals(SyncDecision.SendNow, SyncPolicy.decide(state(wifiOnly = true, onWifi = true))) + } + + @Test + fun `backoff deadline holds the loop until it passes`() { + assertEquals(SyncDecision.Wait(5_000), SyncPolicy.decide(state(retryAtMs = 5_000))) + assertEquals(SyncDecision.SendNow, SyncPolicy.decide(state(retryAtMs = 999))) + } + + @Test + fun `a pause is not bypassed by an ordinary kick`() { + assertEquals( + SyncDecision.Paused(SyncPauseReason.PROTOCOL), + SyncPolicy.decide(state(pause = SyncPauseReason.PROTOCOL)), + ) + assertEquals( + SyncDecision.Paused(SyncPauseReason.AUTHENTICATION), + SyncPolicy.decide(state(credentialReady = false)), + ) + } + + @Test + fun `the master switch stops the uploader outright and outranks every other state`() { + assertEquals( + SyncDecision.Wait(1_000 + SyncPolicy.IDLE_INTERVAL_MS), + SyncPolicy.decide(state(enabled = false)), + ) + // Not a pause: switched off is not a broken uploader waiting to be resumed. + assertEquals( + SyncDecision.Wait(1_000 + SyncPolicy.IDLE_INTERVAL_MS), + SyncPolicy.decide(state(enabled = false, pause = SyncPauseReason.PROTOCOL)), + ) + assertEquals(SyncActivity.DISABLED, SyncPolicy.describe(state(enabled = false))) + assertEquals( + SyncActivity.DISABLED, + SyncPolicy.describe(state(enabled = false, credentialReady = false)), + ) + assertEquals( + SyncActivity.DISABLED, + SyncPolicy.describe(state(enabled = false, pause = SyncPauseReason.AUTHENTICATION)), + ) + } + + @Test + fun `a phone with no credential reads as signed out, not as a broken backup`() { + assertEquals(SyncActivity.SIGNED_OUT, SyncPolicy.describe(state(credentialReady = false))) + assertEquals( + SyncActivity.SIGNED_OUT, + SyncPolicy.describe(state(credentialReady = false, pause = SyncPauseReason.AUTHENTICATION)), + ) + } + + @Test + fun `every waiting reason is named separately`() { + assertEquals(SyncActivity.UP_TO_DATE, SyncPolicy.describe(state(pendingRows = 0))) + assertEquals(SyncActivity.SYNCING, SyncPolicy.describe(state())) + assertEquals(SyncActivity.OFFLINE, SyncPolicy.describe(state(online = false))) + assertEquals(SyncActivity.OFFLINE, SyncPolicy.describe(state(onlineBlocked = true))) + assertEquals( + SyncActivity.WAITING_FOR_WIFI, + SyncPolicy.describe(state(wifiOnly = true, onWifi = false)), + ) + assertEquals(SyncActivity.SYNCING, SyncPolicy.describe(state(wifiOnly = true, onWifi = true))) + } + + @Test + fun `a pause outranks everything except being signed out`() { + assertEquals( + SyncActivity.PAUSED, + SyncPolicy.describe(state(pendingRows = 0, pause = SyncPauseReason.PROTOCOL)), + ) + assertEquals( + SyncActivity.PAUSED, + SyncPolicy.describe(state(online = false, pause = SyncPauseReason.ROW_TOO_LARGE)), + ) + } + + @Test + fun `a batch waiting on backoff still reads as syncing`() { + assertEquals(SyncActivity.SYNCING, SyncPolicy.describe(state(retryAtMs = 60_000))) + } + + @Test + fun `backoff doubles from the first step and stops at the cap`() { + assertEquals(SyncPolicy.BACKOFF_START_MS, SyncPolicy.nextBackoffMs(0)) + assertEquals(60_000, SyncPolicy.nextBackoffMs(30_000)) + assertEquals(SyncPolicy.BACKOFF_MAX_MS, SyncPolicy.nextBackoffMs(SyncPolicy.BACKOFF_MAX_MS)) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncWireTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncWireTest.kt new file mode 100644 index 000000000..dd18345da --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncWireTest.kt @@ -0,0 +1,358 @@ +package expo.modules.vescapecore.sync + +import expo.modules.vescapecore.telemetry.AlertRuleEntity +import expo.modules.vescapecore.telemetry.AppSettingEntity +import expo.modules.vescapecore.telemetry.BoardEntity +import expo.modules.vescapecore.telemetry.BoardWarningEntity +import expo.modules.vescapecore.telemetry.SyncActionEntity +import expo.modules.vescapecore.telemetry.TelemetryFrameEntity +import expo.modules.vescapecore.telemetry.TelemetryMinuteBucketEntity +import expo.modules.vescapecore.telemetry.VescFaultCaptureEntity +import expo.modules.vescapecore.telemetry.VescFaultCaptureSampleEntity +import expo.modules.vescapecore.telemetry.VescFaultOccurrenceEntity +import org.junit.Assert.assertEquals +import org.junit.Assert.assertThrows +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * Wire encoding and the bounds it refuses on. The valid/invalid boundary cases mirror the server's + * own schema (`vescape-server` `src/sync/protocol.ts`), so a row this side accepts is a row that + * side can store — a batch is whole or refused, and a bad row must never reach transport. + * + * @parity /modules/vescape-core/ios/sync/SyncWireTests.swift + */ +class SyncWireTest { + private fun board(id: String = "board-1", name: String = "Board") = BoardEntity( + id = id, + name = name, + bleId = null, + createdAt = 10, + updatedAt = 20, + ) + + private fun frame(boardId: String? = "board-1", speed: Int? = 100) = TelemetryFrameEntity( + id = 5, + capturedAtMs = 1_000, + elapsedRealtimeMs = 500, + boardId = boardId, + canId = null, + flags = 1, + changedMask1 = 3, + changedMask2 = 0, + speedCentiKmh = speed, + batteryVoltageMv = null, + motorCurrentMa = null, + batteryCurrentMa = null, + dutyPermille = null, + pitchCentiDeg = null, + rollCentiDeg = null, + balancePitchCentiDeg = null, + balanceCurrentMa = null, + erpm = null, + state = null, + switchState = null, + adc1Milli = null, + adc2Milli = null, + odometerCm = null, + tempMosfetDeciC = null, + tempMotorDeciC = null, + latitudeE7 = null, + longitudeE7 = null, + gpsSpeedCentiMps = null, + bearingCentiDeg = null, + accuracyCm = null, + altitudeCm = null, + locationTimestampMs = null, + ) + + @Test + fun `a board encodes exactly the declared fields, nulls included`() { + assertEquals( + """{"id":"board-1","name":"Board","bleId":null,"transport":null,"createdAt":10,"updatedAt":20}""", + SyncWire.board(board()), + ) + } + + /** "Cleared" and "not mentioned" are different intents, and only one survives a missing key. */ + @Test + fun `nullable columns are explicit nulls, never omitted keys`() { + assertTrue(SyncWire.telemetryFrame(frame(speed = null)).contains("\"speedCentiKmh\":null")) + } + + @Test + fun `text is escaped so the body stays parseable`() { + val encoded = SyncWire.board(board(name = "He said \"go\"\n")) + assertTrue(encoded.contains("""\"go\"""")) + assertTrue(encoded.contains("""\n""")) + } + + @Test + fun `a key at the length limit is valid and one over is refused`() { + val atLimit = "b".repeat(MAX_SYNC_KEY_LENGTH) + SyncWire.board(board(id = atLimit)) + assertThrows(SyncProtocolException::class.java) { + SyncWire.board(board(id = "b".repeat(MAX_SYNC_KEY_LENGTH + 1))) + } + } + + @Test + fun `an empty key is refused where the server names it, and allowed where the phone derives it`() { + assertThrows(SyncProtocolException::class.java) { SyncWire.board(board(id = "")) } + SyncWire.appSetting(AppSettingEntity(key = "mapStyleKey", valueJson = "\"\"", updatedAt = 1)) + } + + /** A sample that names no Board has nowhere to go on the server, so it never reaches transport. */ + @Test + fun `a frame without a board is a protocol error`() { + val error = assertThrows(SyncProtocolException::class.java) { + SyncWire.telemetryFrame(frame(boardId = null)) + } + assertEquals("boardId", error.field) + } + + @Test + fun `integer bounds are enforced at the edge, not left to the server`() { + SyncWire.telemetryFrame(frame(speed = Int.MAX_VALUE)) + val error = assertThrows(SyncProtocolException::class.java) { + SyncWire.telemetryMinuteBucket(bucket(sampleCount = -1)) + } + assertEquals("sampleCount", error.field) + } + + @Test + fun `a non-finite number is refused because JSON cannot express it`() { + val error = assertThrows(SyncProtocolException::class.java) { + SyncRowWriter(SyncTable.ALERTS).number("threshold", Double.NaN) + } + assertEquals("threshold", error.field) + } + + /** An action reads like the row it names: flat identity fields, not a nested envelope. */ + @Test + fun `a delete action expands into the identity its target declares`() { + assertEquals( + """{"target":"boardSetting","boardId":"board-1","key":"transport","deletedAt":9}""", + SyncWire.deleteAction( + SyncActionEntity(id = 1, target = "boardSetting", boardId = "board-1", key = "transport", deletedAt = 9), + ), + ) + assertEquals( + """{"target":"tuneProfile","id":"profile-1","deletedAt":4}""", + SyncWire.deleteAction( + SyncActionEntity(id = 2, target = "tuneProfile", boardId = null, key = "profile-1", deletedAt = 4), + ), + ) + assertThrows(SyncProtocolException::class.java) { + SyncWire.deleteAction( + SyncActionEntity(id = 3, target = "somethingElse", boardId = null, key = "x", deletedAt = 1), + ) + } + } + + /** + * The whole reason the occurrence carries its own change timestamp: dismissal is a Rider edit the + * restore has to preserve, and `lastObservedAtMs` cannot express it. + */ + @Test + fun `a fault occurrence carries its own change timestamp, not just the last observation`() { + assertEquals( + """{"id":"fault-1","boardId":"board-1","code":6,"occurredAtMs":1000,""" + + """"lastObservedAtMs":2000,"clearedAtMs":null,"dismissed":true,"updatedAt":9000}""", + SyncWire.vescFaultOccurrence( + VescFaultOccurrenceEntity( + id = "fault-1", + boardId = "board-1", + code = 6, + occurredAtMs = 1_000, + lastObservedAtMs = 2_000, + clearedAtMs = null, + dismissed = true, + updatedAt = 9_000, + syncSeq = 4, + ), + ), + ) + } + + /** A Capture is immutable, so it carries no change timestamp — and no cursor either. */ + @Test + fun `a fault capture encodes exactly the declared fields`() { + assertEquals( + """{"occurrenceId":"fault-1","boardId":"board-1","startedAtMs":500,"openedAtMs":1000,""" + + """"sampleCount":42}""", + SyncWire.vescFaultCapture( + VescFaultCaptureEntity( + occurrenceId = "fault-1", + boardId = "board-1", + startedAtMs = 500, + openedAtMs = 1_000, + sampleCount = 42, + syncSeq = 7, + ), + ), + ) + } + + /** + * The local autoincrement id restarts on a fresh install, so it can never be identity: a restored + * phone's re-upload has to be an idempotent no-op, keyed on the Occurrence and the capture time. + */ + @Test + fun `a capture sample sends no local row id and nulls what the firmware never reported`() { + val encoded = SyncWire.vescFaultCaptureSample( + VescFaultCaptureSampleEntity( + id = 31, + occurrenceId = "fault-1", + capturedAtMs = 1_500, + speed = 12.5, + dutyCycle = null, + erpm = null, + batteryVoltage = null, + batteryCurrent = null, + motorCurrent = null, + tempMosfet = null, + tempMotor = null, + pitch = null, + roll = null, + balancePitch = null, + adc1 = null, + adc2 = null, + state = 4, + ), + ) + + assertTrue(encoded.startsWith("""{"occurrenceId":"fault-1","capturedAtMs":1500,"speed":12.5,""")) + assertTrue(encoded.contains(""""dutyCycle":null""")) + assertTrue(encoded.endsWith(""""state":4}""")) + assertTrue("the local row id must never cross the wire", !encoded.contains("\"id\"")) + } + + /** + * A decoded Board sample is the one thing on the wire the app did not author — it received it. + * Refusing a non-finite float here would pause every table's backup on a permanent protocol + * error that no retry can clear, over a reading the firmware itself could not express. Absent is + * what these nullable columns already mean, so an unusable reading is absent too. + */ + @Test + fun `an unusable firmware reading is absent rather than a permanent protocol pause`() { + val encoded = SyncWire.vescFaultCaptureSample( + VescFaultCaptureSampleEntity( + id = 32, + occurrenceId = "fault-1", + capturedAtMs = 1_500, + speed = Double.NaN, + dutyCycle = Double.POSITIVE_INFINITY, + erpm = Double.NEGATIVE_INFINITY, + batteryVoltage = 78.9, + batteryCurrent = null, + motorCurrent = null, + tempMosfet = null, + tempMotor = null, + pitch = null, + roll = null, + balancePitch = null, + adc1 = null, + adc2 = null, + state = 4, + ), + ) + + assertTrue(encoded.contains(""""speed":null""")) + assertTrue(encoded.contains(""""dutyCycle":null""")) + assertTrue(encoded.contains(""""erpm":null""")) + assertTrue("a usable reading beside an unusable one still lands", encoded.contains(""""batteryVoltage":78.9""")) + } + + /** + * A rule restored without its kind looks configured and fires at the wrong point; a field the + * server has dropped rejects the entire batch. Both are asserted on the exact bytes. + */ + @Test + fun `an alert carries the kind its thresholds are read under`() { + assertEquals( + """{"boardId":"board-1","id":"rule-1","controlId":"duty","threshold":70,""" + + """"thresholdMax":null,"thresholdKind":"configRelative","configFieldId":"tiltback_duty",""" + + """"thresholdOffset":-5,"thresholdMaxOffset":null,"enabled":true,"soundType":"beep",""" + + """"repeatEverySeconds":30,"beepCount":2,"source":"preset","createdAt":10,"updatedAt":20}""", + SyncWire.alert( + AlertRuleEntity( + boardId = "board-1", + id = "rule-1", + controlId = "duty", + threshold = 70.0, + thresholdMax = null, + thresholdKind = "configRelative", + configFieldId = "tiltback_duty", + thresholdOffset = -5.0, + thresholdMaxOffset = null, + enabled = true, + soundType = "beep", + createdAt = 10, + repeatEverySeconds = 30, + beepCount = 2, + source = "preset", + updatedAt = 20, + ), + ), + ) + } + + /** + * Without the stamp the server has nothing to compare, and a re-detection wins or loses + * arbitrarily against whatever it already holds. + */ + @Test + fun `a board warning carries the stamp the server judges two writes on`() { + assertEquals( + """{"boardId":"board-1","kind":"cellSpread","severity":"warning","firstDetectedAt":10,""" + + """"lastDetectedAt":20,"payloadJson":"{}","updatedAt":21}""", + SyncWire.boardWarning( + BoardWarningEntity( + boardId = "board-1", + kind = "cellSpread", + severity = "warning", + firstDetectedAt = 10, + lastDetectedAt = 20, + payloadJson = "{}", + updatedAt = 21, + ), + ), + ) + } + + /** + * Both columns are gone from the server's schema, which validates a batch whole: sending either + * one rejects every row in it, not just the field. + */ + @Test + fun `the columns the server dropped are not sent at all`() { + assertTrue(!SyncWire.telemetryFrame(frame()).contains("faultCode")) + assertTrue(!SyncWire.telemetryMinuteBucket(bucket(sampleCount = 1)).contains("faultCount")) + } + + private fun bucket(sampleCount: Int) = TelemetryMinuteBucketEntity( + bucketStartMs = 60_000, + boardId = "board-1", + sampleCount = sampleCount, + firstSampleAtMs = 60_000, + lastSampleAtMs = 60_500, + sumAbsSpeedCentiKmh = 1, + movingSpeedSampleCount = null, + sumMovingAbsSpeedCentiKmh = null, + maxAbsSpeedCentiKmh = 1, + minBatteryVoltageMv = null, + maxMotorCurrentAbsMa = 0, + maxBatteryCurrentAbsMa = 0, + batteryUsedWhMilli = 0, + batteryRegenWhMilli = 0, + maxDutyAbsPermille = 0, + firstOdometerCm = null, + lastOdometerCm = null, + gpsPointCount = 0, + preciseGpsPointCount = 0, + gpsDistanceCm = 0, + maxGpsSpeedCentiMps = null, + updatedAt = 1, + ) +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/BoardTombstoneTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/BoardTombstoneTest.kt index 93921b357..469711c10 100644 --- a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/BoardTombstoneTest.kt +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/BoardTombstoneTest.kt @@ -17,7 +17,7 @@ import java.lang.reflect.Proxy * * Room's `@Query` has BINARY retention and its generated implementation keeps the SQL in a * method-local string, so a JVM unit test has no runtime handle on the statements Room will run — - * the read/delete contracts are asserted against the DAO source. + * the read/delete contracts are asserted against the DAO source, as in [SyncCursorMigrationTest]. * * @parity /modules/vescape-core/ios/telemetry/BoardTombstoneTests.swift */ @@ -99,7 +99,7 @@ class BoardTombstoneTest { assertFalse("a DELETE on boards survives", dao.contains("DELETE FROM boards")) assertTrue( "the delete path does not stamp a tombstone", - dao.contains("insertBoardRow(board.copy(deletedAt = deletedAt))"), + dao.contains("upsertBoard(board.copy(deletedAt = tombstonedAt, updatedAt = tombstonedAt))"), ) } @@ -109,7 +109,7 @@ class BoardTombstoneTest { val dao = daoSource() val body = dao.substringAfter("suspend fun deleteBoardWithSettings").substringBefore("\n }") - for (call in listOf("deleteBoardSettings(id)", "deleteBoardWarnings(id)", "deleteAlertRules(id)")) { + for (call in listOf("deleteBoardSettingsRaw(id)", "deleteBoardWarningsRaw(id)", "deleteAlertRulesRaw(id)")) { assertTrue("the delete path dropped `$call`", body.contains(call)) } } diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/FavoriteSummaryBuilderTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/FavoriteSummaryBuilderTest.kt index 806504aa7..d61ab7cf7 100644 --- a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/FavoriteSummaryBuilderTest.kt +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/FavoriteSummaryBuilderTest.kt @@ -264,6 +264,7 @@ class FavoriteSummaryBuilderTest { firstLongitudeE7 = null, firstMovingAtMs = bucketStartMs, lastMovingAtMs = bucketStartMs + 9_000, + updatedAt = bucketStartMs + 9_000, ) private fun favorite(distanceCm: Long?) = FavoriteEntity( diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/ProfileStatsRepositoryTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/ProfileStatsRepositoryTest.kt index da54e4098..79e5d4cb8 100644 --- a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/ProfileStatsRepositoryTest.kt +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/ProfileStatsRepositoryTest.kt @@ -175,6 +175,7 @@ class ProfileStatsRepositoryTest { maxGpsSpeedCentiMps = 9_999, firstMovingAtMs = firstMoving, lastMovingAtMs = lastMoving, + updatedAt = end, firstLatitudeE7 = latitudeE7, firstLongitudeE7 = longitudeE7, ) diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/RideHistoryPagingTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/RideHistoryPagingTest.kt index c968059b6..dc39c2401 100644 --- a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/RideHistoryPagingTest.kt +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/RideHistoryPagingTest.kt @@ -69,5 +69,6 @@ class RideHistoryPagingTest { lastMovingAtMs = end, firstLatitudeE7 = null, firstLongitudeE7 = null, + updatedAt = end, ) } diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncActionLogTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncActionLogTest.kt new file mode 100644 index 000000000..1cf6b52b1 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncActionLogTest.kt @@ -0,0 +1,245 @@ +package expo.modules.vescapecore.telemetry + +import android.database.Cursor +import androidx.room.migration.Migration +import androidx.sqlite.db.SupportSQLiteDatabase +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import java.io.File +import java.lang.reflect.Proxy + +/** + * The Sync Action log (#282): an append-only record of semantic removals, which no surviving row can + * express. A deleted row cannot carry a Change Timestamp saying it is gone. + * + * Room's `@Query` has BINARY retention and its generated implementation keeps the SQL in a + * method-local string, so a JVM unit test has no runtime handle on the statements Room will run — + * the classification contract is asserted against the DAO source, as in [BoardTombstoneTest]. The + * behavioural half (which action lands, with which stamp) runs against a real database on iOS. + * + * @parity /modules/vescape-core/ios/telemetry/SyncActionLogTests.swift + */ +class SyncActionLogTest { + /** + * Every DAO function that removes rows from a syncable table, and why it is allowed to. + * + * `semantic` appends a Sync Action; `parentCascade` and `maintenance` deliberately do not. A new + * delete has to be classified here before the source scan below will accept it — that is the whole + * point of the map, and it mirrors the server's own structural test. + */ + private val semantic = setOf( + "deletePrivacyZone", + "deleteBoardSetting", + "deleteAlertRule", + "deleteAppSetting", + "deleteTuneProfileSafe", + "deleteBoardWarning", + "deleteBoardWarnings", + "deleteFavorite", + // Tombstones the Board and raw-deletes its configuration under one Board action. + "deleteBoardWithSettings", + ) + + private val parentCascade = setOf( + "deleteBoardSettingsRaw", + "deleteAlertRulesRaw", + "deleteBoardWarningsRaw", + "deleteTuneHistoryForProfileRaw", + "deleteFavoriteMediaForFavoriteRaw", + // The row-level primitives the semantic wrappers above own; never called from outside the DAO. + "deletePrivacyZoneRow", + "deleteBoardSettingRow", + "deleteAlertRuleRow", + "deleteAppSettingRow", + "deleteTuneProfileRow", + "deleteBoardWarningRow", + "deleteFavoriteRow", + ) + + /** Retention, orphan sweeps and the wipe behind a database restore. Never Rider intent. */ + private val maintenance = setOf( + "deleteExclusionsRange", + "clearExclusions", + "deleteExclusionsBefore", + "deleteFramesBefore", + "deleteMarkersBefore", + "deleteBucketsBefore", + "deleteDiagnosticEventsBefore", + "deleteBefore", + // Cursor-gated retention (#284): the same sweep, refusing to prune a row the uploader has not + // delivered yet. + "deleteBeforeGated", + "deleteFramesBeforeUpTo", + "deleteMarkersBeforeUpTo", + "deleteBucketsBeforeUpTo", + "deleteDiagnosticEventsBeforeUpTo", + "deleteExclusionsBeforeUpTo", + "deleteFramesRange", + "deleteMarkersRange", + "deleteBucketsRange", + "deleteRange", + "deleteFramesRangeAllDevices", + "deleteMarkersRangeAllDevices", + "deleteBucketsRangeAllDevices", + "deleteRangeAllDevices", + "clearFrames", + "clearMarkers", + "clearBuckets", + "clearDiagnosticEvents", + "clearAll", + "deleteFavoriteMedia", + "deleteOrphanFavoriteMedia", + // Board-owned decode caches, rebuilt from the board on the next read. A removal is never + // durable state the server has to learn, which holds at every call site alike: dropped with + // the Board, on a link mismatch, and on a Rider dismissing a change notice. + "deleteBoardConfigValues", + "deleteMotorConfigValues", + "deleteBoardConfigChangeNotice", + // Transport state, pruned only behind the accepted-action cursor. + "deleteSyncActionsThrough", + "pruneUploadedSyncActions", + ) + + private fun daoSource(): String = + File("src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt").readText() + + /** + * DAO source split into `fun name` -> that declaration, its body, and the annotation block above + * it — `@Query` is where a raw delete keeps its SQL. + */ + private fun daoFunctions(): Map { + val source = daoSource() + val starts = Regex("suspend fun (\\w+)").findAll(source).toList() + return starts.mapIndexed { index, match -> + val next = starts.getOrNull(index + 1)?.range?.first ?: source.length + var chunk = source.substring(match.range.first, next) + // Trailing text belongs to the next declaration's annotations, not to this body. + chunk.lastIndexOf("\n @").takeIf { it >= 0 }?.let { chunk = chunk.substring(0, it) } + val annotations = source.substring(0, match.range.first) + .substringAfterLast("\n\n") + match.groupValues[1] to annotations + chunk + }.toMap() + } + + private fun deletingFunctions(): Map = + daoFunctions().filter { (_, body) -> body.contains("DELETE FROM") } + + @Test + fun `every delete against a table is classified`() { + val classified = semantic + parentCascade + maintenance + val unclassified = deletingFunctions().keys - classified + assertTrue( + "Unclassified deletes in TelemetryDao: $unclassified — classify each as semantic, " + + "parent cascade or maintenance", + unclassified.isEmpty(), + ) + val stale = classified - daoFunctions().keys + assertTrue("Classified names that no longer exist: $stale", stale.isEmpty()) + } + + @Test + fun `semantic removals append an action and maintenance never does`() { + val functions = daoFunctions() + for (name in semantic) { + val body = functions.getValue(name) + // Either it appends the action itself, or it delegates to a wrapper that does — a clear-all is + // one action per removed row, not one for the sweep. + val delegates = (semantic - name).any { body.contains("$it(") } + assertTrue( + "$name is classified semantic but appends no Sync Action", + body.contains("appendDeleteAction") || delegates, + ) + } + for (name in parentCascade + maintenance) { + assertTrue( + "$name is a raw delete but appends a Sync Action", + !functions.getValue(name).contains("appendDeleteAction"), + ) + } + } + + /** + * The retention boundary, made structural: a target can only name configuration or current state. + * Giving one of the pruned tables a target would make the server delete exactly the rides the + * backup exists to preserve. Mirrors the server's `DELETE_ACTION_TARGETS` test. + */ + @Test + fun `no retained table can be named by a delete target`() { + val retained = setOf( + "telemetry_frames", + "telemetry_markers", + "telemetry_minute_buckets", + "metric_exclusion_ranges", + "diagnostic_events", + "tune_history_entries", + "favorite_media", + "sync_actions", + "sync_sequences", + ) + val named = DeleteTarget.entries.map { it.table }.toSet() + assertEquals(emptySet(), named intersect retained) + assertEquals(DeleteTarget.entries.size, named.size) + } + + /** The log is append-only and keyed on its own cursor; no trigger writes it. */ + @Test + fun `migration creates the log keyed on an autoincrement cursor`() { + val sql = migrationSql(TelemetryDatabase.MIGRATION_45_46).joinToString("\n") + assertTrue(sql, sql.contains("CREATE TABLE IF NOT EXISTS sync_actions")) + assertTrue(sql, sql.contains("id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL")) + assertTrue(sql, sql.contains("deleted_at INTEGER NOT NULL")) + assertTrue("the log must not be driven by a trigger", !sql.contains("CREATE TRIGGER")) + } + + /** Additive and guarded, so re-running the migration is a no-op rather than a duplicate table. */ + @Test + fun `migration is additive and re-runnable`() { + val sql = migrationSql(TelemetryDatabase.MIGRATION_45_46) + assertTrue(sql.isNotEmpty()) + for (statement in sql) { + assertTrue("not guarded: $statement", statement.contains("IF NOT EXISTS")) + assertTrue("not additive: $statement", !statement.contains("DROP ") && !statement.contains("DELETE ")) + } + } + + /** The accepted cursor is checkpointed first; pruning reads it back rather than trusting a caller. */ + @Test + fun `pruning is gated on the committed cursor`() { + val prune = daoFunctions().getValue("pruneUploadedSyncActions") + assertTrue(prune, prune.contains("getSyncSequence(SYNC_ACTIONS_UPLOADED_CURSOR)")) + val commit = daoFunctions().getValue("commitSyncActionCursorRow") + assertTrue("the cursor must never move backwards", commit.contains("MAX(:value")) + } + + private fun migrationSql(migration: Migration): List { + val sql = mutableListOf() + val db = Proxy.newProxyInstance( + SupportSQLiteDatabase::class.java.classLoader, + arrayOf(SupportSQLiteDatabase::class.java), + ) { _, method, args -> + when (method.name) { + "execSQL" -> { + sql += args?.firstOrNull() as String + null + } + "query" -> emptyCursor() + else -> throw UnsupportedOperationException(method.name) + } + } as SupportSQLiteDatabase + migration.migrate(db) + return sql + } + + private fun emptyCursor(): Cursor = Proxy.newProxyInstance( + Cursor::class.java.classLoader, + arrayOf(Cursor::class.java), + ) { _, method, _ -> + when (method.name) { + "getColumnIndex" -> 0 + "moveToNext" -> false + "close" -> null + else -> throw UnsupportedOperationException(method.name) + } + } as Cursor +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncCursorMigrationTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncCursorMigrationTest.kt new file mode 100644 index 000000000..437a1d700 --- /dev/null +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncCursorMigrationTest.kt @@ -0,0 +1,469 @@ +package expo.modules.vescapecore.telemetry + +import android.database.Cursor +import androidx.room.migration.Migration +import androidx.sqlite.db.SupportSQLiteDatabase +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import java.io.File +import java.lang.reflect.Proxy + +/** + * Incremental-sync cursors: schema 42→43 adds `updated_at` to `boards`, `alerts` and + * `telemetry_minute_buckets`, backfills it from each table's best evidence of last change, and + * indexes it. Schema 43→44 then splits the two jobs that column was doing — `sync_seq` carries the + * Sync Cursor, `updated_at` stays the last-write-wins timestamp. Every write path has to move both. + * + * @parity /modules/vescape-core/ios/telemetry/SyncCursorMigrationTests.swift + */ +class SyncCursorMigrationTest { + /** Table → the column its pre-43 rows backfill from. */ + private val backfillSource = mapOf( + "boards" to "created_at", + "alerts" to "created_at", + "telemetry_minute_buckets" to "last_sample_at_ms", + ) + + private fun migrationSql(migration: Migration): List { + val sql = mutableListOf() + val db = Proxy.newProxyInstance( + SupportSQLiteDatabase::class.java.classLoader, + arrayOf(SupportSQLiteDatabase::class.java), + ) { _, method, args -> + when (method.name) { + "execSQL" -> { + sql += args?.firstOrNull() as String + null + } + "query" -> emptyCursor() + else -> throw UnsupportedOperationException(method.name) + } + } as SupportSQLiteDatabase + migration.migrate(db) + return sql + } + + private fun emptyCursor(): Cursor = Proxy.newProxyInstance( + Cursor::class.java.classLoader, + arrayOf(Cursor::class.java), + ) { _, method, _ -> + when (method.name) { + "getColumnIndex" -> 0 + "moveToNext" -> false + "close" -> null + else -> throw UnsupportedOperationException(method.name) + } + } as Cursor + + private fun migrationSql(): List = migrationSql(TelemetryDatabase.MIGRATION_42_43) + + private fun daoSource(): String = + File("src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt").readText() + + @Test + fun migrationAddsUpdatedAtColumnAndIndexToEverySyncedTable() { + val sql = migrationSql() + + for (table in backfillSource.keys) { + assertTrue( + "missing updated_at column on $table", + sql.any { it == "ALTER TABLE $table ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0" }, + ) + assertTrue( + "missing updated_at index on $table", + sql.any { + it == "CREATE INDEX IF NOT EXISTS index_${table}_updated_at ON $table(updated_at)" + }, + ) + } + } + + /** + * The backfill is the whole point of shipping this as a migration rather than a plain column add: + * a row left at the `DEFAULT 0` would report epoch zero to the server and get re-synced forever. + */ + @Test + fun migrationBackfillsExistingRowsInsteadOfLeavingThemAtZero() { + val sql = migrationSql() + + for ((table, source) in backfillSource) { + val added = sql.indexOf("ALTER TABLE $table ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + val backfilled = sql.indexOf("UPDATE $table SET updated_at = $source") + assertTrue("missing backfill for $table", backfilled >= 0) + // The backfill only works once the column exists. + assertTrue("backfill for $table runs before the column is added", backfilled > added) + } + } + + @Test + fun migrationsTargetTheCurrentSchemaVersion() { + assertEquals(48, TELEMETRY_DATABASE_VERSION) + assertEquals(42, TelemetryDatabase.MIGRATION_42_43.startVersion) + assertEquals(43, TelemetryDatabase.MIGRATION_42_43.endVersion) + assertEquals(43, TelemetryDatabase.MIGRATION_43_44.startVersion) + assertEquals(44, TelemetryDatabase.MIGRATION_43_44.endVersion) + assertEquals(44, TelemetryDatabase.MIGRATION_44_45.startVersion) + assertEquals(45, TelemetryDatabase.MIGRATION_44_45.endVersion) + assertEquals(47, TelemetryDatabase.MIGRATION_47_48.startVersion) + assertEquals(48, TelemetryDatabase.MIGRATION_47_48.endVersion) + } + + /** + * The regression this whole change exists to prevent. `setAlertRuleEnabled` is a targeted UPDATE + * rather than an entity round-trip, so it is the one write path that can silently skip both sync + * columns — toggling an alert would then never reach the server. + * + * Asserted against the DAO source because Room's `@Query` has BINARY retention (invisible to + * runtime reflection) and its generated implementation keeps the SQL in a method-local string. + * A JVM unit test has no other handle on the statement Room will actually run. + */ + @Test + fun setAlertRuleEnabledQueryMovesBothSyncColumns() { + // The statement is written as a concatenation to stay inside the line limit; join it back up + // before matching so the test sees the string Room will compile. + val dao = daoSource().replace(Regex("""\"\s*\+\s*\""""), "") + val query = Regex("""\"(UPDATE alerts SET[^"]*)\"""").find(dao)?.groupValues?.get(1) + + assertEquals( + "UPDATE alerts SET enabled = :enabled, updated_at = MAX(updated_at + 1, :updatedAt), " + + "sync_seq = :syncSeq WHERE board_id = :boardId AND id = :id", + query, + ) + } + + // MARK: Sync Cursor sequence (#275) + + /** + * The Sync Cursor scan runs on `sync_seq`, not on `updated_at`. A wall clock that steps backwards + * lands a write below a cursor the phone has already passed, and the scan never picks it up; a + * counter cannot regress. + */ + @Test + fun syncSeqMigrationAddsColumnIndexAndCounterToEverySyncedTable() { + val sql = migrationSql(TelemetryDatabase.MIGRATION_43_44) + + assertTrue( + "missing sync_sequences table", + sql.any { it.contains("CREATE TABLE IF NOT EXISTS sync_sequences") }, + ) + for (table in SYNC_SEQ_TABLES_V44) { + assertTrue( + "missing sync_seq column on $table", + sql.any { it == "ALTER TABLE $table ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0" }, + ) + assertTrue( + "missing sync_seq index on $table", + sql.any { it == "CREATE INDEX IF NOT EXISTS index_${table}_sync_seq ON $table(sync_seq)" }, + ) + assertTrue( + "missing counter seed for $table", + sql.any { it.contains("INSERT OR REPLACE INTO sync_sequences") && it.contains("'$table'") }, + ) + } + } + + /** + * Existing rows need distinct, increasing positions and the counter has to resume above all of + * them, or the first writes after upgrade reuse numbers the scan would order wrongly. + */ + @Test + fun syncSeqMigrationBackfillsExistingRowsBeforeSeedingTheCounter() { + val sql = migrationSql(TelemetryDatabase.MIGRATION_43_44) + + for (table in SYNC_SEQ_TABLES_V44) { + val backfilled = sql.indexOf("UPDATE $table SET sync_seq = rowid") + val seeded = sql.indexOfFirst { + it.contains("INSERT OR REPLACE INTO sync_sequences") && it.contains("'$table'") + } + assertTrue("missing sync_seq backfill for $table", backfilled >= 0) + assertTrue("counter for $table is seeded before its rows are numbered", seeded > backfilled) + } + } + + /** Every entity write path stamps a fresh position, including the merge branch for buckets. */ + @Test + fun everyEntityWritePathAllocatesASyncSeq() { + val dao = daoSource() + + for (marker in listOf( + "syncSeq = nextSyncSeq(SYNC_SEQ_BOARDS)", + "syncSeq = nextSyncSeq(SYNC_SEQ_ALERTS)", + "syncSeq = nextSyncSeq(SYNC_SEQ_MINUTE_BUCKETS)", + "syncSeq = nextSyncSeq(SYNC_SEQ_BOARD_SETTINGS)", + "syncSeq = nextSyncSeq(SYNC_SEQ_BOARD_WARNINGS)", + "syncSeq = nextSyncSeq(SYNC_SEQ_PRIVACY_ZONES)", + "syncSeq = nextSyncSeq(SYNC_SEQ_TUNE_PROFILES)", + "syncSeq = nextSyncSeq(SYNC_SEQ_FAVORITES)", + )) { + assertTrue("no write path allocates via `$marker`", dao.contains(marker)) + } + // The bucket merge folds into the stored row, so the fresh position has to survive the fold. + assertTrue("bucket merge drops the new sync_seq", dao.contains("syncSeq = next.syncSeq")) + } + + // MARK: The six remaining mutable tables (#281) + + private fun remainingTablesSql(): List = + migrationSql(TelemetryDatabase.MIGRATION_44_45) + + @Test + fun remainingTablesGainColumnIndexAndCounter() { + val sql = remainingTablesSql() + + for (table in SYNC_SEQ_TABLES_V45) { + assertTrue( + "missing sync_seq column on $table", + sql.any { it == "ALTER TABLE $table ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0" }, + ) + assertTrue( + "missing sync_seq index on $table", + sql.any { it == "CREATE INDEX IF NOT EXISTS index_${table}_sync_seq ON $table(sync_seq)" }, + ) + val backfilled = sql.indexOf("UPDATE $table SET sync_seq = rowid") + val seeded = sql.indexOfFirst { + it.contains("INSERT OR REPLACE INTO sync_sequences") && it.contains("'$table'") + } + assertTrue("missing sync_seq backfill for $table", backfilled >= 0) + assertTrue("counter for $table is seeded before its rows are numbered", seeded > backfilled) + } + } + + /** + * `board_warnings` is the one table of the six that never had a wall-clock stamp: without it the + * server has nothing to compare and every re-detection would win or lose arbitrarily. + */ + @Test + fun boardWarningsGainUpdatedAtBackfilledFromItsNewestDetection() { + val sql = remainingTablesSql() + + val added = sql.indexOf("ALTER TABLE board_warnings ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + val backfilled = sql.indexOf("UPDATE board_warnings SET updated_at = last_detected_at") + assertTrue("missing updated_at on board_warnings", added >= 0) + assertTrue("backfill runs before the column is added", backfilled > added) + } + + /** + * Every step is guarded on the column being absent, so a re-run adds nothing and renumbers + * nothing — the counter would otherwise be re-seeded below positions already handed out. + */ + @Test + fun remainingTablesMigrationIsGuardedForReRun() { + val guarded = TelemetryDatabase.MIGRATION_44_45 + val sql = migrationSql(guarded) + + // `migrationSql` answers every column probe with an empty cursor, i.e. "column absent", so this + // run is the first-time path. The guarded statements are exactly the ones missing from a re-run. + assertTrue( + "column adds are unguarded", + sql.any { it.startsWith("ALTER TABLE") }, + ) + for (statement in sql.filter { it.startsWith("CREATE INDEX") }) { + assertTrue("index create is not idempotent: $statement", statement.contains("IF NOT EXISTS")) + } + } + + /** + * Rider identity and this phone's session state live in `app_settings` but name the phone, not the + * Rider (#277). They are excluded by never being given a cursor position: 0 sits below every Sync + * Cursor, so no scan sees the row. The migration's `rowid` backfill has to be undone for them. + */ + @Test + fun phoneLocalSettingsKeysAreExcludedFromTheCursor() { + val sql = remainingTablesSql() + + val reset = sql.single { it.startsWith("UPDATE app_settings SET sync_seq = 0") } + for (key in NOT_SYNCED_SETTING_KEYS) { + assertTrue("phone-local key $key is left syncable", reset.contains("'$key'")) + } + assertTrue("riderName must stay on the phone", "riderName" in NOT_SYNCED_SETTING_KEYS) + assertTrue("liveHistoryLimit is Rider config, not phone state", "liveHistoryLimit" !in NOT_SYNCED_SETTING_KEYS) + + assertTrue( + "app settings write path ignores the phone-local list", + daoSource().contains("if (phoneLocal) 0L else nextSyncSeq(SYNC_SEQ_APP_SETTINGS)"), + ) + } + + /** + * Every targeted `UPDATE` on the six tables bypasses the entity round-trip, which is exactly the + * shape that made `setAlertRuleEnabled` regress: it has to move both columns in its own SQL. + */ + @Test + fun targetedUpdatesOnTheSixTablesMoveBothSyncColumns() { + val dao = daoSource().replace(Regex("""\"\s*\+\s*\""""), "") + + for (statement in Regex("""\"(UPDATE (?:privacy_zones|tune_profiles) SET[^"]*)\"""").findAll(dao)) { + val sql = statement.groupValues[1] + assertTrue("targeted update does not ratchet: $sql", sql.contains("MAX(updated_at + 1, :updatedAt)")) + assertTrue("targeted update does not move the cursor: $sql", sql.contains("sync_seq = :syncSeq")) + } + } + + /** + * The bucket merge used to freeze `updated_at` at the stored value on a backwards clock step, on + * the premise that the server upserts this table unconditionally. It does not — the same + * last-write-wins guard applies, so a frozen stamp is a scanned, sent, silently dropped row. + */ + @Test + fun bucketMergeRatchetsLikeBoardsAndAlerts() { + val dao = daoSource() + + assertTrue( + "bucket merge does not ratchet", + dao.contains("updatedAt = ratchetUpdatedAt(updatedAt, next.updatedAt)"), + ) + assertTrue( + "the retired unconditional-upsert claim is still in the source", + !dao.contains("unconditional upsert"), + ) + } + + // MARK: VESC Fault Evidence (#288) + + private fun faultEvidenceSql(): List = migrationSql(TelemetryDatabase.MIGRATION_47_48) + + @Test + fun faultEvidenceTablesGainColumnIndexAndCounter() { + val sql = faultEvidenceSql() + + for (table in SYNC_SEQ_TABLES_V48) { + assertTrue( + "missing sync_seq column on $table", + sql.any { it == "ALTER TABLE $table ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0" }, + ) + assertTrue( + "missing sync_seq index on $table", + sql.any { it == "CREATE INDEX IF NOT EXISTS index_${table}_sync_seq ON $table(sync_seq)" }, + ) + val backfilled = sql.indexOf("UPDATE $table SET sync_seq = rowid") + val seeded = sql.indexOfFirst { + it.contains("INSERT OR REPLACE INTO sync_sequences") && it.contains("'$table'") + } + assertTrue("missing sync_seq backfill for $table", backfilled >= 0) + assertTrue("counter for $table is seeded before its rows are numbered", seeded > backfilled) + } + // The samples are append-only on an AUTOINCREMENT id, which already is their cursor. + assertTrue( + "capture samples must not be given a redundant counter", + sql.none { it.contains("vesc_fault_capture_samples") }, + ) + } + + /** + * An occurrence changes when the Rider dismisses it, with nothing else about the row moving, so + * the stamp cannot be derived from `last_observed_at` at read time — but it is the truthful + * moment an existing occurrence last changed, so it is what the backfill uses. Left at the + * `DEFAULT 0` the row would report epoch zero and lose every race the server judges. + */ + @Test + fun faultOccurrencesGainUpdatedAtBackfilledFromTheLastObservation() { + val sql = faultEvidenceSql() + + val added = sql.indexOf( + "ALTER TABLE vesc_fault_occurrences ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0", + ) + val backfilled = sql.indexOf("UPDATE vesc_fault_occurrences SET updated_at = last_observed_at") + assertTrue("missing updated_at on vesc_fault_occurrences", added >= 0) + assertTrue("backfill runs before the column is added", backfilled > added) + } + + /** + * Dismissal is a targeted `UPDATE`, the shape that made `setAlertRuleEnabled` regress — and here + * it is the only write that moves `updated_at` at all, because nothing else about the row + * changes. Without the ratchet the Rider's acknowledgement never reaches the server. + */ + @Test + fun dismissingAnOccurrenceMovesBothSyncColumns() { + val dao = daoSource().replace(Regex("""\"\s*\+\s*\""""), "") + val query = Regex("""\"(UPDATE vesc_fault_occurrences SET dismissed[^"]*)\"""") + .find(dao)?.groupValues?.get(1) + + assertEquals( + "UPDATE vesc_fault_occurrences SET dismissed = :dismissed, " + + "updated_at = MAX(updated_at + 1, :updatedAt), sync_seq = :syncSeq WHERE id = :id", + query, + ) + } + + /** The lifecycle advance is the other targeted update on the table, and carries both columns. */ + @Test + fun faultEvidenceWritePathsAllocateASyncSeq() { + val dao = daoSource() + + for (marker in listOf( + "syncSeq = nextSyncSeq(SYNC_SEQ_VESC_FAULT_OCCURRENCES)", + "nextSyncSeq(SYNC_SEQ_VESC_FAULT_CAPTURES)", + )) { + assertTrue("no write path allocates via `$marker`", dao.contains(marker)) + } + assertTrue( + "the fault upsert does not ratchet", + dao.contains("ratchetUpdatedAt(getVescFaultUpdatedAt(fault.id), fault.lastObservedAtMs)"), + ) + val lifecycle = dao.replace(Regex("""\"\s*\+\s*\""""), "") + .substringAfter("UPDATE vesc_fault_occurrences SET last_observed_at") + .substringBefore("\"") + assertTrue("the lifecycle advance drops updated_at", lifecycle.contains("updated_at = :updatedAt")) + assertTrue("the lifecycle advance drops the cursor", lifecycle.contains("sync_seq = :syncSeq")) + } + + // MARK: Last-write-wins ratchet (#275) + + /** + * The server keeps its stored row unless the incoming stamp is strictly newer, so a rewound clock + * that stamps at or below it is a silently dropped edit — freezing the value is not enough. + */ + @Test + fun ratchetStepsPastAStampTheClockCannotBeat() { + assertEquals(1_000L, ratchetUpdatedAt(null, 1_000L)) + // Clock ahead of the stored row: truthful wall clock, no inflation. + assertEquals(5_000L, ratchetUpdatedAt(1_000L, 5_000L)) + // Clock rewound below it, or stalled on it: strictly above. + assertEquals(5_001L, ratchetUpdatedAt(5_000L, 1_000L)) + assertEquals(5_001L, ratchetUpdatedAt(5_000L, 5_000L)) + } + + @Test + fun boardAndAlertUpsertsRatchetAgainstTheStoredStamp() { + val dao = daoSource() + + assertTrue( + "board upsert does not ratchet", + dao.contains("ratchetUpdatedAt(getBoardUpdatedAt(board.id), board.updatedAt)"), + ) + assertTrue( + "alert upsert does not ratchet", + dao.contains("ratchetUpdatedAt(getAlertRuleUpdatedAt(rule.boardId, rule.id), rule.updatedAt)"), + ) + } + + @Test + fun boardAndAlertRuleBridgeShapesCarryTheCursor() { + val board = mapOf( + "id" to "board-1", + "name" to "ADV", + "createdAt" to 1_000L, + // Native ignores a bridge-supplied cursor and stamps its own. + "updatedAt" to 1L, + ).toBoardEntity(now = 2_000L) + + assertEquals(1_000L, board.createdAt) + assertEquals(2_000L, board.updatedAt) + assertEquals(2_000L, board.toMap(emptyList())["updatedAt"]) + + val rule = mapOf( + "boardId" to "board-1", + "id" to "rule-1", + "controlId" to "duty", + "threshold" to 70.0, + "enabled" to true, + "createdAt" to 1_000L, + "updatedAt" to 1L, + ).toAlertRuleEntity(now = 2_000L) + + assertEquals(1_000L, rule.createdAt) + assertEquals(2_000L, rule.updatedAt) + assertEquals(2_000L, rule.toMap()["updatedAt"]) + } +} diff --git a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/TelemetryBoardIdMigrationTest.kt b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/TelemetryBoardIdMigrationTest.kt index 2c208a07e..80a8dc226 100644 --- a/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/TelemetryBoardIdMigrationTest.kt +++ b/modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/TelemetryBoardIdMigrationTest.kt @@ -63,7 +63,7 @@ class TelemetryBoardIdMigrationTest { @Test fun migrationTargetsTheCurrentSchemaVersion() { - assertEquals(42, TELEMETRY_DATABASE_VERSION) + assertEquals(48, TELEMETRY_DATABASE_VERSION) assertEquals(41, TelemetryDatabase.MIGRATION_41_42.startVersion) assertEquals(42, TelemetryDatabase.MIGRATION_41_42.endVersion) } diff --git a/modules/vescape-core/ios/VescapeCoreModule.swift b/modules/vescape-core/ios/VescapeCoreModule.swift index 4d350e0f0..f93a21870 100644 --- a/modules/vescape-core/ios/VescapeCoreModule.swift +++ b/modules/vescape-core/ios/VescapeCoreModule.swift @@ -79,7 +79,7 @@ public class VescapeCoreModule: Module { // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt `Events` // @parity /modules/vescape-core/src/index.ts `VescapeCoreEvents` - Events("onDevice", "onError", "onLiveState", "onLiveTick", "onLiveSeries", "onFocusedSeries", "onTelemetryHistory", "onBms", "onBmsSeries", "onLocation", "onReplayPhoneHeading", "onTelemetryRebuildProgress", "onBoardProbeProgress", "onAppDataChanged", "onGroupRideConnection", "onGroupRideSnapshot", "onGroupRideCreated", "onGroupRideUpdated", "onGroupRideEnded", "onGroupRideJoined", "onGroupRideRoster", "onGroupRideError", "onBoardWarnings", "onVescFaults", "onBoardConfigValues", "onMotorConfigValues", "onBoardConfigChangeNotice", "onBoardLights", "onAppStatus", "onNavigation", "onRouteProgress", "onWeather") + Events("onDevice", "onError", "onLiveState", "onLiveTick", "onLiveSeries", "onFocusedSeries", "onTelemetryHistory", "onBms", "onBmsSeries", "onLocation", "onReplayPhoneHeading", "onTelemetryRebuildProgress", "onBoardProbeProgress", "onAppDataChanged", "onGroupRideConnection", "onGroupRideSnapshot", "onGroupRideCreated", "onGroupRideUpdated", "onGroupRideEnded", "onGroupRideJoined", "onGroupRideRoster", "onGroupRideError", "onBoardWarnings", "onVescFaults", "onBoardConfigValues", "onMotorConfigValues", "onBoardConfigChangeNotice", "onBoardLights", "onAppStatus", "onNavigation", "onRouteProgress", "onWeather", "onSyncStatus") // Track per-event JS listeners so native skips emitting into the void, and gate the whole // firehose on app foreground (see `frontendActive`). Mirrors Android's observing + lifecycle @@ -184,6 +184,12 @@ public class VescapeCoreModule: Module { self.sendEvent("onWeather", ["weather": WeatherCoordinator.shared.current?.map]) } OnStopObserving("onWeather") { self.observedEvents.remove("onWeather") } + OnStartObserving("onSyncStatus") { + self.observedEvents.insert("onSyncStatus") + // Late subscriber: replay the current backup status so JS is immediately consistent. + self.sendSyncStatus(SyncCoordinator.shared.status().toMap()) + } + OnStopObserving("onSyncStatus") { self.observedEvents.remove("onSyncStatus") } OnCreate { // Native owns App Status truth; JS mirrors it. Push every successful refresh (late @@ -206,6 +212,12 @@ public class VescapeCoreModule: Module { // Cold start: fetch App Status before JS asks. A foreground event arriving right after is // coalesced into this request. AppStatusCoordinator.shared.refresh() + // The Device Token outlives the process, so a signed-in phone has to pick the uploader back + // up here: provisioning only happens once, and nothing else would start the loop again. + // Native owns backup state; JS mirrors it. Push every transition (late subscribers replay + // above and through `getSyncStatus`). + SyncCoordinator.shared.onStatusChanged = { [weak self] status in self?.sendSyncStatus(status) } + SyncCoordinator.shared.resumeIfBound() self.attachToCoordinator() // Launch-time honesty pass (ADR 0034): a process killed mid-ride leaves its Live Activity // rendering a confident live session forever. Guarded by the coordinator so a running or @@ -442,6 +454,20 @@ public class VescapeCoreModule: Module { Function("clearDeviceCredential") { NativeAuthCoordinator.shared.clear() } + // The Rider confirmed the destructive Account change; native performs the ordered transition. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt `confirmSyncAccountReset` + AsyncFunction("confirmSyncAccountReset") { + (serverUrl: String, deviceToken: String, accountId: String) async throws -> [String: Any?] in + try await NativeAuthCoordinator.shared.confirmAccountReset( + serverUrl: serverUrl, + token: deviceToken, + accountId: accountId + ) + } + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt `getSyncStatus` + AsyncFunction("getSyncStatus") { () -> [String: Any?] in + SyncCoordinator.shared.status().toMap() + } // Stable Vescape route keeps the app decoupled from the final store destination. // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt `openAppUpdate` @@ -1203,7 +1229,11 @@ public class VescapeCoreModule: Module { createdAt: 0, repeatEverySeconds: normalizedAlertRepeatSeconds((value["repeatEverySeconds"] as? NSNumber)?.doubleValue), beepCount: normalizedAlertBeepCount((value["beepCount"] as? NSNumber)?.intValue), - source: nil + source: nil, + source: nil, + // Ephemeral: the preview rule is never persisted, so it has no last-write-wins timestamp to + // carry and never reaches the upload scan. + updatedAt: 0 ) } @@ -1662,6 +1692,18 @@ public class VescapeCoreModule: Module { } } + /// Emit `onSyncStatus` with the uploader's current state. `sendEvent` must run on the main thread; + /// drop the emit when no JS listener is attached — the replay on subscribe and the next transition + /// self-heal it. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt `onSyncStatus` + /// @parity /modules/vescape-core/src/index.ts `SyncStatusEvent` + private func sendSyncStatus(_ status: [String: Any?]) { + DispatchQueue.main.async { + guard self.shouldEmitToFrontend("onSyncStatus") else { return } + self.sendEvent("onSyncStatus", status) + } + } + /// Map Point failures carry a code JS branches on; anything else is an unexpected native fault. /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/mappoints/MapPointApi.kt `MapPointApiException` private static func rejectMapPoint(_ promise: Promise, _ error: Error) { diff --git a/modules/vescape-core/ios/alerts/AlertEngine.swift b/modules/vescape-core/ios/alerts/AlertEngine.swift index 1153c9a01..865e98f1d 100644 --- a/modules/vescape-core/ios/alerts/AlertEngine.swift +++ b/modules/vescape-core/ios/alerts/AlertEngine.swift @@ -56,6 +56,10 @@ internal struct AlertRule { /// Free-text provenance tag mirroring TS `AlertRule.source`: `manual` (or nil) or `preset`. /// JS authors and regenerates preset rules; native only persists the string. let source: String? + /// Incremental-sync cursor: epoch ms of the last write to this row, from the same clock as + /// `createdAt`. Equal to `createdAt` on insert and bumped on every mutation — including the + /// targeted enable/disable update — so a toggled rule is visible to sync. + let updatedAt: Int64 } /// Adds Legal Mode's per-Board speed warning to in-memory rules. No Alert Rule row is materialized. @@ -86,7 +90,9 @@ internal func withLegalModeOverlay( enabled: true, soundType: "preset:tick", createdAt: 0, - source: nil + source: nil, + // In-memory overlay: no row is ever persisted, so the sync cursor is meaningless here. + updatedAt: 0 ), ] } diff --git a/modules/vescape-core/ios/alerts/AlertEngineTests.swift b/modules/vescape-core/ios/alerts/AlertEngineTests.swift index e414c8f9d..bebc59261 100644 --- a/modules/vescape-core/ios/alerts/AlertEngineTests.swift +++ b/modules/vescape-core/ios/alerts/AlertEngineTests.swift @@ -29,7 +29,8 @@ final class AlertEngineTests: XCTestCase { createdAt: 0, repeatEverySeconds: repeatEverySeconds, beepCount: beepCount, - source: nil + source: nil, + updatedAt: 0 ) } diff --git a/modules/vescape-core/ios/api/ApiResult.swift b/modules/vescape-core/ios/api/ApiResult.swift index 66abebad7..d4f36372c 100644 --- a/modules/vescape-core/ios/api/ApiResult.swift +++ b/modules/vescape-core/ios/api/ApiResult.swift @@ -57,6 +57,10 @@ struct ApiRequest { struct ApiResponse { let status: Int let body: String + /// Lowercased response headers. Only what a caller has to act on crosses this seam today: a `429` + /// carries its delay in `Retry-After`, and guessing one instead would either hammer the server or + /// stall a drain far longer than it asked for. + var headers: [String: String] = [:] } /// The single HTTP seam. Production wires `URLSession`; tests wire a fake and never reach the diff --git a/modules/vescape-core/ios/api/VescapeApi.swift b/modules/vescape-core/ios/api/VescapeApi.swift index 8f0ed2536..cb49e468d 100644 --- a/modules/vescape-core/ios/api/VescapeApi.swift +++ b/modules/vescape-core/ios/api/VescapeApi.swift @@ -56,6 +56,32 @@ final class VescapeApi { return await send(request, authenticated: !token.isEmpty, parse: parse) } + /// One call whose status code is the answer, not an error to classify. The uploader needs `409`, + /// `413` and `429` kept apart — each has a different recovery — so it reads the raw exchange while + /// still going through this class's credential, headers and 401 policy. + /// + /// Never retried here: `POST /api/sync` carries no create key, and the caller's own backoff is + /// what decides when the same batch is offered again. + /// + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/api/VescapeApi.kt `exchange` + func exchange( + _ method: HttpMethod, + path: String, + rawBody: String?, + auth: AuthMode = .required + ) async -> ApiResponse? { + guard let token = token(for: auth) else { return ApiResponse(status: 401, body: "") } + let request = ApiRequest( + method: method, + url: url(path: path, query: [:]), + headers: headers(token: token.isEmpty ? nil : token, hasBody: rawBody != nil), + body: rawBody + ) + guard let response = try? await transport.execute(request) else { return nil } + if response.status == 401 && !token.isEmpty { onUnauthorized() } + return response + } + /// Resolved bearer token, empty when the call goes out anonymously, `nil` when a required /// credential is missing. A credential minted against another origin belongs to another /// environment, so it counts as missing rather than being sent to this one. @@ -195,6 +221,15 @@ struct UrlSessionApiTransport: ApiTransport { guard let http = response as? HTTPURLResponse else { throw NSError(domain: "VescapeApi", code: -2) } - return ApiResponse(status: http.statusCode, body: String(data: data, encoding: .utf8) ?? "") + var headers: [String: String] = [:] + for (name, value) in http.allHeaderFields { + guard let name = name as? String, let value = value as? String else { continue } + headers[name.lowercased()] = value + } + return ApiResponse( + status: http.statusCode, + body: String(data: data, encoding: .utf8) ?? "", + headers: headers + ) } } diff --git a/modules/vescape-core/ios/auth/NativeAuthCoordinator.swift b/modules/vescape-core/ios/auth/NativeAuthCoordinator.swift index aa107a781..13db6cd57 100644 --- a/modules/vescape-core/ios/auth/NativeAuthCoordinator.swift +++ b/modules/vescape-core/ios/auth/NativeAuthCoordinator.swift @@ -51,16 +51,48 @@ final class NativeAuthCoordinator { throw NSError(domain: "NativeAuth", code: -5) } - let credential = DeviceCredential( - serverUrl: origin, - token: token, - accountId: accountId, - expiresAt: nil + // The database is claimed before the credential is stored: a second Account must not be able to + // upload from a database full of the first Account's Boards, Ride History and locations. The + // Rider confirms the destructive reset, and only then does `confirmAccountReset` finish this. + guard SyncCoordinator.shared.bindAccount(accountId) else { + var state = stateMap() + state["accountChangeRequiresReset"] = true + return state + } + + try store.write( + DeviceCredential(serverUrl: origin, token: token, accountId: accountId, expiresAt: nil) + ) + await MainActor.run { + AppStatusCoordinator.shared.refresh() + } + SyncCoordinator.shared.start() + return stateMap() + } + + /// The Rider confirmed that all local app data is erased and cannot yet be restored. + /// + /// One ordered transition: stop the uploader, invalidate in-flight work, replace the app-data + /// database, clear Sync Cursors and pending Sync Actions, bind the fresh database to the new + /// Account, install the new Device Token, start the uploader. Cancelling never reaches here, so + /// the old database and Account binding stay untouched. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/auth/NativeAuthCoordinator.kt `confirmAccountReset` + func confirmAccountReset( + serverUrl: String, + token: String, + accountId: String + ) async throws -> [String: Any?] { + let origin = serverUrl.hasSuffix("/") ? String(serverUrl.dropLast()) : serverUrl + try await SyncCoordinator.shared.resetForAccount(accountId) + // The token is installed before the uploader starts: a loop running on the previous Account's + // credential against the new Account's database is exactly what this ordering exists to prevent. + try store.write( + DeviceCredential(serverUrl: origin, token: token, accountId: accountId, expiresAt: nil) ) - try store.write(credential) await MainActor.run { AppStatusCoordinator.shared.refresh() } + SyncCoordinator.shared.start() return stateMap() } @@ -81,5 +113,10 @@ final class NativeAuthCoordinator { store.clear() } - func clear() { store.clear() } + func clear() { + store.clear() + // Signing out stops the uploader but keeps the Account binding, so data recorded while signed + // out stays protected from retention for the same Account. + SyncCoordinator.shared.stop() + } } diff --git a/modules/vescape-core/ios/config/BoardConfigStore.swift b/modules/vescape-core/ios/config/BoardConfigStore.swift index 825f8b687..66915bf3e 100644 --- a/modules/vescape-core/ios/config/BoardConfigStore.swift +++ b/modules/vescape-core/ios/config/BoardConfigStore.swift @@ -241,6 +241,10 @@ struct BoardConfigStore { try? writer.write { db in try db.execute(sql: "DELETE FROM board_config_values WHERE board_id = ?", arguments: [boardId]) try db.execute(sql: "DELETE FROM board_config_change_notices WHERE board_id = ?", arguments: [boardId]) + // Motor Config Values are decoded against the same link and keyed by the same Board, so they + // are stale for the same reason the other two are. The Android peer already dropped them + // here; leaving them behind kept rows no reader could reach once the Board was gone. + try db.execute(sql: "DELETE FROM motor_config_values WHERE board_id = ?", arguments: [boardId]) } } } diff --git a/modules/vescape-core/ios/faults/VescFaultCaptureStore.swift b/modules/vescape-core/ios/faults/VescFaultCaptureStore.swift index e1bd8535c..cdeea8c58 100644 --- a/modules/vescape-core/ios/faults/VescFaultCaptureStore.swift +++ b/modules/vescape-core/ios/faults/VescFaultCaptureStore.swift @@ -37,13 +37,18 @@ struct VescFaultCaptureStore: VescFaultCaptureStoring { board_id TEXT NOT NULL, started_at INTEGER NOT NULL, opened_at INTEGER NOT NULL, - sample_count INTEGER NOT NULL + sample_count INTEGER NOT NULL, + sync_seq INTEGER NOT NULL DEFAULT 0 ) """) try db.execute(sql: """ CREATE INDEX IF NOT EXISTS index_vesc_fault_captures_board_id ON vesc_fault_captures(board_id) """) + try db.execute(sql: """ + CREATE INDEX IF NOT EXISTS index_vesc_fault_captures_sync_seq + ON vesc_fault_captures(sync_seq) + """) try db.execute(sql: """ CREATE TABLE IF NOT EXISTS vesc_fault_capture_samples ( id INTEGER PRIMARY KEY AUTOINCREMENT, @@ -69,24 +74,32 @@ struct VescFaultCaptureStore: VescFaultCaptureStoring { CREATE INDEX IF NOT EXISTS index_vesc_fault_capture_samples_occurrence_id_captured_at ON vesc_fault_capture_samples(occurrence_id, captured_at) """) + // The Capture write path allocates a Sync Cursor position, so the counter table has to exist + // wherever this schema does — including the test seams that build from here, not from a migrator. + try createSyncSequencesTable(db) } // MARK: - Writes + /// A Capture carries no Change Timestamp — it is a past snapshot with no lifecycle, and the server + /// treats a re-send as a no-op rather than an upsert. It still takes a fresh Sync Cursor position + /// on every write: the sample count is rewritten as the window fills, and a row left at its old + /// position would be one the scan has already passed. func upsertCapture(_ capture: VescFaultCapture) { guard let writer = resolveWriter() else { return } try? writer.write { db in try db.execute( sql: """ INSERT INTO vesc_fault_captures - (occurrence_id, board_id, started_at, opened_at, sample_count) - VALUES (?, ?, ?, ?, ?) + (occurrence_id, board_id, started_at, opened_at, sample_count, sync_seq) + VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT(occurrence_id) DO UPDATE SET - sample_count = excluded.sample_count + sample_count = excluded.sample_count, + sync_seq = excluded.sync_seq """, arguments: [ capture.occurrenceId, capture.boardId, capture.startedAtMs, capture.openedAtMs, - capture.sampleCount, + capture.sampleCount, try nextSyncSeq(db, syncSeqVescFaultCaptures), ] ) } diff --git a/modules/vescape-core/ios/faults/VescFaultStore.swift b/modules/vescape-core/ios/faults/VescFaultStore.swift index cb6a77d3c..95f542d3a 100644 --- a/modules/vescape-core/ios/faults/VescFaultStore.swift +++ b/modules/vescape-core/ios/faults/VescFaultStore.swift @@ -43,13 +43,22 @@ struct VescFaultStore: VescFaultStoring { occurred_at INTEGER NOT NULL, last_observed_at INTEGER NOT NULL, cleared_at INTEGER, - dismissed INTEGER NOT NULL + dismissed INTEGER NOT NULL, + updated_at INTEGER NOT NULL DEFAULT 0, + sync_seq INTEGER NOT NULL DEFAULT 0 ) """) try db.execute(sql: """ CREATE INDEX IF NOT EXISTS index_vesc_fault_occurrences_board_id_occurred_at ON vesc_fault_occurrences(board_id, occurred_at) """) + try db.execute(sql: """ + CREATE INDEX IF NOT EXISTS index_vesc_fault_occurrences_sync_seq + ON vesc_fault_occurrences(sync_seq) + """) + // The write path allocates a Sync Cursor position, so the counter table has to exist wherever + // this schema does — including the test seams that build from here rather than from a migrator. + try createSyncSequencesTable(db) } // MARK: - Reads @@ -100,18 +109,28 @@ struct VescFaultStore: VescFaultStoring { try db.execute( sql: """ INSERT INTO vesc_fault_occurrences - (id, board_id, code, occurred_at, last_observed_at, cleared_at, dismissed) - VALUES (?, ?, ?, ?, ?, ?, ?) + (id, board_id, code, occurred_at, last_observed_at, cleared_at, dismissed, updated_at, + sync_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) -- Lifecycle writes deliberately leave `dismissed` alone: a heartbeat carrying a stale -- in-memory snapshot must never un-dismiss what the rider just acknowledged. Dismissal -- has its own statement. + -- + -- `updated_at` is authored here from the observation time rather than read off the wall + -- clock: it is the moment this write is about, and it is what the migration backfills + -- existing rows from, so the two agree. The `MAX(updated_at + 1, ?)` fold is the same + -- ratchet `ratchetUpdatedAt` applies, expressed in SQL because the conflicting row is + -- already in hand. ON CONFLICT(id) DO UPDATE SET last_observed_at = excluded.last_observed_at, - cleared_at = excluded.cleared_at + cleared_at = excluded.cleared_at, + updated_at = MAX(vesc_fault_occurrences.updated_at + 1, excluded.updated_at), + sync_seq = excluded.sync_seq """, arguments: [ occurrence.id, occurrence.boardId, occurrence.code, occurrence.occurredAtMs, occurrence.lastObservedAtMs, occurrence.clearedAtMs, occurrence.dismissed, + occurrence.lastObservedAtMs, try nextSyncSeq(db, syncSeqVescFaultOccurrences), ] ) } @@ -121,13 +140,22 @@ struct VescFaultStore: VescFaultStoring { } } + /// Both sync columns move here explicitly: an acknowledgement changes the row without the fault + /// being observed again, so nothing else would carry it to the server — and a stamp frozen at the + /// stored value satisfies the scan while the server's last-write-wins guard drops the edit. @discardableResult func setDismissed(_ id: String, _ dismissed: Bool) -> Bool { guard let writer = resolveWriter() else { return false } return (try? writer.write { db -> Bool in try db.execute( - sql: "UPDATE vesc_fault_occurrences SET dismissed = ? WHERE id = ?", - arguments: [dismissed, id] + sql: """ + UPDATE vesc_fault_occurrences + SET dismissed = ?, updated_at = MAX(updated_at + 1, ?), sync_seq = ? + WHERE id = ? + """, + arguments: [ + dismissed, telemetryNowMs(), try nextSyncSeq(db, syncSeqVescFaultOccurrences), id, + ] ) return db.changesCount > 0 }) ?? false diff --git a/modules/vescape-core/ios/faults/VescFaultStoreTests.swift b/modules/vescape-core/ios/faults/VescFaultStoreTests.swift index 8dfa9b9b9..69299d2dc 100644 --- a/modules/vescape-core/ios/faults/VescFaultStoreTests.swift +++ b/modules/vescape-core/ios/faults/VescFaultStoreTests.swift @@ -74,4 +74,58 @@ final class VescFaultStoreTests: XCTestCase { // No foreign key, no cascade: the evidence outlives the Board record on purpose. XCTAssertEqual(store.getForBoard("board").count, 1) } + + // MARK: - Sync columns (#430) + + private func syncColumns(_ id: String) throws -> (updatedAt: Int64, syncSeq: Int64) { + try queue.read { db in + let row = try XCTUnwrap( + try Row.fetchOne( + db, + sql: "SELECT updated_at, sync_seq FROM vesc_fault_occurrences WHERE id = ?", + arguments: [id] + ) + ) + return (row["updated_at"] as Int64, row["sync_seq"] as Int64) + } + } + + /// The whole reason the Occurrence carries its own Change Timestamp: acknowledging a fault edits + /// the row without the fault being observed again, so nothing else would carry it to the server. + func testDismissalMovesBothSyncColumns() throws { + store.upsert(occurrence("a")) + let before = try syncColumns("a") + + XCTAssertTrue(store.setDismissed("a", true)) + + let after = try syncColumns("a") + XCTAssertGreaterThan(after.updatedAt, before.updatedAt) + XCTAssertGreaterThan(after.syncSeq, before.syncSeq) + } + + /// A rewound clock must not leave the row stamped at or below the copy the server already holds: + /// its upsert guard keeps the stored row unless the incoming stamp is strictly newer, so a frozen + /// stamp satisfies the scan and still loses the dismissal. + func testDismissalNeverStampsAtOrBelowTheStoredValue() throws { + store.upsert(occurrence("a")) + let ahead = Int64(Date().timeIntervalSince1970 * 1000) + 3_600_000 + try queue.write { db in + try db.execute(sql: "UPDATE vesc_fault_occurrences SET updated_at = ?", arguments: [ahead]) + } + + XCTAssertTrue(store.setDismissed("a", true)) + + XCTAssertEqual(try syncColumns("a").updatedAt, ahead + 1) + } + + /// A lifecycle write is a change the server has to see too — a heartbeat that only moves + /// `last_observed_at` still has to leave the row above the Sync Cursor. + func testLifecycleUpsertMovesTheCursorPosition() throws { + store.upsert(occurrence("a")) + let before = try syncColumns("a") + + store.upsert(occurrence("a", clearedAtMs: 9_000)) + + XCTAssertGreaterThan(try syncColumns("a").syncSeq, before.syncSeq) + } } diff --git a/modules/vescape-core/ios/recording/RecordingCoordinator.swift b/modules/vescape-core/ios/recording/RecordingCoordinator.swift index 333b90a0b..340f5990b 100644 --- a/modules/vescape-core/ios/recording/RecordingCoordinator.swift +++ b/modules/vescape-core/ios/recording/RecordingCoordinator.swift @@ -83,7 +83,7 @@ internal final class RecordingCoordinator { if let config = activeConfig, enabled { recordMarker(markerType, config: config) } - store.flushBlocking() + flushTelemetryBlocking() activeConfig = nil enabled = false startedAtMs = nil @@ -91,7 +91,7 @@ internal final class RecordingCoordinator { func failSession() { finishDebugRecording(status: "error") - store.flushBlocking() + flushTelemetryBlocking() activeConfig = nil enabled = false startedAtMs = nil @@ -111,7 +111,7 @@ internal final class RecordingCoordinator { if enabled { recordMarker("app_stop", config: config, message: "Recording stopped") } - store.flushBlocking() + flushTelemetryBlocking() enabled = false startedAtMs = nil return true @@ -144,6 +144,15 @@ internal final class RecordingCoordinator { ) } + /// The three ways recording stops — the session finishing, failing, or the Rider switching it + /// off. The flush has to land before the kick, or the uploader scans a ride missing its tail. + /// + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/recording/RecordingCoordinator.kt `flushTelemetryBlocking` + private func flushTelemetryBlocking() { + store.flushBlocking() + SyncCoordinator.shared.notifyRecordingStopped() + } + private func finishDebugRecording(status: String) { recorder?.finish(status: status) recorder = nil diff --git a/modules/vescape-core/ios/sync/FakeSyncServer.swift b/modules/vescape-core/ios/sync/FakeSyncServer.swift new file mode 100644 index 000000000..5b3285950 --- /dev/null +++ b/modules/vescape-core/ios/sync/FakeSyncServer.swift @@ -0,0 +1,67 @@ +import Foundation +@testable import VescapeCore + +/// A server that stores what it is sent and answers with the accepted map the real one would. +/// +/// It exists so a test can assert the only thing that matters end to end: every row the Rider owns +/// reached the server. The engine's own return values cannot show that — a cursor advanced past a +/// row that was never in a batch reports `sent` and looks identical to a correct pass. +/// +/// Stores rows by identity and upserts, exactly like the real one, so a re-send after a lost +/// checkpoint is a no-op rather than a duplicate. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncServer.kt +final class FakeSyncServer { + + /// Every row the server holds, by table and cursor. + var stored: Set = [] + + /// Bodies received, including the ones answered with a failure. + var received: [String] = [] + + /// Rows written, counting re-sends — the cost of failing toward re-sending. + private(set) var writes = 0 + + /// Queued failures, consumed one per request before the server stores anything. + var failures: [SyncResponse] = [] + + /// Fires after the batch is stored but before the response is returned. + var afterStore: (() -> Void)? + + /// Store the next batch and then answer as if the response never arrived. The engine cannot tell + /// this apart from a batch that was never applied, which is exactly why it re-sends. + var loseNextResponse = false + + private let byWire = Dictionary(uniqueKeysWithValues: SyncTable.allCases.map { ($0.wire, $0) }) + + func send(_ body: String) -> SyncResponse { + received.append(body) + if !failures.isEmpty { return failures.removeFirst() } + + let batch = (try? JSONSerialization.jsonObject(with: Data(body.utf8))) as? [String: Any] ?? [:] + var counts: [SyncTable: Int] = [:] + for (wire, value) in batch { + guard let table = byWire[wire], let rows = value as? [[String: Any]] else { continue } + counts[table] = rows.count + writes += rows.count + // Rows from the real store carry their own columns, not a test cursor. Identity tracking is + // best-effort so the same server works for both; `writes` counts every row either way. + for row in rows { + guard let cursor = (row["c"] as? NSNumber)?.int64Value else { continue } + stored.insert(SyncRowRef(table: table, cursor: cursor)) + } + } + afterStore?() + if loseNextResponse { + loseNextResponse = false + return .transient(reason: "timeout") + } + return .accepted(body: accepted(counts)) + } + + /// The server answers for every table it knows, not only the ones the batch carried. + private func accepted(_ counts: [SyncTable: Int]) -> String { + let body = SyncTable.allCases.map { "\"\($0.wire)\":\(counts[$0] ?? 0)" }.joined(separator: ",") + return "{\"accepted\":{\(body)}}" + } +} diff --git a/modules/vescape-core/ios/sync/FakeSyncSource.swift b/modules/vescape-core/ios/sync/FakeSyncSource.swift new file mode 100644 index 000000000..c9d7358a0 --- /dev/null +++ b/modules/vescape-core/ios/sync/FakeSyncSource.swift @@ -0,0 +1,98 @@ +import Foundation +@testable import VescapeCore + +/// A `SyncSource` that models the database the way `SyncStore` actually behaves: rows keyed by their +/// cursor position, a scan that serves strictly `cursor > committed` in ascending order, and a +/// commit that ratchets each table's cursor forward and never backwards. +/// +/// The point of modelling it rather than counting is that the loss bug is a cursor bug. A source +/// that decrements a row counter accepts an advance set naming the wrong position — the exact +/// mistake that makes a row unreachable forever — because the next scan is not derived from what was +/// committed. Here it is, so a wrong advance shows up as a row nobody is ever offered again. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/FakeSyncSource.kt +final class FakeSyncSource: SyncSource { + + /// Cursor positions present per table, ascending — the rows on disk. + private let rows: [SyncTable: [Int64]] + + /// How far each table has been accepted. Absent means nothing delivered. + var cursors: [SyncTable: Int64] = [:] + + /// Every advance set handed to `commit`, in order. + var committed: [[SyncTable: Int64]] = [] + + var currentGeneration: Int64 = 0 + var failures: [(SyncPauseReason, String)] = [] + var encodeFailure: SyncProtocolError? + var commitFailure: Error? + + /// Caps one scan below the engine's own row limit, so a drain takes more than one pass. + var scanLimit = Int.max + + init(_ seed: [SyncTable: [Int64]]) { + rows = seed.mapValues { $0.sorted() } + } + + convenience init(rows count: Int) { + self.init([.boards: count > 0 ? (1...Int64(count)).map { $0 } : []]) + } + + /// Rows the scan will still offer. Zero means the backlog is drained. + var remaining: Int { + rows.reduce(0) { total, entry in + total + entry.value.filter { $0 > cursor(entry.key) }.count + } + } + + private func cursor(_ table: SyncTable) -> Int64 { cursors[table] ?? 0 } + + /// Each row carries its own cursor on the wire, so a fake server can report back exactly which + /// rows it stored. Without row identity in the body, "the server received everything" is not a + /// claim a test can make. + private func rowJson(_ cursor: Int64) -> String { "{\"c\":\(cursor)}" } + + /// Mirrors `SyncStore.pending`: table order, one shared row budget, forward from each cursor. + func pending(rowLimit: Int) throws -> [SyncPendingTable] { + if let encodeFailure { throw encodeFailure } + var tables: [SyncPendingTable] = [] + var budget = min(rowLimit, scanLimit) + for table in SyncTable.allCases { + if budget <= 0 { break } + guard let positions = rows[table] else { continue } + let pending = positions.filter { $0 > cursor(table) }.prefix(budget) + if pending.isEmpty { continue } + tables.append( + SyncPendingTable(table: table, rows: pending.map { SyncPendingRow(cursor: $0, json: rowJson($0)) }) + ) + budget -= pending.count + } + return tables + } + + func pendingCount() -> Int { remaining } + + /// Mirrors `commitSyncCursor`: `MAX(existing, incoming)`, so a cursor never moves backwards. + func commit(_ advances: [SyncTable: Int64]) throws { + if let commitFailure { throw commitFailure } + committed.append(advances) + for (table, position) in advances { cursors[table] = max(cursor(table), position) } + } + + func generation() -> Int64 { currentGeneration } + + func recordPermanentFailure(_ reason: SyncPauseReason, detail: String) { + failures.append((reason, detail)) + } + + /// Every row on disk, for the loss invariant. + func allRows() -> Set { + Set(rows.flatMap { table, positions in positions.map { SyncRowRef(table: table, cursor: $0) } }) + } +} + +/// One row's identity, so a test can compare what is on disk against what the server holds. +struct SyncRowRef: Hashable { + let table: SyncTable + let cursor: Int64 +} diff --git a/modules/vescape-core/ios/sync/SyncAccepted.swift b/modules/vescape-core/ios/sync/SyncAccepted.swift new file mode 100644 index 000000000..d839f284c --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncAccepted.swift @@ -0,0 +1,94 @@ +import Foundation + +/// The `200` body: what the server took, per table. +/// +/// Validated exactly before any cursor moves. A missing table, an extra table, a non-integer count +/// or a count that differs from what was submitted is a protocol failure — the server applies a +/// batch whole, so anything else means the two sides disagree about what was stored, and advancing a +/// cursor on that disagreement is unrecoverable. +/// +/// Parsed here rather than with `JSONSerialization` so the rule behaves identically on both +/// platforms. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncAccepted.kt +enum SyncAccepted { + /// Accepted counts by table, or nil when the body is not exactly the expected response. + static func parse(_ body: String) -> [SyncTable: Int]? { + var counts: [SyncTable: Int] = [:] + var scanner = Scanner(body) + guard scanner.expect("{"), scanner.expectKey("accepted"), scanner.expect("{") else { return nil } + if scanner.peek() != "}" { + while true { + guard let name = scanner.string(), let table = SyncTable(rawValue: name) else { return nil } + if counts[table] != nil || !scanner.expect(":") { return nil } + guard let value = scanner.integer() else { return nil } + counts[table] = value + if scanner.expect(",") { continue } + break + } + } + guard scanner.expect("}"), scanner.expect("}"), scanner.atEnd() else { return nil } + return counts.count == SyncTable.allCases.count ? counts : nil + } + + /// True when the response accounts for exactly the rows submitted, table by table. + static func matches(submitted: [SyncTable: Int], accepted: [SyncTable: Int]) -> Bool { + SyncTable.allCases.allSatisfy { accepted[$0] == (submitted[$0] ?? 0) } + } + + private struct Scanner { + private let source: [Character] + private var index = 0 + + init(_ source: String) { + self.source = Array(source) + } + + mutating func atEnd() -> Bool { + skipSpace() + return index >= source.count + } + + mutating func peek() -> Character? { + skipSpace() + return index < source.count ? source[index] : nil + } + + mutating func expect(_ character: Character) -> Bool { + guard peek() == character else { return false } + index += 1 + return true + } + + mutating func expectKey(_ name: String) -> Bool { + string() == name && expect(":") + } + + mutating func string() -> String? { + guard expect("\"") else { return nil } + var value = "" + // Counts and table names carry no escapes; a body that needs them is not this response. + while index < source.count, source[index] != "\"" { + value.append(source[index]) + index += 1 + } + guard index < source.count else { return nil } + index += 1 + return value + } + + mutating func integer() -> Int? { + skipSpace() + var digits = "" + while index < source.count, source[index].isNumber { + digits.append(source[index]) + index += 1 + } + return digits.isEmpty ? nil : Int(digits) + } + + private mutating func skipSpace() { + while index < source.count, source[index].isWhitespace { index += 1 } + } + } +} diff --git a/modules/vescape-core/ios/sync/SyncAcceptedTests.swift b/modules/vescape-core/ios/sync/SyncAcceptedTests.swift new file mode 100644 index 000000000..840a5b227 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncAcceptedTests.swift @@ -0,0 +1,54 @@ +import XCTest +@testable import VescapeCore + +/// The `200` body is the last thing standing between an accepted batch and a cursor that can never +/// be walked back, so it is validated exactly rather than trusted. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncAcceptedTest.kt +final class SyncAcceptedTests: XCTestCase { + private func body( + _ counts: [SyncTable: Int] = [:], + tables: [SyncTable] = SyncTable.allCases + ) -> String { + let pairs = tables.map { "\"\($0.wire)\":\(counts[$0] ?? 0)" }.joined(separator: ",") + return "{\"accepted\":{\(pairs)}}" + } + + func testEveryTableAccountedForParses() { + let parsed = SyncAccepted.parse(body([.boards: 3])) + XCTAssertEqual(parsed?[.boards], 3) + XCTAssertEqual(parsed?[.favorites], 0) + } + + func testAMissingTableAnExtraTableOrADuplicateIsRefused() { + XCTAssertNil(SyncAccepted.parse(body(tables: Array(SyncTable.allCases.dropFirst())))) + XCTAssertNil(SyncAccepted.parse("{\"accepted\":{\"unknownTable\":0}}")) + XCTAssertNil(SyncAccepted.parse("{\"accepted\":{\"boards\":1,\"boards\":1}}")) + } + + func testAnythingThatIsNotThisResponseIsRefused() { + XCTAssertNil(SyncAccepted.parse("")) + XCTAssertNil(SyncAccepted.parse("{}")) + XCTAssertNil(SyncAccepted.parse("{\"ok\":true}")) + XCTAssertNil(SyncAccepted.parse(body() + "trailing")) + } + + func testCountsHaveToEqualWhatWasSubmitted() throws { + let submitted: [SyncTable: Int] = [.boards: 2] + XCTAssertTrue( + SyncAccepted.matches(submitted: submitted, accepted: try XCTUnwrap(SyncAccepted.parse(body(submitted)))) + ) + XCTAssertFalse( + SyncAccepted.matches( + submitted: submitted, + accepted: try XCTUnwrap(SyncAccepted.parse(body([.boards: 1]))) + ) + ) + XCTAssertFalse( + SyncAccepted.matches( + submitted: submitted, + accepted: try XCTUnwrap(SyncAccepted.parse(body([.boards: 2, .alerts: 1]))) + ) + ) + } +} diff --git a/modules/vescape-core/ios/sync/SyncBatchBuilder.swift b/modules/vescape-core/ios/sync/SyncBatchBuilder.swift new file mode 100644 index 000000000..eea98689a --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncBatchBuilder.swift @@ -0,0 +1,140 @@ +import Foundation + +/// One row waiting to be uploaded: its cursor position and the compact JSON the server will read. +/// +/// The JSON is encoded once, by the wire layer, so the builder measures the bytes that will actually +/// be sent rather than estimating from an object graph. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncBatchBuilder.kt `SyncPendingRow` +struct SyncPendingRow: Equatable { + let cursor: Int64 + let json: String + let byteCount: Int + + init(cursor: Int64, json: String) { + self.cursor = cursor + self.json = json + self.byteCount = json.utf8.count + } +} + +/// One table's pending rows, in cursor order. +struct SyncPendingTable: Equatable { + let table: SyncTable + let rows: [SyncPendingRow] +} + +/// What the builder made of the pending rows. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncBatchBuilder.kt `SyncBatchBuild` +enum SyncBatchBuild: Equatable { + /// Nothing pending. + case empty + + /// A batch and the cursor advance set describing exactly the rows in it. Cursors are committed + /// only after the server accepts, and only these positions move. + case ready(SyncBuiltBatch) + + /// One row cannot fit a batch of its own. Never skipped and never quarantined: the engine pauses + /// with the row retained, because dropping it would silently lose data a Rider believes is backed + /// up. + case rowTooLarge(table: SyncTable, cursor: Int64, byteCount: Int) +} + +struct SyncBuiltBatch: Equatable { + let body: String + /// Table order preserved, so a test can assert parents precede children. + let tables: [SyncTable] + let counts: [SyncTable: Int] + let advances: [SyncTable: Int64] + let rowCount: Int + let byteCount: Int +} + +/// Fills a Sync Batch from per-table pending rows. +/// +/// Pure: no database, no clock, no network. It walks `SyncTable` declaration order — the order the +/// server applies a batch in — and stops at whichever cap comes first. Ordering by backlog size +/// would produce a batch whose children arrive before their parents, which the server refuses whole. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncBatchBuilder.kt `SyncBatchBuilder` +enum SyncBatchBuilder { + static func build( + _ pending: [SyncPendingTable], + rowCap: Int = maxSyncBatchRows, + byteCap: Int = maxSyncBatchBytes + ) -> SyncBatchBuild { + let order = SyncTable.allCases + let ordered = pending + .filter { !$0.rows.isEmpty } + .sorted { left, right in + (order.firstIndex(of: left.table) ?? 0) < (order.firstIndex(of: right.table) ?? 0) + } + if ordered.isEmpty { return .empty } + + var body = "{" + var tables: [SyncTable] = [] + var counts: [SyncTable: Int] = [:] + var advances: [SyncTable: Int64] = [:] + var rowCount = 0 + // `{}`; every other cost below is added as the exact bytes appended. + var byteCount = 2 + + for group in ordered { + if rowCount >= rowCap { break } + // `,"appSettings":[]` — the separating comma only once a table is already open. + let header = (tables.isEmpty ? "" : ",") + "\"" + group.table.wire + "\":[" + let tableOverhead = header.utf8.count + 1 + if byteCount + tableOverhead > byteCap { break } + + var opened = false + var truncated = false + for row in group.rows { + if rowCount >= rowCap { break } + let rowCost = row.byteCount + (opened ? 1 : 0) + let overhead = opened ? 0 : tableOverhead + if byteCount + overhead + rowCost > byteCap { + // A row no empty batch could carry is a permanent local protocol error, not a cap hit. + if tables.isEmpty, !opened, 2 + tableOverhead + row.byteCount > byteCap { + return .rowTooLarge(table: group.table, cursor: row.cursor, byteCount: row.byteCount) + } + truncated = true + break + } + + if !opened { + body += header + byteCount += tableOverhead + tables.append(group.table) + counts[group.table] = 0 + opened = true + } else { + body += "," + } + body += row.json + byteCount += rowCost + rowCount += 1 + counts[group.table] = (counts[group.table] ?? 0) + 1 + advances[group.table] = row.cursor + } + if opened { body += "]" } + // A table cut short by the byte cap may still hold a parent — a Board whose settings, alerts + // or Tune Profiles sit further down this same batch. Carrying on would send the child ahead of + // it, and the server refuses that whole batch on the foreign key. The rest waits for the next + // batch, which starts where this one stopped. + if truncated { break } + } + + if tables.isEmpty { return .empty } + body += "}" + return .ready( + SyncBuiltBatch( + body: body, + tables: tables, + counts: counts, + advances: advances, + rowCount: rowCount, + byteCount: byteCount + ) + ) + } +} diff --git a/modules/vescape-core/ios/sync/SyncBatchBuilderTests.swift b/modules/vescape-core/ios/sync/SyncBatchBuilderTests.swift new file mode 100644 index 000000000..e5979a4d3 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncBatchBuilderTests.swift @@ -0,0 +1,103 @@ +import XCTest +@testable import VescapeCore + +/// The batch builder is pure: no database, no clock, no network. What it has to get right is the +/// order tables go out in, the two caps, and an advance set that describes exactly the rows sent. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncBatchBuilderTest.kt +final class SyncBatchBuilderTests: XCTestCase { + private func rows(_ count: Int, size: Int = 10, from: Int64 = 1) -> [SyncPendingRow] { + (0.. SyncBuiltBatch { + guard case .ready(let batch) = build else { + throw XCTSkip("expected a batch, got \(build)") + } + return batch + } + + func testWalksServerTableOrderRegardlessOfBacklogSize() throws { + let batch = try ready( + SyncBatchBuilder.build([ + SyncPendingTable(table: .telemetryFrames, rows: rows(5)), + SyncPendingTable(table: .boards, rows: rows(1)), + SyncPendingTable(table: .appSettings, rows: rows(1)), + ]) + ) + + XCTAssertEqual(batch.tables, [.appSettings, .boards, .telemetryFrames]) + } + + func testAdvanceSetNamesTheLastRowActuallyIncluded() throws { + let batch = try ready( + SyncBatchBuilder.build( + [ + SyncPendingTable(table: .boards, rows: rows(2, from: 40)), + SyncPendingTable(table: .favorites, rows: rows(3, from: 7)), + ], + rowCap: 4 + ) + ) + + XCTAssertEqual(batch.rowCount, 4) + XCTAssertEqual(batch.counts, [.boards: 2, .favorites: 2]) + XCTAssertEqual(batch.advances, [.boards: 41, .favorites: 8]) + } + + func testExactlyAtAndOneOverTheRowCapBehaveIdentically() throws { + let atCap = try ready(SyncBatchBuilder.build([SyncPendingTable(table: .boards, rows: rows(3))], rowCap: 3)) + XCTAssertEqual(atCap.rowCount, 3) + + let overCap = try ready(SyncBatchBuilder.build([SyncPendingTable(table: .boards, rows: rows(4))], rowCap: 3)) + XCTAssertEqual(overCap.rowCount, 3) + XCTAssertEqual(overCap.advances[.boards], 3) + } + + /// The cap is on the bytes actually sent, so the encoded body is what gets measured. + func testByteCapCountsTheEncodedBodyBoundaryIncluded() throws { + let pending = [SyncPendingTable(table: .boards, rows: rows(2, size: 8))] + let one = try ready(SyncBatchBuilder.build(pending, byteCap: Int.max)) + XCTAssertEqual(one.body.utf8.count, one.byteCount) + + let atCap = try ready(SyncBatchBuilder.build(pending, byteCap: one.byteCount)) + XCTAssertEqual(atCap.rowCount, 2) + + let oneUnder = try ready(SyncBatchBuilder.build(pending, byteCap: one.byteCount - 1)) + XCTAssertEqual(oneUnder.rowCount, 1) + XCTAssertEqual(oneUnder.body.utf8.count, oneUnder.byteCount) + } + + func testMeasuresUtf8BytesRatherThanCharacters() throws { + let row = SyncPendingRow(cursor: 1, json: "\"ąęółśż\"") + let batch = try ready(SyncBatchBuilder.build([SyncPendingTable(table: .boards, rows: [row])])) + XCTAssertEqual(batch.body.utf8.count, batch.byteCount) + } + + func testARowNoEmptyBatchCouldCarryIsAPermanentErrorNotASilentSkip() { + let huge = SyncPendingRow(cursor: 9, json: "\"" + String(repeating: "x", count: 500) + "\"") + let build = SyncBatchBuilder.build([SyncPendingTable(table: .boards, rows: [huge])], byteCap: 100) + XCTAssertEqual(build, .rowTooLarge(table: .boards, cursor: 9, byteCount: huge.byteCount)) + } + + /// A Board left behind by the byte cap must not be followed by its Alert Rules in the same batch — + /// the server writes them in this order and refuses the whole batch on the foreign key. + func testATableTruncatedByTheByteCapEndsTheBatch() throws { + let pending = [ + SyncPendingTable(table: .boards, rows: rows(2, size: 40)), + SyncPendingTable(table: .alerts, rows: rows(1, size: 4)), + ] + let full = try ready(SyncBatchBuilder.build(pending, byteCap: Int.max)) + XCTAssertEqual(full.rowCount, 3) + + let truncated = try ready(SyncBatchBuilder.build(pending, byteCap: full.byteCount - 20)) + XCTAssertEqual(truncated.tables, [.boards]) + XCTAssertEqual(truncated.counts[.boards], 1) + XCTAssertEqual(truncated.body.utf8.count, truncated.byteCount) + } + + func testNothingPendingIsIdleNotAnEmptyBatch() { + XCTAssertEqual(SyncBatchBuilder.build([]), .empty) + XCTAssertEqual(SyncBatchBuilder.build([SyncPendingTable(table: .boards, rows: [])]), .empty) + } +} diff --git a/modules/vescape-core/ios/sync/SyncCoordinator.swift b/modules/vescape-core/ios/sync/SyncCoordinator.swift new file mode 100644 index 000000000..9ed08fa9a --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncCoordinator.swift @@ -0,0 +1,471 @@ +import Foundation +import Network + +/// What JS renders. Native owns every transition; JS only asks and shows. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt `SyncStatus` +struct SyncStatus { + let accountId: String? + let pendingRows: Int + let activity: SyncActivity + let pause: SyncPauseReason? + let lastUploadAtMs: Int64? + + func toMap() -> [String: Any?] { + [ + "accountId": accountId, + "pendingRows": pendingRows, + "activity": activity.slug, + "pause": pause?.slug, + "lastUploadAtMs": lastUploadAtMs, + ] + } +} + +/// The uploader's lifecycle: the loop, the kicks, and the Account binding it runs under. +/// +/// Runs inside the window the app already keeps alive — the existing background modes during a ride, +/// the foreground otherwise. Deliberately no `BGTaskScheduler`: a ride that ends offline on a phone +/// that is never reopened waits for the next app open or the next ride. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt +final class SyncCoordinator { + static let shared = SyncCoordinator() + + internal static let syncPath = "/api/sync" + + /// Samples persisted this recently mean a ride is producing, Idle Pause included. + private static let sampleActivityWindowMs: Int64 = 60_000 + + /// A drain is a burst, not a loop that can never yield to the rest of the process. + private static let maxDrainSteps = 50 + + private let lock = NSLock() + private var generation: Int64 = 0 + private var lastSamplePersistedAtMs: Int64 = 0 + private var lastUploadAtMs: Int64? + private var wifiOnly = false + /// The master switch. Off by default: nothing uploads until the Rider turns backup on. + private var enabled = false + private var onWifi = false + private var online = true + /// Failure keys already recorded this process, so a wedged batch writes one event, not a stream. + private var recordedFailures = Set() + private var loop: Task? + /// Every pass chains onto this, so scan → send → commit never interleaves with another pass or + /// with an Account reset. Cancelled by `stop()` together with the loop. + private var chain: Task? + + private let monitor = NWPathMonitor() + private lazy var store = SyncStore( + generation: { [weak self] in self?.currentGeneration() ?? 0 }, + onPermanentFailure: { [weak self] reason, detail in + self?.recordPermanentFailure(reason, detail: detail) + } + ) + private lazy var engine = SyncEngine( + source: store, + transport: { [weak self] body in + await self?.post(body) ?? .transient(reason: "stopped") + }, + environment: { [weak self] in + self?.environment() ?? SyncEnvironment( + ridingSamples: false, + enabled: false, + online: false, + wifiOnly: false, + onWifi: false, + credentialReady: false, + onlineBlocked: true + ) + } + ) + + private init() { + monitor.pathUpdateHandler = { [weak self] path in + guard let self else { return } + let reachable = path.status == .satisfied + self.lock.lock() + let regained = reachable && !self.online + self.online = reachable + self.onWifi = path.usesInterfaceType(.wifi) + self.lock.unlock() + // Connectivity regained is one of the immediate kicks, next to ride end and sign-in. + if regained { self.kick() } + } + monitor.start(queue: DispatchQueue(label: "app.vescape.sync.path")) + } + + var pauseReason: SyncPauseReason? { engine.pauseReason } + + /// Wired by the module: every status transition, pushed to JS. Native owns the state; JS renders + /// it and never derives one of its own. + var onStatusChanged: (([String: Any?]) -> Void)? + + /// Last status handed out, so an unchanged status emits nothing and raises no second notification. + private var publishedActivity: String? + private var publishedPause: String? + private var publishedPending: Int? + private var publishedUploadAtMs: Int64? + private var publishedAccountId: String? + + /// Recording persisted samples: the ride cadence follows sample production, not session presence. + func notifySamplesPersisted(atMs: Int64 = telemetryNowMs()) { + lock.lock() + lastSamplePersistedAtMs = atMs + lock.unlock() + } + + /// The ride ended and its last samples are on disk. Called after the final flush, so the kick + /// scans a complete ride rather than one missing its tail. + /// + /// This is the moment with the largest fresh backlog and the moment a Rider is most likely to open + /// the app and look at the status line, which is why it does not wait for the next tick. + /// + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt `notifyRecordingStopped` + func notifyRecordingStopped() { + kick() + } + + /// A Rider changed something durable and small — a Favorite pinned, renamed or unpinned. One row, + /// created by hand, and the Rider is looking at the screen that says whether it is backed up, so a + /// five-minute wait reads as the backup not working. + /// + /// Deliberately not wired to telemetry writes: those arrive at 2 Hz and already have the ride + /// cadence. This is for edits a Rider makes, which are rare and individually visible. + /// + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt `notifyRiderEdit` + func notifyRiderEdit() { + kick() + } + + /// The master switch, from the App Setting native owns. Off stops the loop outright — no scan, no + /// request, no backoff, no pause notification — rather than leaving a loop that decides to do + /// nothing every five minutes. + func setEnabled(_ value: Bool) { + lock.lock() + let changed = enabled != value + enabled = value + lock.unlock() + guard changed else { return } + if value { + start() + } else { + stop() + // A switched-off uploader asks the Rider for nothing: an earlier pause is no longer theirs to + // act on until they turn backup back on. + SyncNotifier.shared.update(nil) + } + publishStatus() + } + + /// The "Back up over Wi-Fi only" App Setting, pushed by `AppDataRepository` on every write and + /// read back on launch. Native reads the setting itself — JS never carries the switch to the + /// uploader. + func setWifiOnly(_ enabled: Bool) { + lock.lock() + let changed = wifiOnly != enabled + wifiOnly = enabled + lock.unlock() + guard changed else { return } + kick() + publishStatus() + } + + func status() -> SyncStatus { + lock.lock() + let uploadedAt = lastUploadAtMs + lock.unlock() + let environment = environment() + let pending = store.pendingCount() + let pause = engine.pauseReason + return SyncStatus( + accountId: store.boundAccountId(), + pendingRows: pending, + activity: SyncPolicy.describe( + SyncState( + nowMs: telemetryNowMs(), + pendingRows: pending, + ridingSamples: environment.ridingSamples, + enabled: environment.enabled, + online: environment.online, + wifiOnly: environment.wifiOnly, + onWifi: environment.onWifi, + credentialReady: environment.credentialReady, + onlineBlocked: environment.onlineBlocked, + pause: pause, + // Backoff is invisible to the Rider: a batch waiting to be retried is still syncing. + retryAtMs: 0 + ) + ), + pause: pause, + lastUploadAtMs: uploadedAt + ) + } + + /// Emit the current status when it differs from the last one, and raise the notification a pause + /// needs: a permanent failure does not resolve through ordinary retry, so a backup that stopped + /// weeks ago must not wait for the Rider to open the social sheet. + private func publishStatus() { + let status = status() + lock.lock() + let unchanged = status.activity.slug == publishedActivity + && status.pause?.slug == publishedPause + && status.pendingRows == publishedPending + && status.lastUploadAtMs == publishedUploadAtMs + && status.accountId == publishedAccountId + let previousPause = publishedPause + publishedActivity = status.activity.slug + publishedPause = status.pause?.slug + publishedPending = status.pendingRows + publishedUploadAtMs = status.lastUploadAtMs + publishedAccountId = status.accountId + lock.unlock() + guard !unchanged else { return } + if status.pause?.slug != previousPause { SyncNotifier.shared.update(status.pause) } + onStatusChanged?(status.toMap()) + } + + /// Pick the uploader back up on a cold launch: the credential outlives the process, so a phone + /// that was signed in stays signed in, and nothing else would ever start the loop again. Binding + /// the stored Account is a no-op when this database already belongs to it, and cannot claim a + /// database that belongs to another one. + func resumeIfBound() { + // The switch is a durable App Setting, so the uploader restores it before the first pass of this + // process — otherwise a cold launch on mobile data would upload once before JS loaded. + let settings = AppDataRepository.shared.getSettings() + setWifiOnly(settings["syncWifiOnly"] as? Bool ?? false) + // Binding still happens with the switch off — it is what makes this database's Account known, + // and starting the loop is the only thing the switch gates. + if let credential = DeviceCredentialStore.shared.read() { + bindAccount(credential.accountId) + } + setEnabled((settings["syncEnabled"] as? Bool) ?? false) + publishStatus() + } + + func start() { + lock.lock() + let running = enabled + lock.unlock() + guard running, loop == nil else { return } + loop = Task { [weak self] in + while !Task.isCancelled { + guard let self else { return } + let waitMs = await self.serialized { await self.pass() } + self.publishStatus() + try? await Task.sleep(nanoseconds: UInt64(max(waitMs, 0)) * 1_000_000) + } + } + } + + /// Stops the loop and every pass in flight, so nothing is left running over a replaced database. + func stop() { + loop?.cancel() + loop = nil + chain?.cancel() + chain = nil + } + + /// Connectivity regained, ride ended, sign-in: send now rather than waiting for the next tick. + func kick() { + lock.lock() + let running = enabled + lock.unlock() + guard running else { return } + guard loop != nil else { return start() } + Task { [weak self] in + guard let self else { return } + _ = await self.serialized { await self.pass() } + self.publishStatus() + } + } + + /// Runs `work` after whatever is already queued, so a scan, its request and its cursor commit + /// always complete against one database — an Account reset waits its turn rather than landing in + /// the middle. + private func serialized(_ work: @escaping () async -> T) async -> T { + lock.lock() + let previous = chain + let task = Task { + await previous?.value + return await work() + } + // The chain only has to say "the previous link finished", so its own value is discarded. + chain = Task { _ = await task.value } + lock.unlock() + return await task.value + } + + /// One pass, draining while the server keeps accepting: a `200` with rows still pending sends + /// again straight away, so a long backlog drains instead of trickling. + private func pass() async -> Int64 { + var drains = 0 + while drains < Self.maxDrainSteps { + switch await engine.runOnce() { + case .sent(_, let morePending): + lock.lock() + lastUploadAtMs = telemetryNowMs() + lock.unlock() + if !morePending { return interval() } + drains += 1 + // Nothing was accepted, but the next attempt differs — a narrowed byte target. + case .retry: + drains += 1 + case .waiting(let untilMs): + return min(max(untilMs - telemetryNowMs(), 0), SyncPolicy.backoffMaxMs) + case .paused: + return SyncPolicy.idleIntervalMs + case .idle: + return interval() + } + } + // A drain that never finishes yields rather than spinning; the next tick resumes it. + return SyncPolicy.rideIntervalMs + } + + private func interval() -> Int64 { + samplesProducing() ? SyncPolicy.rideIntervalMs : SyncPolicy.idleIntervalMs + } + + private func samplesProducing() -> Bool { + lock.lock() + defer { lock.unlock() } + return telemetryNowMs() - lastSamplePersistedAtMs < Self.sampleActivityWindowMs + } + + private func currentGeneration() -> Int64 { + lock.lock() + defer { lock.unlock() } + return generation + } + + private func environment() -> SyncEnvironment { + lock.lock() + let reachable = online + let wifi = onWifi + let meteredOnly = wifiOnly + let running = enabled + lock.unlock() + let status = AppStatusCoordinator.shared.current?.version.status + return SyncEnvironment( + ridingSamples: samplesProducing(), + enabled: running, + online: reachable, + wifiOnly: meteredOnly, + onWifi: wifi, + credentialReady: DeviceCredentialStore.shared.read() != nil, + onlineBlocked: status == .onlineBlocked || status == .appBlocked + ) + } + + /// The Sync endpoints are Online Capabilities behind the App Status gate, and they authenticate + /// with the shared Device Token, so the whole call goes through `VescapeApi`. + private func post(_ body: String) async -> SyncResponse { + let api = VescapeApi.forOrigin(AppStatusCoordinator.serverBaseUrl) + guard let response = await api.exchange(.post, path: Self.syncPath, rawBody: body) else { + return .transient(reason: "network") + } + switch response.status { + case 200: return .accepted(body: response.body) + case 401: return .unauthorized + case 413: return .tooLarge + case 429: return .rateLimited(retryAfterMs: retryAfterMs(response.headers)) + case 500...599: return .transient(reason: "http \(response.status)") + case 400...499: return .invalid(status: response.status, error: errorSlug(response.body)) + // A `2xx` that is not the accepted map is a protocol failure, not a success to interpret. + default: return .invalid(status: response.status, error: "unexpected-success") + } + } + + /// The server's own delay in seconds, or the first backoff step when it named none. + private func retryAfterMs(_ headers: [String: String]) -> Int64 { + guard let value = headers["retry-after"], let seconds = Int64(value.trimmingCharacters(in: .whitespaces)) + else { return SyncPolicy.backoffStartMs } + return seconds * 1_000 + } + + private func errorSlug(_ body: String) -> String { + guard let data = body.data(using: .utf8), + let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let error = json["error"] as? String, !error.isEmpty + else { return "invalid-request" } + return error + } + + // Account binding — the Device Token exchange returns a stable server Account id, and the first + // Account claims this database. + + /// Claim the local database for `accountId` when it is unbound or already belongs to it. + /// + /// False means a different Account: cursors are deliberately not reset over the existing rows, + /// because that would upload the previous Account's Boards, Ride History, locations and settings + /// to the new one. The Rider has to confirm the destructive reset first. + @discardableResult + func bindAccount(_ accountId: String) -> Bool { + let bound = store.bindAccount(accountId) + if bound { + engine.resume() + kick() + } + return bound + } + + /// The Account change transition, in the one order that cannot leak data between Accounts: stop + /// the loop, invalidate in-flight work, replace the database, clear cursors and pending actions, + /// bind the new Account, then start again. + /// + /// The wipe is local maintenance and emits no Sync Actions to either Account — replacing the file + /// removes the log with everything else. + func resetForAccount(_ accountId: String) async throws { + stop() + // Queued behind any pass still in flight: one that started before `stop()` finishes its scan, + // send and commit against the old database before the file is replaced, and none can start + // midway through the transition. + let outcome: Result = await serialized { [self] in + lock.lock() + // Every in-flight response now belongs to a previous Account and can no longer commit. + generation += 1 + recordedFailures.removeAll() + lastUploadAtMs = nil + lock.unlock() + + do { + try TelemetryDatabase.replaceWithFreshDatabase() + guard store.bindAccount(accountId) else { + throw SyncStoreError.databaseUnavailable + } + engine.resume() + return .success(()) + } catch { + return .failure(error) + } + } + try outcome.get() + publishStatus() + // Deliberately not started here: the caller installs the new Device Token first, so the loop + // never runs with the previous Account's credential against the new Account's database. + } + + /// One coalesced Diagnostic Event per failure class, table and cursor. Metadata only: an error + /// code, a table, a cursor and the app version — never row contents, coordinates, the Device + /// Token, the server body or an opaque database error. + private func recordPermanentFailure(_ reason: SyncPauseReason, detail: String) { + let key = "\(reason.slug):\(detail)" + lock.lock() + let isNew = recordedFailures.insert(key).inserted + lock.unlock() + guard isNew else { return } + + TelemetryRepository.shared.recordDiagnosticEvent( + eventName: "sync_upload_paused", + properties: [ + "operation": "sync", + "phase": reason.slug, + "message": "Sync upload paused", + "sync_failure": reason.slug, + "sync_detail": detail, + "app_version": AppStatusCoordinator.installedMarketingVersion(), + ] + ) + } +} diff --git a/modules/vescape-core/ios/sync/SyncDrainTests.swift b/modules/vescape-core/ios/sync/SyncDrainTests.swift new file mode 100644 index 000000000..a4e34fa5e --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncDrainTests.swift @@ -0,0 +1,204 @@ +import XCTest +@testable import VescapeCore + +/// The invariant the whole uploader exists to hold: **a row the Rider owns is never left behind.** +/// +/// Every other sync test checks one decision in isolation. These run a backlog all the way to zero +/// against a server that stores what it is sent and a source that scans forward from what was +/// committed, then compare the two sets. That closes the loop the unit tests leave open — a cursor +/// advanced past a row that never went in a batch is indistinguishable from a correct pass when you +/// only look at the engine's return value. +/// +/// The direction of failure is asserted too: after a lost response or a lost checkpoint, rows may be +/// re-sent (the server upserts them) but must never be skipped. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncDrainTest.kt +final class SyncDrainTests: XCTestCase { + + private var now: Int64 = 1_000 + + override func setUp() { + super.setUp() + now = 1_000 + } + + private func engine(_ source: SyncSource, _ server: FakeSyncServer) -> SyncEngine { + SyncEngine( + source: source, + transport: { body in server.send(body) }, + environment: { + SyncEnvironment( + ridingSamples: false, + enabled: true, + online: true, + wifiOnly: false, + onWifi: false, + credentialReady: true, + onlineBlocked: false + ) + }, + clock: { self.now } + ) + } + + /// Runs passes until the backlog is drained, stepping the clock over any backoff so a transient + /// failure costs a wait rather than ending the test. Bounded: a loop that stops making progress + /// fails here rather than hanging. + private func drain(_ engine: SyncEngine, _ source: FakeSyncSource, maxPasses: Int = 200) async { + var passes = 0 + while source.remaining > 0, passes < maxPasses { + switch await engine.runOnce() { + case let .waiting(untilMs): now = max(now, untilMs) + 1 + case .paused: return + default: break + } + passes += 1 + } + } + + private func backlog(_ tables: [SyncTable: Int]) -> FakeSyncSource { + FakeSyncSource(tables.mapValues { count in (1...Int64(count)).map { $0 } }) + } + + func testAFullDrainDeliversEveryRowExactlyOnce() async { + let source = backlog([.boards: 5, .favorites: 3]) + let server = FakeSyncServer() + + await drain(engine(source, server), source) + + XCTAssertEqual(source.allRows(), server.stored) + XCTAssertEqual(source.remaining, 0) + // Nothing failed, so nothing had to be re-sent. + XCTAssertEqual(server.writes, source.allRows().count) + } + + /// The scan is the only thing that decides what goes next, so a small budget must not open a gap. + func testABacklogLargerThanOneScanStillLosesNothing() async { + let source = backlog([.boards: 17, .alerts: 11, .favorites: 4]) + source.scanLimit = 3 + let server = FakeSyncServer() + + await drain(engine(source, server), source) + + XCTAssertEqual(source.allRows(), server.stored) + XCTAssertEqual(server.writes, source.allRows().count) + } + + /// The response was lost, not the write. The engine cannot tell those apart, so it re-sends — and + /// the rows must arrive, once, because the server upserts on identity. + func testABatchStoredButNeverAcknowledgedIsResentNotSkipped() async { + let source = backlog([.boards: 6]) + source.scanLimit = 2 + let server = FakeSyncServer() + server.loseNextResponse = true + let engine = engine(source, server) + + _ = await engine.runOnce() + XCTAssertTrue(server.stored.contains(SyncRowRef(table: .boards, cursor: 1))) + XCTAssertTrue(source.committed.isEmpty, "nothing may be checkpointed") + + await drain(engine, source) + + XCTAssertEqual(source.allRows(), server.stored) + XCTAssertEqual(source.remaining, 0) + } + + /// The server took the rows; the checkpoint did not land. Re-sending is the only safe direction. + func testALostCursorCommitResendsTheSameRowsAndStillDrains() async { + let source = backlog([.boards: 6]) + source.scanLimit = 2 + let server = FakeSyncServer() + source.commitFailure = SyncStoreError.databaseUnavailable + + let engine = engine(source, server) + _ = await engine.runOnce() + XCTAssertTrue(source.committed.isEmpty, "nothing may be checkpointed") + XCTAssertEqual(source.remaining, 6) + + source.commitFailure = nil + await drain(engine, source) + + XCTAssertEqual(source.allRows(), server.stored) + // The first batch went twice: failing toward a re-send is the whole design. + XCTAssertGreaterThan(server.writes, source.allRows().count) + } + + func testATransientFailurePartWayThroughADrainLosesNothing() async { + let source = backlog([.boards: 9, .privacyZones: 5]) + source.scanLimit = 2 + let server = FakeSyncServer() + let engine = engine(source, server) + + _ = await engine.runOnce() + server.failures = [ + .transient(reason: "5xx"), + .transient(reason: "5xx"), + .rateLimited(retryAfterMs: 30_000), + ] + + await drain(engine, source) + + XCTAssertEqual(source.allRows(), server.stored) + XCTAssertEqual(source.remaining, 0) + } + + /// A committed cursor is a promise that everything below it reached the server. Checked after + /// every pass rather than at the end, because a mid-drain violation self-heals by the time the + /// backlog is empty and would otherwise go unseen. + func testNoCursorEverMovesPastARowTheServerDoesNotHold() async { + let source = backlog([.boards: 8, .alerts: 6, .favorites: 5]) + source.scanLimit = 3 + let server = FakeSyncServer() + server.failures = [.transient(reason: "5xx")] + let engine = engine(source, server) + + var passes = 0 + while source.remaining > 0, passes < 100 { + switch await engine.runOnce() { + case let .waiting(untilMs): now = max(now, untilMs) + 1 + case .paused: passes = 100 + default: break + } + for (table, cursor) in source.cursors where cursor > 0 { + for position in 1...cursor { + XCTAssertTrue( + server.stored.contains(SyncRowRef(table: table, cursor: position)), + "\(table) cursor reached \(cursor) but the server never received \(position)" + ) + } + } + passes += 1 + } + + XCTAssertEqual(source.allRows(), server.stored) + } + + /// The Account changed while the request was in flight. The response belongs to the previous + /// database, so nothing may be checkpointed — and every row stays pending for whoever owns it now. + func testAResponseThatOutlivedItsAccountCheckpointsNothingAndStrandsNoRow() async { + let source = backlog([.boards: 4]) + let server = FakeSyncServer() + server.afterStore = { source.currentGeneration += 1 } + + let pass = await engine(source, server).runOnce() + + XCTAssertEqual(pass, .idle) + XCTAssertTrue(source.committed.isEmpty) + XCTAssertTrue(source.cursors.isEmpty) + XCTAssertEqual(source.remaining, 4) + } + + /// A permanent pause must strand the batch in place: retained, not consumed. + func testARefusedBatchLeavesTheWholeBacklogPending() async { + let source = backlog([.boards: 4, .favorites: 2]) + let server = FakeSyncServer() + server.failures = [.invalid(status: 409, error: "dependency-conflict")] + + let pass = await engine(source, server).runOnce() + + XCTAssertEqual(pass, .paused(.protocolFailure)) + XCTAssertTrue(source.committed.isEmpty) + XCTAssertEqual(source.remaining, 6) + XCTAssertTrue(server.stored.isEmpty) + } +} diff --git a/modules/vescape-core/ios/sync/SyncEngine.swift b/modules/vescape-core/ios/sync/SyncEngine.swift new file mode 100644 index 000000000..69bd8834b --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncEngine.swift @@ -0,0 +1,222 @@ +import Foundation + +/// What the transport made of one `POST /api/sync`. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncEngine.kt `SyncResponse` +enum SyncResponse { + /// `2xx`. The body still has to be exactly the accepted map before anything is committed. + case accepted(body: String) + /// `400`, `409`, `422` or any other unknown `4xx`: wrong request, not a bad moment. + case invalid(status: Int, error: String) + /// `401`: the Device Token is dead. Only sign-in resolves it. + case unauthorized + /// `413`: over the wire byte bound. Retried with a smaller target, never with fewer rows dropped. + case tooLarge + /// `429`, with the server's own delay. + case rateLimited(retryAfterMs: Int64) + /// `5xx`, a network error or a timeout — the batch may or may not have been applied. + case transient(reason: String) +} + +/// The database side of the uploader: what is pending, and where the cursors are. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncEngine.kt `SyncSource` +protocol SyncSource { + /// Pending rows per table, already encoded, capped at `rowLimit` rows in total. + func pending(rowLimit: Int) throws -> [SyncPendingTable] + + /// Rows waiting across every table. Cheap enough to ask on every tick. + func pendingCount() -> Int + + /// Commit the advance set in its own transaction, after the response. Never alongside the rows: a + /// cursor advanced past rows the server did not take is unrecoverable, whereas a cursor left + /// behind is a re-send the server upserts idempotently. Always fail toward re-sending. + /// + /// Throws rather than swallowing a write failure: an uncommitted cursor leaves the same rows + /// pending, and a caller that believed the checkpoint landed would resend them without pause. + func commit(_ advances: [SyncTable: Int64]) throws + + /// Bumped by an Account change. Captured before a request and re-read before the commit, so a + /// response belonging to the previous Account becomes a no-op instead of advancing a cursor over + /// the fresh database. + func generation() -> Int64 + + /// One coalesced, metadata-only Diagnostic Event for a permanent failure. + func recordPermanentFailure(_ reason: SyncPauseReason, detail: String) +} + +/// Environment the policy reads. Owned by the caller, so the engine keeps no platform types. +struct SyncEnvironment { + let ridingSamples: Bool + /// The Rider's master switch, read from the App Setting native owns. + let enabled: Bool + let online: Bool + let wifiOnly: Bool + let onWifi: Bool + let credentialReady: Bool + let onlineBlocked: Bool +} + +/// What one pass did, for the loop and for tests. +enum SyncPass: Equatable { + case idle + /// Nothing was accepted, but the next attempt differs from this one — a narrowed byte target. + case retry + case sent(rowCount: Int, morePending: Bool) + case waiting(untilMs: Int64) + case paused(SyncPauseReason) +} + +/// The uploader: scan forward from each Sync Cursor, send a small batch, advance only what the +/// server accepted. +/// +/// Owns transport policy, backoff and the permanent pause; the two interesting decisions — which +/// rows go in a batch, and whether to send at all — live in `SyncBatchBuilder` and `SyncPolicy`, +/// which are pure. Drives no timer of its own: `SyncCoordinator` owns the loop and the kicks. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncEngine.kt `SyncEngine` +final class SyncEngine { + /// Below this a batch cannot hold a realistic row, so shrinking further only hides the real fault. + private static let minByteTarget = 16 * 1024 + + private let source: SyncSource + private let transport: (String) async -> SyncResponse + private let environment: () -> SyncEnvironment + private let clock: () -> Int64 + + private var retryAtMs: Int64 = 0 + private var backoffMs: Int64 = 0 + private var byteTarget = maxSyncBatchBytes + private(set) var pauseReason: SyncPauseReason? + + init( + source: SyncSource, + transport: @escaping (String) async -> SyncResponse, + environment: @escaping () -> SyncEnvironment, + clock: @escaping () -> Int64 = { telemetryNowMs() } + ) { + self.source = source + self.transport = transport + self.environment = environment + self.clock = clock + } + + /// Clears a pause. Sign-in and an Account reset are the only things that may. + func resume() { + pauseReason = nil + retryAtMs = 0 + backoffMs = 0 + byteTarget = maxSyncBatchBytes + } + + /// One pass: decide, send, commit. A `200` with rows still pending returns `morePending`, so the + /// loop sends again immediately rather than trickling a long backlog one tick at a time. + func runOnce() async -> SyncPass { + let env = environment() + let decision = SyncPolicy.decide( + SyncState( + nowMs: clock(), + pendingRows: source.pendingCount(), + ridingSamples: env.ridingSamples, + enabled: env.enabled, + online: env.online, + wifiOnly: env.wifiOnly, + onWifi: env.onWifi, + credentialReady: env.credentialReady, + onlineBlocked: env.onlineBlocked, + pause: pauseReason, + retryAtMs: retryAtMs + ) + ) + switch decision { + case .paused(let reason): return .paused(reason) + case .wait(let atMs): return .waiting(untilMs: atMs) + case .sendNow: return await send() + } + } + + private func send() async -> SyncPass { + // Captured before the rows are read, not after: an Account reset between the scan and the + // request would otherwise leave a batch of the previous Account's rows looking current, and its + // cursor advance would land on the fresh database. + let generation = source.generation() + let pending: [SyncPendingTable] + do { + pending = try source.pending(rowLimit: maxSyncBatchRows) + } catch let error as SyncProtocolError { + return pause(.protocolFailure, detail: "\(error.table.wire).\(error.field)") + } catch { + return pause(.protocolFailure, detail: "encode") + } + + switch SyncBatchBuilder.build(pending, rowCap: maxSyncBatchRows, byteCap: byteTarget) { + case .empty: + return .idle + case .rowTooLarge(let table, let cursor, _): + return pause(.rowTooLarge, detail: "\(table.wire)@\(cursor)") + case .ready(let batch): + return await deliver(batch, generation: generation) + } + } + + private func deliver(_ batch: SyncBuiltBatch, generation: Int64) async -> SyncPass { + let response = await transport(batch.body) + // A response that outlived its Account cannot touch the fresh database it would land in. + if source.generation() != generation { return .idle } + + switch response { + case .accepted(let body): return accept(batch, body: body) + case .unauthorized: return pause(.authentication, detail: "401") + case .invalid(let status, let error): return pause(.protocolFailure, detail: "\(status):\(error)") + case .tooLarge: return shrink(batch) + case .rateLimited(let retryAfterMs): return backOff(max(retryAfterMs, 0)) + case .transient: + backoffMs = SyncPolicy.nextBackoffMs(backoffMs) + return backOff(backoffMs) + } + } + + private func accept(_ batch: SyncBuiltBatch, body: String) -> SyncPass { + guard let accepted = SyncAccepted.parse(body), + SyncAccepted.matches(submitted: batch.counts, accepted: accepted) + else { + return pause(.protocolFailure, detail: "acceptedMismatch") + } + do { + try source.commit(batch.advances) + } catch { + // The server took the rows but the checkpoint did not land. Backing off re-sends the identical + // batch, which the server upserts idempotently — reporting success here would spin instead, + // because the same rows are still pending. + backoffMs = SyncPolicy.nextBackoffMs(backoffMs) + return backOff(backoffMs) + } + backoffMs = 0 + retryAtMs = 0 + byteTarget = maxSyncBatchBytes + return .sent(rowCount: batch.rowCount, morePending: source.pendingCount() > 0) + } + + /// `413` narrows the byte target instead of dropping anything. Once the target can no longer hold + /// even one row, that row is a permanent local protocol error — it is retained, not skipped. + private func shrink(_ batch: SyncBuiltBatch) -> SyncPass { + let table = batch.tables.first + let detail = "\(table?.wire ?? "batch")@\(table.flatMap { batch.advances[$0] } ?? 0)" + if batch.rowCount <= 1 { return pause(.rowTooLarge, detail: detail) } + // Already as small as a batch gets: halving again would resend the same bytes forever, so the + // disagreement about the wire limit is treated as what it is — permanent, with the rows kept. + if byteTarget <= Self.minByteTarget { return pause(.rowTooLarge, detail: detail) } + + byteTarget = max(byteTarget / 2, Self.minByteTarget) + return .retry + } + + private func backOff(_ delayMs: Int64) -> SyncPass { + retryAtMs = clock() + delayMs + return .waiting(untilMs: retryAtMs) + } + + private func pause(_ reason: SyncPauseReason, detail: String) -> SyncPass { + pauseReason = reason + source.recordPermanentFailure(reason, detail: detail) + return .paused(reason) + } +} diff --git a/modules/vescape-core/ios/sync/SyncEngineTests.swift b/modules/vescape-core/ios/sync/SyncEngineTests.swift new file mode 100644 index 000000000..f62ba39a5 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncEngineTests.swift @@ -0,0 +1,225 @@ +import XCTest +@testable import VescapeCore + +/// The engine against a fake transport: the cases that decide whether a Rider's data survives — a +/// wedged batch, a failure part-way through a drain, a dead token, and a response that outlived the +/// Account it was sent for. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncEngineTest.kt +final class SyncEngineTests: XCTestCase { + /// Two rows per scan, so a backlog of four takes two passes — the shape these cases were written + /// against. + private func FakeSource(rows: Int) -> FakeSyncSource { + let source = FakeSyncSource(rows: rows) + source.scanLimit = 2 + return source + } + + private func accepted(boards: Int) -> String { + let counts = SyncTable.allCases + .map { "\"\($0.wire)\":\($0 == .boards ? boards : 0)" } + .joined(separator: ",") + return "{\"accepted\":{\(counts)}}" + } + + private func environment() -> SyncEnvironment { + SyncEnvironment( + ridingSamples: false, + enabled: true, + online: true, + wifiOnly: false, + onWifi: false, + credentialReady: true, + onlineBlocked: false + ) + } + + private func engine( + _ source: SyncSource, + _ responses: [SyncResponse], + sent: Sent = Sent() + ) -> SyncEngine { + var queue = responses + return SyncEngine( + source: source, + transport: { body in + sent.bodies.append(body) + return queue.isEmpty ? .transient(reason: "no response queued") : queue.removeFirst() + }, + environment: environment, + clock: { 1_000 } + ) + } + + final class Sent { + var bodies: [String] = [] + } + + func testAValid200AdvancesOnlyTheRowsItAccountedFor() async { + let source = FakeSource(rows: 2) + let engine = engine(source, [.accepted(body: accepted(boards: 2))]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .sent(rowCount: 2, morePending: false)) + XCTAssertEqual(source.committed, [[.boards: 2]]) + } + + func testAMismatchedAcceptedCountIsAProtocolFailureAndMovesNoCursor() async { + let source = FakeSource(rows: 2) + let engine = engine(source, [.accepted(body: accepted(boards: 1))]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .paused(.protocolFailure)) + XCTAssertTrue(source.committed.isEmpty) + XCTAssertEqual(source.failures.first?.1, "acceptedMismatch") + } + + func testAMalformedSuccessBodyNeverAdvancesACursor() async { + let source = FakeSource(rows: 2) + let engine = engine(source, [.accepted(body: "not json")]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .paused(.protocolFailure)) + XCTAssertTrue(source.committed.isEmpty) + } + + func testARefusedBatchLeavesEveryCursorUntouchedAndDoesNotRetryOnAKick() async { + let source = FakeSource(rows: 2) + let sent = Sent() + let engine = engine(source, [.invalid(status: 409, error: "dependency-conflict")], sent: sent) + + let first = await engine.runOnce() + XCTAssertEqual(first, .paused(.protocolFailure)) + let onKick = await engine.runOnce() + XCTAssertEqual(onKick, .paused(.protocolFailure)) + XCTAssertTrue(source.committed.isEmpty) + // The paused engine never reached the transport a second time. + XCTAssertEqual(sent.bodies.count, 1) + } + + func testAFailurePartWayThroughADrainLeavesCursorsAtTheLastAcceptedBatch() async { + let source = FakeSource(rows: 4) + let engine = engine( + source, + [.accepted(body: accepted(boards: 2)), .transient(reason: "5xx")] + ) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .sent(rowCount: 2, morePending: true)) + let second = await engine.runOnce() + if case .waiting = second {} else { XCTFail("expected a backoff wait") } + XCTAssertEqual(source.committed, [[.boards: 2]]) + } + + func testADeadTokenStopsTheLoopForSignIn() async { + let source = FakeSource(rows: 2) + let engine = engine(source, [.unauthorized]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .paused(.authentication)) + XCTAssertEqual(engine.pauseReason, .authentication) + XCTAssertTrue(source.committed.isEmpty) + } + + func testAResponseFromThePreviousAccountCannotAdvanceACursor() async { + let source = FakeSource(rows: 2) + let engine = SyncEngine( + source: source, + transport: { _ in + // The Account changed while this request was in flight. + source.currentGeneration += 1 + return .accepted(body: self.accepted(boards: 2)) + }, + environment: environment, + clock: { 1_000 } + ) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .idle) + XCTAssertTrue(source.committed.isEmpty) + } + + func testATimeoutAfterTheServerCommittedResendsTheIdenticalBatch() async { + let source = FakeSource(rows: 2) + let sent = Sent() + let engine = engine( + source, + [.transient(reason: "timeout"), .accepted(body: accepted(boards: 2))], + sent: sent + ) + + _ = await engine.runOnce() + engine.resume() + _ = await engine.runOnce() + XCTAssertEqual(sent.bodies.count, 2) + XCTAssertEqual(sent.bodies.first, sent.bodies.last) + } + + func test413PausesOnASingleRowRatherThanSkippingIt() async { + let source = FakeSource(rows: 1) + let engine = engine(source, [.tooLarge]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .paused(.rowTooLarge)) + XCTAssertTrue(source.committed.isEmpty) + XCTAssertEqual(source.remaining, 1) + } + + /// A shrink accepted nothing, so it must not be reported as an upload. + func test413OnAMultiRowBatchNarrowsTheTargetAndRetries() async { + let source = FakeSource(rows: 4) + let engine = engine(source, [.tooLarge]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .retry) + XCTAssertTrue(source.committed.isEmpty) + XCTAssertEqual(source.remaining, 4) + } + + /// Halving forever against a server that keeps refusing would be an unbounded request storm. + func test413AtTheSmallestBatchPausesInsteadOfResendingForever() async { + let source = FakeSource(rows: 4) + let engine = engine(source, Array(repeating: SyncResponse.tooLarge, count: 10)) + + var outcome = await engine.runOnce() + var passes = 0 + while outcome == .retry, passes < 10 { + outcome = await engine.runOnce() + passes += 1 + } + XCTAssertEqual(outcome, .paused(.rowTooLarge)) + XCTAssertTrue(source.committed.isEmpty) + } + + /// The server took the rows; the checkpoint did not land. Resending is safe, claiming success is not. + func testAFailedCursorCommitBacksOffInsteadOfReportingAnUpload() async { + let source = FakeSource(rows: 2) + source.commitFailure = SyncStoreError.databaseUnavailable + let engine = engine(source, [.accepted(body: accepted(boards: 2))]) + + let outcome = await engine.runOnce() + if case .waiting = outcome {} else { XCTFail("expected a backoff wait") } + XCTAssertTrue(source.committed.isEmpty) + XCTAssertEqual(source.remaining, 2) + } + + func test429WaitsForTheServersOwnDelay() async { + let source = FakeSource(rows: 2) + let engine = engine(source, [.rateLimited(retryAfterMs: 90_000)]) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .waiting(untilMs: 91_000)) + XCTAssertNil(engine.pauseReason) + } + + func testARowThatCannotBeEncodedPausesWithTheRowRetained() async { + let source = FakeSource(rows: 2) + source.encodeFailure = SyncProtocolError(table: .boards, field: "id", problem: "must not be empty") + let engine = engine(source, []) + + let pass = await engine.runOnce() + XCTAssertEqual(pass, .paused(.protocolFailure)) + XCTAssertEqual(source.failures.first?.1, "boards.id") + XCTAssertEqual(source.remaining, 2) + } +} diff --git a/modules/vescape-core/ios/sync/SyncJson.swift b/modules/vescape-core/ios/sync/SyncJson.swift new file mode 100644 index 000000000..ded0be18b --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncJson.swift @@ -0,0 +1,163 @@ +import Foundation + +/// A row the server could never store. Permanent for this phone: retrying the same bytes cannot make +/// it succeed, so the engine pauses with the row retained rather than skipping it. +/// +/// It names the table and the field only — never the value, which may be a coordinate, a Rider's +/// text or a token. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncJson.kt `SyncProtocolException` +struct SyncProtocolError: Error, Equatable { + let table: SyncTable + let field: String + let problem: String +} + +/// A compact JSON object writer that validates as it writes. +/// +/// Deliberately not `JSONSerialization`: this has to produce the exact bytes measured against the +/// wire byte cap, in a stable field order. The bounds it enforces are the server's own +/// (`vescape-server` `src/sync/protocol.ts`), applied before transport so a wedged batch is +/// impossible rather than merely unlikely. +/// +/// Nullable columns are written as explicit nulls: "cleared" and "not mentioned" are different +/// intents, and a missing key cannot express the first. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncJson.kt `SyncRowWriter` +/// @parity /modules/vescape-server/src/sync/protocol.ts +final class SyncRowWriter { + private let table: SyncTable + private var out = "{" + + init(_ table: SyncTable) { + self.table = table + } + + func build() -> String { out + "}" } + + /// An identifier the phone chose: a Board id, a settings key, an event name. Never empty. + @discardableResult + func keyText(_ field: String, _ value: String) throws -> SyncRowWriter { + if value.isEmpty { throw fail(field, "must not be empty") } + return try boundedText(field, value) + } + + @discardableResult + func nullableKeyText(_ field: String, _ value: String?) throws -> SyncRowWriter { + guard let value else { return raw(field, "null") } + return try keyText(field, value) + } + + /// A key column the phone derives rather than names, so it may legitimately be empty — a sanitizer + /// writes `""` as the device id of a sample captured with no Board connected. + @discardableResult + func derivedKeyText(_ field: String, _ value: String?) throws -> SyncRowWriter { + guard let value else { return raw(field, "null") } + return try boundedText(field, value) + } + + /// Text the server stores opaquely and hands back unchanged. Uncapped, like the server's. + @discardableResult + func text(_ field: String, _ value: String?) -> SyncRowWriter { + guard let value else { return raw(field, "null") } + return raw(field, quote(value)) + } + + @discardableResult + func bool(_ field: String, _ value: Bool) -> SyncRowWriter { + raw(field, value ? "true" : "false") + } + + /// Epoch ms, or a duration in ms: non-negative and inside the JSON-safe integer range. + @discardableResult + func timestamp(_ field: String, _ value: Int64?) throws -> SyncRowWriter { + try bounded(field, value, 0, syncSafeIntMax) + } + + @discardableResult + func int32(_ field: String, _ value: Int64?) throws -> SyncRowWriter { + try bounded(field, value, syncInt32Min, syncInt32Max) + } + + @discardableResult + func count(_ field: String, _ value: Int64?) throws -> SyncRowWriter { + try bounded(field, value, 0, syncInt32Max) + } + + /// A 64-bit column that is not a timestamp — an odometer reading. + @discardableResult + func int64(_ field: String, _ value: Int64?) throws -> SyncRowWriter { + try bounded(field, value, -syncSafeIntMax, syncSafeIntMax) + } + + /// A real number. Neither infinity nor NaN is expressible in JSON. + @discardableResult + func number(_ field: String, _ value: Double?) throws -> SyncRowWriter { + guard let value else { return raw(field, "null") } + if !value.isFinite { throw fail(field, "must be finite") } + let whole = Int64(exactly: value.rounded(.towardZero)) ?? 0 + return raw(field, value == Double(whole) ? String(whole) : String(value)) + } + + /// A measurement the firmware reported, where non-finite means the reading is unusable rather + /// than that the row is malformed. + /// + /// ``number(_:_:)`` refuses infinity and NaN, which is right for a value a Rider authored — one + /// there is a bug worth stopping for. A decoded Board sample is the opposite case: the app did + /// not choose the value, it received it, and one bad float would pause every table's backup on a + /// permanent protocol error that no retry can clear. These columns are all nullable precisely + /// because a field the firmware did not send is absent, so an unusable one is absent too. + /// + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncJson.kt `reading` + @discardableResult + func reading(_ field: String, _ value: Double?) throws -> SyncRowWriter { + try number(field, value.flatMap { $0.isFinite ? $0 : nil }) + } + + private func bounded(_ field: String, _ value: Int64?, _ min: Int64, _ max: Int64) throws -> SyncRowWriter { + guard let value else { return raw(field, "null") } + if value < min || value > max { throw fail(field, "is out of bounds") } + return raw(field, String(value)) + } + + private func boundedText(_ field: String, _ value: String) throws -> SyncRowWriter { + // UTF-16 code units, matching the server's compiled `value.length <= 128` and Kotlin's + // `String.length`. Swift's `count` is grapheme clusters, which would let a key through here that + // the server refuses — and a refused batch is a permanent pause. + if value.utf16.count > maxSyncKeyLength { + throw fail(field, "exceeds \(maxSyncKeyLength) characters") + } + return raw(field, quote(value)) + } + + @discardableResult + private func raw(_ field: String, _ encoded: String) -> SyncRowWriter { + if out.count > 1 { out += "," } + out += quote(field) + ":" + encoded + return self + } + + private func fail(_ field: String, _ problem: String) -> SyncProtocolError { + SyncProtocolError(table: table, field: field, problem: problem) + } + + private func quote(_ value: String) -> String { + var quoted = "\"" + for character in value.unicodeScalars { + switch character { + case "\"": quoted += "\\\"" + case "\\": quoted += "\\\\" + case "\n": quoted += "\\n" + case "\r": quoted += "\\r" + case "\t": quoted += "\\t" + default: + if character.value < 0x20 { + quoted += String(format: "\\u%04x", character.value) + } else { + quoted.unicodeScalars.append(character) + } + } + } + return quoted + "\"" + } +} diff --git a/modules/vescape-core/ios/sync/SyncNotifier.swift b/modules/vescape-core/ios/sync/SyncNotifier.swift new file mode 100644 index 000000000..7834e65b1 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncNotifier.swift @@ -0,0 +1,45 @@ +import Foundation +import UserNotifications + +/// The one notification backup raises: it has stopped, and only the Rider can restart it. +/// +/// Deliberately narrow — ordinary retries, offline stretches and a metered connection say nothing. +/// A `SyncPauseReason` does not resolve on its own, and a backup that has silently stopped for weeks +/// is the failure this feature can least afford, so each reason gets one actionable notification and +/// is cleared again the moment the pause lifts. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncNotifier.kt +final class SyncNotifier { + static let shared = SyncNotifier() + + private static let identifier = "vescape.backupPaused" + + private init() {} + + /// Show the notification for `reason`, replacing any previous one, or clear it when nil. + func update(_ reason: SyncPauseReason?) { + let center = UNUserNotificationCenter.current() + guard let reason else { + center.removePendingNotificationRequests(withIdentifiers: [Self.identifier]) + center.removeDeliveredNotifications(withIdentifiers: [Self.identifier]) + return + } + let content = UNMutableNotificationContent() + content.title = "Backup paused" + content.body = Self.text(reason) + content.sound = .default + center.add( + UNNotificationRequest(identifier: Self.identifier, content: content, trigger: nil) + ) + } + + /// What the Rider has to do, in the same three shapes the account widget names. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncNotifier.kt `text` + static func text(_ reason: SyncPauseReason) -> String { + switch reason { + case .authentication: return "Sign in again to keep backing up your rides." + case .protocolFailure: return "Update Vescape to keep backing up your rides." + case .rowTooLarge: return "Backup hit an error. Check the event log in settings." + } + } +} diff --git a/modules/vescape-core/ios/sync/SyncPolicy.swift b/modules/vescape-core/ios/sync/SyncPolicy.swift new file mode 100644 index 000000000..748a5ed65 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncPolicy.swift @@ -0,0 +1,119 @@ +import Foundation + +/// How the uploader ran out of road. A paused engine is not woken by ordinary timer kicks. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt `SyncPauseReason` +/// @parity /modules/vescape-core/src/index.ts `SyncPauseReason` +enum SyncPauseReason: String { + /// No Device Token, or the server rejected the one we hold. Sign-in is the only way out. + case authentication + + /// The server refused this batch on its contents, or answered `2xx` with something unreadable. + case protocolFailure = "protocol" + + /// A single row cannot fit inside the wire byte cap. Retained, never skipped. + case rowTooLarge + + var slug: String { rawValue } +} + +/// The backup state the Rider is shown. Derived from the same `SyncState` the loop decides on, so +/// the status line can never disagree with what the uploader is actually doing. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt `SyncActivity` +/// @parity /modules/vescape-core/src/index.ts `SyncActivity` +enum SyncActivity: String { + /// The master switch is off. Nothing is scanned, sent, retried or reported. + case disabled + /// No credential: backup has never been turned on, or the Rider signed out. + case signedOut + case upToDate + case syncing + case waitingForWifi + case offline + /// Stopped on a permanent failure; `SyncStatus.pause` names which one. + case paused + + var slug: String { rawValue } +} + +/// What the loop should do next. +enum SyncDecision: Equatable { + case sendNow + /// Nothing to do until this moment; the loop re-decides then or when a kick lands. + case wait(atMs: Int64) + /// Stopped until the named condition changes. Timer and connectivity kicks do not bypass it. + case paused(SyncPauseReason) +} + +/// Everything the decision depends on, read once by the caller so the decision itself stays pure. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt `SyncState` +struct SyncState { + let nowMs: Int64 + /// Rows waiting across every table. Zero means idle, not finished. + let pendingRows: Int + /// A Board Session is producing samples — Idle Pause halts production without ending the session. + let ridingSamples: Bool + /// The Rider's master switch. Off means the uploader does nothing at all. + let enabled: Bool + let online: Bool + /// Metered-connection setting; the uploader waits for Wi-Fi rather than failing. + let wifiOnly: Bool + let onWifi: Bool + let credentialReady: Bool + /// The App Status gate closed, like every other Online Capability. + let onlineBlocked: Bool + /// Set by a permanent failure; cleared only by sign-in or an Account reset. + let pause: SyncPauseReason? + /// Backoff or `Retry-After` deadline; before it, nothing is sent. + let retryAtMs: Int64 +} + +/// The one place that turns state into "send, wait, or stopped". +/// +/// Pure: no database, no clock, no network. The clock is `SyncState.nowMs` and the caller owns it. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt `SyncPolicy` +enum SyncPolicy { + /// Cadence while a ride is producing samples: a crash loses at most this much. + static let rideIntervalMs: Int64 = 30_000 + + /// Cadence when nothing is pending. Cheap, because it is a no-op. + static let idleIntervalMs: Int64 = 5 * 60_000 + + static let backoffStartMs: Int64 = 30_000 + static let backoffMaxMs: Int64 = 15 * 60_000 + + static func decide(_ state: SyncState) -> SyncDecision { + // The master switch is checked before everything, including a pause: switched off is not a + // broken uploader waiting to be resumed, it is one that is not running. + if !state.enabled { return .wait(atMs: state.nowMs + idleIntervalMs) } + if let pause = state.pause { return .paused(pause) } + if !state.credentialReady { return .paused(.authentication) } + + let interval = state.ridingSamples ? rideIntervalMs : idleIntervalMs + if state.pendingRows <= 0 { return .wait(atMs: state.nowMs + interval) } + // Offline, metered, or gated: a pause in the loop, never a failure that moves backoff. + if !state.online || state.onlineBlocked { return .wait(atMs: state.nowMs + interval) } + if state.wifiOnly && !state.onWifi { return .wait(atMs: state.nowMs + interval) } + if state.retryAtMs > state.nowMs { return .wait(atMs: state.retryAtMs) } + return .sendNow + } + + /// The same state, as the one line the Rider reads. + /// + /// Signed out wins over the pause it produces: a phone with no credential is not a broken backup, + /// it is one that was never turned on. Everything below the pause is ordinary waiting. + static func describe(_ state: SyncState) -> SyncActivity { + if !state.enabled { return .disabled } + if !state.credentialReady { return .signedOut } + if state.pause != nil { return .paused } + if state.pendingRows <= 0 { return .upToDate } + if !state.online || state.onlineBlocked { return .offline } + if state.wifiOnly && !state.onWifi { return .waitingForWifi } + return .syncing + } + + /// Next backoff step: doubling from `backoffStartMs`, capped, and reset to 0 on success. + static func nextBackoffMs(_ previousMs: Int64) -> Int64 { + previousMs <= 0 ? backoffStartMs : min(previousMs * 2, backoffMaxMs) + } +} diff --git a/modules/vescape-core/ios/sync/SyncPolicyTests.swift b/modules/vescape-core/ios/sync/SyncPolicyTests.swift new file mode 100644 index 000000000..99a706820 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncPolicyTests.swift @@ -0,0 +1,114 @@ +import XCTest +@testable import VescapeCore + +/// The send/wait/paused decision, with no database, clock or network behind it. +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncPolicyTest.kt +final class SyncPolicyTests: XCTestCase { + private func state( + pendingRows: Int = 1, + ridingSamples: Bool = false, + enabled: Bool = true, + online: Bool = true, + wifiOnly: Bool = false, + onWifi: Bool = false, + credentialReady: Bool = true, + onlineBlocked: Bool = false, + pause: SyncPauseReason? = nil, + retryAtMs: Int64 = 0 + ) -> SyncState { + SyncState( + nowMs: 1_000, + pendingRows: pendingRows, + ridingSamples: ridingSamples, + enabled: enabled, + online: online, + wifiOnly: wifiOnly, + onWifi: onWifi, + credentialReady: credentialReady, + onlineBlocked: onlineBlocked, + pause: pause, + retryAtMs: retryAtMs + ) + } + + func testPendingRowsOnALiveConnectionSendNow() { + XCTAssertEqual(SyncPolicy.decide(state()), .sendNow) + } + + func testCadenceFollowsSampleProductionNotSessionPresence() { + XCTAssertEqual( + SyncPolicy.decide(state(pendingRows: 0, ridingSamples: true)), + .wait(atMs: 1_000 + SyncPolicy.rideIntervalMs) + ) + XCTAssertEqual( + SyncPolicy.decide(state(pendingRows: 0)), + .wait(atMs: 1_000 + SyncPolicy.idleIntervalMs) + ) + } + + /// Offline, metered and gated are pauses in the loop, never failures that move backoff. + func testOfflineMeteredAndClosedGateAllWait() { + let idle = SyncDecision.wait(atMs: 1_000 + SyncPolicy.idleIntervalMs) + XCTAssertEqual(SyncPolicy.decide(state(online: false)), idle) + XCTAssertEqual(SyncPolicy.decide(state(wifiOnly: true, onWifi: false)), idle) + XCTAssertEqual(SyncPolicy.decide(state(onlineBlocked: true)), idle) + XCTAssertEqual(SyncPolicy.decide(state(wifiOnly: true, onWifi: true)), .sendNow) + } + + func testBackoffDeadlineHoldsTheLoopUntilItPasses() { + XCTAssertEqual(SyncPolicy.decide(state(retryAtMs: 5_000)), .wait(atMs: 5_000)) + XCTAssertEqual(SyncPolicy.decide(state(retryAtMs: 999)), .sendNow) + } + + func testAPauseIsNotBypassedByAnOrdinaryKick() { + XCTAssertEqual(SyncPolicy.decide(state(pause: .protocolFailure)), .paused(.protocolFailure)) + XCTAssertEqual(SyncPolicy.decide(state(credentialReady: false)), .paused(.authentication)) + } + + func testTheMasterSwitchStopsTheUploaderOutrightAndOutranksEveryOtherState() { + XCTAssertEqual( + SyncPolicy.decide(state(enabled: false)), + .wait(atMs: 1_000 + SyncPolicy.idleIntervalMs) + ) + // Not a pause: switched off is not a broken uploader waiting to be resumed. + XCTAssertEqual( + SyncPolicy.decide(state(enabled: false, pause: .protocolFailure)), + .wait(atMs: 1_000 + SyncPolicy.idleIntervalMs) + ) + XCTAssertEqual(SyncPolicy.describe(state(enabled: false)), .disabled) + XCTAssertEqual(SyncPolicy.describe(state(enabled: false, credentialReady: false)), .disabled) + XCTAssertEqual(SyncPolicy.describe(state(enabled: false, pause: .authentication)), .disabled) + } + + func testAPhoneWithNoCredentialReadsAsSignedOutNotAsABrokenBackup() { + XCTAssertEqual(SyncPolicy.describe(state(credentialReady: false)), .signedOut) + XCTAssertEqual( + SyncPolicy.describe(state(credentialReady: false, pause: .authentication)), + .signedOut + ) + } + + func testEveryWaitingReasonIsNamedSeparately() { + XCTAssertEqual(SyncPolicy.describe(state(pendingRows: 0)), .upToDate) + XCTAssertEqual(SyncPolicy.describe(state()), .syncing) + XCTAssertEqual(SyncPolicy.describe(state(online: false)), .offline) + XCTAssertEqual(SyncPolicy.describe(state(onlineBlocked: true)), .offline) + XCTAssertEqual(SyncPolicy.describe(state(wifiOnly: true, onWifi: false)), .waitingForWifi) + XCTAssertEqual(SyncPolicy.describe(state(wifiOnly: true, onWifi: true)), .syncing) + } + + func testAPauseOutranksEverythingExceptBeingSignedOut() { + XCTAssertEqual(SyncPolicy.describe(state(pendingRows: 0, pause: .protocolFailure)), .paused) + XCTAssertEqual(SyncPolicy.describe(state(online: false, pause: .rowTooLarge)), .paused) + } + + func testABatchWaitingOnBackoffStillReadsAsSyncing() { + XCTAssertEqual(SyncPolicy.describe(state(retryAtMs: 60_000)), .syncing) + } + + func testBackoffDoublesFromTheFirstStepAndStopsAtTheCap() { + XCTAssertEqual(SyncPolicy.nextBackoffMs(0), SyncPolicy.backoffStartMs) + XCTAssertEqual(SyncPolicy.nextBackoffMs(30_000), 60_000) + XCTAssertEqual(SyncPolicy.nextBackoffMs(SyncPolicy.backoffMaxMs), SyncPolicy.backoffMaxMs) + } +} diff --git a/modules/vescape-core/ios/sync/SyncRetentionTests.swift b/modules/vescape-core/ios/sync/SyncRetentionTests.swift new file mode 100644 index 000000000..b6ba788e1 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncRetentionTests.swift @@ -0,0 +1,138 @@ +import XCTest +import GRDB +@testable import VescapeCore + +/// Cursor-gated retention against a real database: a bound database must not prune a row the +/// uploader has not delivered, and an unbound one must keep the age-only behaviour it shipped with. +/// +/// The Android peer asserts the same predicates against the DAO source, because Room keeps its SQL +/// out of reach of a JVM unit test. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncCursorContractTest.kt +final class SyncRetentionTests: XCTestCase { + private var queue: DatabaseQueue! + private let old: Int64 = 1_000 + private let cutoff: Int64 = 10_000 + + override func setUpWithError() throws { + queue = try DatabaseQueue() + try TelemetryDatabase.migrator.migrate(queue) + } + + override func tearDownWithError() throws { + queue = nil + } + + private func seedFrame(id: Int64, capturedAtMs: Int64) throws { + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO telemetry_frames + (id, captured_at_ms, elapsed_realtime_ms, board_id, flags, changed_mask_1, changed_mask_2) + VALUES (?, ?, 0, 'board-1', 0, 0, 0) + """, + arguments: [id, capturedAtMs] + ) + } + } + + private func seedBucket(startMs: Int64, syncSeq: Int64) throws { + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO telemetry_minute_buckets + (bucket_start_ms, board_id, sample_count, first_sample_at_ms, last_sample_at_ms, + sum_abs_speed_centi_kmh, max_abs_speed_centi_kmh, max_motor_current_abs_ma, + max_battery_current_abs_ma, battery_used_wh_milli, battery_regen_wh_milli, + max_duty_abs_permille, gps_point_count, precise_gps_point_count, + gps_distance_cm, updated_at, sync_seq) + VALUES (?, 'board-1', 1, ?, ?, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ?, ?) + """, + arguments: [startMs, startMs, startMs, startMs, syncSeq] + ) + } + } + + private func frameIds() throws -> [Int64] { + try queue.read { db in try Int64.fetchAll(db, sql: "SELECT id FROM telemetry_frames ORDER BY id") } + } + + private func bucketStarts() throws -> [Int64] { + try queue.read { db in + try Int64.fetchAll(db, sql: "SELECT bucket_start_ms FROM telemetry_minute_buckets ORDER BY bucket_start_ms") + } + } + + private func bind(_ accountId: String = "account-1") throws { + try queue.write { db in + try db.execute( + sql: "INSERT OR REPLACE INTO sync_binding (id, account_id, bound_at) VALUES (0, ?, 0)", + arguments: [accountId] + ) + } + } + + private func sweep() throws { + _ = try queue.write { db in try deleteBeforeGated(db, beforeMs: cutoff) } + } + + func testANeverBoundDatabaseKeepsTheAgeOnlyCleanup() throws { + try seedFrame(id: 1, capturedAtMs: old) + try seedFrame(id: 2, capturedAtMs: cutoff + 1) + + try sweep() + XCTAssertEqual(try frameIds(), [2]) + } + + func testABoundDatabaseRetainsEveryRowItsCursorHasNotPassed() throws { + try bind() + try seedFrame(id: 1, capturedAtMs: old) + try seedFrame(id: 2, capturedAtMs: old) + + try sweep() + XCTAssertEqual(try frameIds(), [1, 2], "a missing cursor protects every row in the table") + + try queue.write { db in try commitSyncCursor(db, syncCursorFrames, 1) } + try sweep() + XCTAssertEqual(try frameIds(), [2], "only rows at or below the accepted cursor may be pruned") + } + + /// A bucket rewritten after its earlier version uploaded gets a fresh `sync_seq`, so it has to + /// survive until that new position is accepted — a row id could not express this. + func testAnOldBucketRewrittenAfterUploadSurvivesUntilItsNewSeqIsAccepted() throws { + try bind() + try seedBucket(startMs: old, syncSeq: 1) + try queue.write { db in try commitSyncCursor(db, syncCursorMinuteBuckets, 1) } + + // The minute is re-merged, which renumbers the row above the accepted cursor. + try queue.write { db in + try db.execute(sql: "UPDATE telemetry_minute_buckets SET sync_seq = 7 WHERE bucket_start_ms = ?", arguments: [old]) + } + try sweep() + XCTAssertEqual(try bucketStarts(), [old]) + + try queue.write { db in try commitSyncCursor(db, syncCursorMinuteBuckets, 7) } + try sweep() + XCTAssertTrue(try bucketStarts().isEmpty) + } + + /// Signing out does not clear the binding, so data recorded afterwards keeps its protection. + func testSignOutKeepsRetentionProtectionForTheBoundAccount() throws { + try bind() + try seedFrame(id: 1, capturedAtMs: old) + + // Nothing about a sign-out touches `sync_binding`. + try sweep() + XCTAssertEqual(try frameIds(), [1]) + } + + func testABindingIsClaimedOnceAndRefusesADifferentAccount() throws { + try queue.write { db in + XCTAssertNil(try String.fetchOne(db, sql: "SELECT account_id FROM sync_binding WHERE id = 0")) + } + try bind("account-1") + try queue.read { db in + XCTAssertEqual(try String.fetchOne(db, sql: "SELECT account_id FROM sync_binding WHERE id = 0"), "account-1") + } + } +} diff --git a/modules/vescape-core/ios/sync/SyncStore.swift b/modules/vescape-core/ios/sync/SyncStore.swift new file mode 100644 index 000000000..7f837944b --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncStore.swift @@ -0,0 +1,262 @@ +import Foundation +import GRDB + +/// Which Vescape Account this local database belongs to. One row, claimed by the first Account to +/// sign in and never rewritten in place: a different Account replaces the whole database, because +/// resetting the cursors over these rows would upload the previous Account's Boards, Ride History, +/// locations and settings to the new one. +/// +/// Signing out does not clear the binding, so data recorded while signed out keeps its retention +/// protection for the same Account. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `SyncBindingEntity` +internal func createSyncBindingTable(_ db: Database) throws { + try db.execute( + sql: """ + CREATE TABLE IF NOT EXISTS sync_binding ( + id INTEGER PRIMARY KEY NOT NULL, + account_id TEXT NOT NULL, + bound_at INTEGER NOT NULL + ) + """ + ) +} + +/// The database went away underneath the uploader — a swap, or a pool that failed to open. +enum SyncStoreError: Error { + case databaseUnavailable +} + +/// How far a table has been accepted. A table with no committed cursor has delivered nothing. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `cursorOf` +internal func syncCursor(_ db: Database, _ name: String) throws -> Int64 { + try Int64.fetchOne( + db, + sql: "SELECT last_value FROM sync_sequences WHERE name = ?", + arguments: [name] + ) ?? 0 +} + +/// Checkpoint how far a table has been accepted. Run after the response and never alongside the +/// rows: a cursor advanced past rows the server did not take is unrecoverable, whereas a cursor left +/// behind is a re-send the server upserts idempotently. Never moves backwards. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `commitSyncCursor` +internal func commitSyncCursor(_ db: Database, _ name: String, _ throughValue: Int64) throws { + try db.execute( + sql: """ + INSERT OR REPLACE INTO sync_sequences (name, last_value) + VALUES (?, MAX(?, COALESCE((SELECT last_value FROM sync_sequences WHERE name = ?), 0))) + """, + arguments: [name, throughValue, name] + ) +} + +/// The database side of the uploader: the forward scan, the cursor commit and the failure record. +/// +/// Encoding happens here rather than in the engine, so the pure batch builder measures the exact +/// bytes that will be sent. Rows are read in `SyncTable` order and the scan stops once the row limit +/// is reached — a table further down waits for the next batch, which is what keeps parents ahead of +/// children. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncStore.kt `SyncStore` +final class SyncStore: SyncSource { + private let generationProvider: () -> Int64 + private let onPermanentFailure: (SyncPauseReason, String) -> Void + private let database: () -> (any DatabaseWriter)? + + init( + generation: @escaping () -> Int64, + onPermanentFailure: @escaping (SyncPauseReason, String) -> Void, + // Injected so the scan and the cursor commit can be run against a real database in a test. The + // default is the shared pool, which is what production always passes. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncStore.kt `database` + database: @escaping () -> (any DatabaseWriter)? = { TelemetryDatabase.pool } + ) { + self.generationProvider = generation + self.onPermanentFailure = onPermanentFailure + self.database = database + } + + /// Resolved per call: an Account reset replaces the whole database under this object. + private var pool: (any DatabaseWriter)? { database() } + + /// Rows that name no Board are not offered: the server keys these tables on the Board and has + /// nowhere to put a row that belongs to none (ADR-0028). They are unowned local rows, not rows a + /// Rider is waiting to see backed up. + /// + /// An Exclusion Range is filtered for the same reason a bucket is, and it is not optional: the + /// sanitizers write `UNKNOWN_TELEMETRY_BOARD_ID` — the empty string — on a range recorded with no + /// Board connected, and an unattributed range names no Board for the server to hang it off. Its + /// composite foreign key refuses that row, which refuses the whole Sync Batch, and since the row is + /// retained the same batch retries forever: backup wedges permanently on one unowned range. + /// + /// The consequence is deliberate: a later owned row carries the cursor past a skipped one, so + /// cursor-gated retention prunes unowned telemetry on age alone, exactly as it did before the + /// Account binding existed. Holding it forever would be the only alternative, because no future + /// upload can ever accept it. + private func scanPredicate(_ table: SyncTable) -> String { + switch table { + case .telemetryFrames: return " AND board_id IS NOT NULL" + case .telemetryMinuteBuckets, .metricExclusionRanges: return " AND board_id != ''" + default: return "" + } + } + + func pending(rowLimit: Int) throws -> [SyncPendingTable] { + guard let pool else { return [] } + var tables: [SyncPendingTable] = [] + var budget = rowLimit + var encodeError: Error? + + try pool.read { db in + for table in SyncTable.allCases { + if budget <= 0 { break } + let cursor = try syncCursor(db, table.cursorKey) + let rows = try Row.fetchAll( + db, + sql: """ + SELECT * FROM \(table.table) + WHERE \(table.cursorColumn) > ?\(scanPredicate(table)) + ORDER BY \(table.cursorColumn) ASC + LIMIT ? + """, + arguments: [cursor, budget] + ) + if rows.isEmpty { continue } + do { + let encoded = try rows.map { row in + SyncPendingRow( + cursor: row[table.cursorColumn] as Int64? ?? 0, + json: try SyncWire.encode(table, row) + ) + } + tables.append(SyncPendingTable(table: table, rows: encoded)) + budget -= encoded.count + } catch { + encodeError = error + return + } + } + } + + if let encodeError { throw encodeError } + return tables + } + + func pendingCount() -> Int { + guard let pool else { return 0 } + return (try? pool.read { db in + var total = 0 + for table in SyncTable.allCases { + let cursor = try syncCursor(db, table.cursorKey) + total += try Int.fetchOne( + db, + sql: """ + SELECT COUNT(*) FROM \(table.table) + WHERE \(table.cursorColumn) > ?\(scanPredicate(table)) + """, + arguments: [cursor] + ) ?? 0 + } + return total + }) ?? 0 + } + + /// Cursors move only here, only after the server accepted. The accepted Sync Action cursor is also + /// what prunes the log, so pruning can never outrun it. + func commit(_ advances: [SyncTable: Int64]) throws { + guard let pool else { throw SyncStoreError.databaseUnavailable } + try pool.write { db in + for (table, cursor) in advances { + try commitSyncCursor(db, table.cursorKey, cursor) + } + } + guard advances[.deleteActions] != nil else { return } + // Pruning is a follow-up to the checkpoint, not part of it: a failure here leaves accepted + // actions on disk, which re-send as no-ops, so it must not fail the commit itself. + try? pool.write { db in + try pruneUploadedSyncActions(db) + } + } + + func generation() -> Int64 { generationProvider() } + + func recordPermanentFailure(_ reason: SyncPauseReason, detail: String) { + onPermanentFailure(reason, detail) + } + + // Account binding. + + func boundAccountId() -> String? { + guard let pool else { return nil } + return try? pool.read { db in + try String.fetchOne(db, sql: "SELECT account_id FROM sync_binding WHERE id = 0") + } + } + + /// Claim this database for `accountId`, or confirm it already belongs to it. False means it + /// belongs to a different Account: the caller has to replace the database first. + @discardableResult + func bindAccount(_ accountId: String) -> Bool { + guard let pool else { return false } + return (try? pool.write { db -> Bool in + if let bound = try String.fetchOne(db, sql: "SELECT account_id FROM sync_binding WHERE id = 0") { + return bound == accountId + } + try db.execute( + sql: "INSERT OR REPLACE INTO sync_binding (id, account_id, bound_at) VALUES (0, ?, ?)", + arguments: [accountId, telemetryNowMs()] + ) + return true + }) ?? false + } +} + +/// Cursor-gated retention. A retention cutoff is only a candidate cutoff: cleanup must not remove a +/// row the uploader has not delivered. The sweep reads its table cursor and deletes in one +/// transaction, so racing an upload fails safe — before the cursor commit the rows are retained, +/// after it the server has accepted them. A missing cursor is 0, protecting every row. +/// +/// Emits no Sync Actions: a retention sweep is maintenance, and `DeleteTarget` has no case that +/// could name a pruned table. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `deleteBeforeGated` +internal func deleteBeforeGated(_ db: Database, beforeMs: Int64) throws -> Int { + let bound = try String.fetchOne(db, sql: "SELECT account_id FROM sync_binding WHERE id = 0") + + if bound == nil { + // Never bound to an Account: the existing age-only cleanup, unchanged. + try db.execute(sql: "DELETE FROM telemetry_frames WHERE captured_at_ms < ?", arguments: [beforeMs]) + let count = db.changesCount + try db.execute(sql: "DELETE FROM telemetry_minute_buckets WHERE bucket_start_ms < ?", arguments: [beforeMs]) + try db.execute(sql: "DELETE FROM telemetry_markers WHERE occurred_at_ms < ?", arguments: [beforeMs]) + try db.execute(sql: "DELETE FROM metric_exclusion_ranges WHERE end_ms < ?", arguments: [beforeMs]) + try db.execute(sql: "DELETE FROM diagnostic_events WHERE occurred_at_ms < ?", arguments: [beforeMs]) + return count + } + + try db.execute( + sql: "DELETE FROM telemetry_frames WHERE captured_at_ms < ? AND id <= ?", + arguments: [beforeMs, try syncCursor(db, syncCursorFrames)] + ) + let count = db.changesCount + // A bucket is protected by `sync_seq`, not by a row id: one rewritten after an earlier version + // uploaded gets a fresh position and has to survive until that one is accepted. + try db.execute( + sql: "DELETE FROM telemetry_minute_buckets WHERE bucket_start_ms < ? AND sync_seq <= ?", + arguments: [beforeMs, try syncCursor(db, syncCursorMinuteBuckets)] + ) + try db.execute( + sql: "DELETE FROM telemetry_markers WHERE occurred_at_ms < ? AND id <= ?", + arguments: [beforeMs, try syncCursor(db, syncCursorMarkers)] + ) + try db.execute( + sql: "DELETE FROM metric_exclusion_ranges WHERE end_ms < ? AND id <= ?", + arguments: [beforeMs, try syncCursor(db, syncCursorExclusionRanges)] + ) + try db.execute( + sql: "DELETE FROM diagnostic_events WHERE occurred_at_ms < ? AND id <= ?", + arguments: [beforeMs, try syncCursor(db, syncCursorDiagnosticEvents)] + ) + return count +} diff --git a/modules/vescape-core/ios/sync/SyncStoreTests.swift b/modules/vescape-core/ios/sync/SyncStoreTests.swift new file mode 100644 index 000000000..d4558bc87 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncStoreTests.swift @@ -0,0 +1,277 @@ +import GRDB +import XCTest +@testable import VescapeCore + +/// `SyncStore` against a real database: the forward scan, the cursor commit, and a whole backlog +/// drained through the real engine. +/// +/// Everything else in the sync suite runs on a fake source. That makes the engine's decisions +/// testable but leaves the store itself — the SQL that decides which rows go next, and the write +/// that says which are safe to forget — never executed. This is where the two meet, so a scan that +/// re-reads a delivered row, or a commit that skips one, fails here rather than on a Rider's phone. +/// +/// The Android peer asserts the same scan predicates against the DAO source, because Room keeps its +/// SQL out of reach of a JVM unit test. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncCursorContractTest.kt +final class SyncStoreTests: XCTestCase { + private var queue: DatabaseQueue! + private var generation: Int64 = 0 + private var now: Int64 = 1_000 + + override func setUpWithError() throws { + queue = try DatabaseQueue() + try TelemetryDatabase.migrator.migrate(queue) + generation = 0 + now = 1_000 + } + + override func tearDownWithError() throws { + queue = nil + } + + private func store() -> SyncStore { + SyncStore( + generation: { self.generation }, + onPermanentFailure: { _, _ in }, + database: { self.queue } + ) + } + + // Seeding. `sync_seq` is passed explicitly: these tests are about the scan reading it, not about + // the write path that hands positions out. + + private func seedBoard(_ id: String, syncSeq: Int64) throws { + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO boards (id, name, ble_id, transport, created_at, updated_at, sync_seq, deleted_at) + VALUES (?, ?, NULL, NULL, 0, 0, ?, NULL) + """, + arguments: [id, "Board \(id)", syncSeq] + ) + } + } + + private func seedAlert(_ id: String, boardId: String, syncSeq: Int64) throws { + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO alerts + (board_id, id, control_id, threshold, threshold_max, enabled, sound_type, created_at, + source, updated_at, sync_seq) + VALUES (?, ?, 'speed', 1.0, NULL, 1, 'beep', 0, NULL, 0, ?) + """, + arguments: [boardId, id, syncSeq] + ) + } + } + + private func seedFrame(id: Int64, boardId: String?) throws { + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO telemetry_frames + (id, captured_at_ms, elapsed_realtime_ms, board_id, flags, changed_mask_1, changed_mask_2) + VALUES (?, 0, 0, ?, 0, 0, 0) + """, + arguments: [id, boardId] + ) + } + } + + private func cursor(_ table: SyncTable) throws -> Int64 { + try queue.read { db in try syncCursor(db, table.cursorKey) } + } + + private func cursors(_ tables: [SyncPendingTable]) -> [SyncTable: [Int64]] { + Dictionary(uniqueKeysWithValues: tables.map { ($0.table, $0.rows.map(\.cursor)) }) + } + + // The scan boundary. + + /// The single most consequential line of SQL in the feature: `>` and not `>=`. Off by one in + /// either direction is silent — one re-sends the same row forever, the other never sends it. + func testAScanServesOnlyRowsStrictlyPastTheCommittedCursor() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try seedBoard("b", syncSeq: 2) + try seedBoard("c", syncSeq: 3) + + XCTAssertEqual(cursors(try store.pending(rowLimit: 100))[.boards], [1, 2, 3]) + + try store.commit([.boards: 2]) + + XCTAssertEqual(cursors(try store.pending(rowLimit: 100))[.boards], [3]) + XCTAssertEqual(store.pendingCount(), 1) + } + + /// A row written after the checkpoint gets a higher position, so it is picked up on the next pass + /// without anything having to notice it arrived. + func testARowWrittenAfterACheckpointIsPickedUpNext() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try store.commit([.boards: 1]) + XCTAssertEqual(store.pendingCount(), 0) + + try seedBoard("b", syncSeq: 2) + + XCTAssertEqual(cursors(try store.pending(rowLimit: 100))[.boards], [2]) + } + + /// A late response carrying a stale checkpoint must not un-deliver rows the store already + /// forgot — `commitSyncCursor` takes the higher of the two. + func testACommitNeverMovesACursorBackwards() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try seedBoard("b", syncSeq: 2) + + try store.commit([.boards: 2]) + try store.commit([.boards: 1]) + + XCTAssertEqual(try cursor(.boards), 2) + XCTAssertEqual(store.pendingCount(), 0) + } + + /// Parents before children, one shared budget: a batch that cannot fit a Board must not spend the + /// rest of its budget on that Board's Alert Rules. + func testTheRowBudgetIsSharedAcrossTablesInServerOrder() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try seedBoard("b", syncSeq: 2) + try seedAlert("r1", boardId: "a", syncSeq: 1) + try seedAlert("r2", boardId: "a", syncSeq: 2) + + let firstPass = try store.pending(rowLimit: 2) + XCTAssertEqual(firstPass.map(\.table), [.boards], "the budget was spent before alerts") + XCTAssertEqual(cursors(firstPass)[.boards], [1, 2]) + + try store.commit([.boards: 2]) + + let secondPass = try store.pending(rowLimit: 2) + XCTAssertEqual(secondPass.map(\.table), [.alerts]) + XCTAssertEqual(cursors(secondPass)[.alerts], [1, 2]) + } + + /// The deliberate deviation in #284: a frame naming no Board can never be accepted, so it is not + /// offered — and, critically, it does not hold the cursor back for the rows that can be. + func testUnownedTelemetryIsNeitherOfferedNorAllowedToWedgeTheScan() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try seedFrame(id: 1, boardId: nil) + try seedFrame(id: 2, boardId: "a") + + let pending = try store.pending(rowLimit: 100) + XCTAssertEqual(cursors(pending)[.telemetryFrames], [2], "the unowned frame must not be offered") + XCTAssertEqual(store.pendingCount(), 2, "one board and one owned frame") + + try store.commit([.telemetryFrames: 2]) + XCTAssertNil(cursors(try store.pending(rowLimit: 100))[.telemetryFrames]) + } + + private func seedExclusionRange(id: Int64, boardId: String) throws { + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO metric_exclusion_ranges (id, board_id, reason, start_ms, end_ms, sample_count) + VALUES (?, ?, 'idle', 0, 1, 0) + """, + arguments: [id, boardId] + ) + } + } + + /// A range recorded with no Board connected carries `UNKNOWN_TELEMETRY_BOARD_ID`, and the server's + /// composite foreign key refuses it — which refuses the whole batch, and since the row is retained + /// the same batch retries forever. Offering it once would wedge backup permanently. + func testAnUnattributedExclusionRangeIsNeitherOfferedNorCounted() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try seedExclusionRange(id: 1, boardId: UNKNOWN_TELEMETRY_BOARD_ID) + try seedExclusionRange(id: 2, boardId: "a") + + let pending = try store.pending(rowLimit: 100) + XCTAssertEqual(cursors(pending)[.metricExclusionRanges], [2]) + XCTAssertEqual(store.pendingCount(), 2, "one board and one owned range") + } + + /// `pendingCount` drives the status line and the "send again immediately" decision. A count that + /// disagrees with the scan reports a drained backlog while rows are still waiting. + func testThePendingCountAgreesWithWhatTheScanWillActuallyOffer() throws { + let store = store() + try seedBoard("a", syncSeq: 1) + try seedAlert("r1", boardId: "a", syncSeq: 1) + try seedFrame(id: 1, boardId: nil) + + let offered = try store.pending(rowLimit: 1_000).reduce(0) { $0 + $1.rows.count } + XCTAssertEqual(store.pendingCount(), offered) + } + + // The whole loop, over the real store. + + /// The round trip: real database, real scan, real encoding, real cursor writes, real engine. The + /// backlog has to reach zero, the server has to receive every row, and a second drain has to be + /// idle — a cursor left short would keep re-sending, a cursor overshot would drop rows here. + func testAWholeBacklogDrainsThroughTheRealStore() async throws { + let store = store() + for index in 1...12 { try seedBoard("board-\(index)", syncSeq: Int64(index)) } + for index in 1...7 { try seedAlert("rule-\(index)", boardId: "board-1", syncSeq: Int64(index)) } + for index in 1...5 { try seedFrame(id: Int64(index), boardId: "board-1") } + + let total = store.pendingCount() + XCTAssertEqual(total, 24) + + let server = FakeSyncServer() + let engine = SyncEngine( + source: store, + transport: { body in server.send(body) }, + environment: { + SyncEnvironment( + ridingSamples: false, + enabled: true, + online: true, + wifiOnly: false, + onWifi: false, + credentialReady: true, + onlineBlocked: false + ) + }, + clock: { self.now } + ) + + var passes = 0 + while store.pendingCount() > 0, passes < 100 { + switch await engine.runOnce() { + case let .waiting(untilMs): now = max(now, untilMs) + 1 + case .paused: XCTFail("the drain must not pause"); return + default: break + } + passes += 1 + } + + XCTAssertEqual(store.pendingCount(), 0) + XCTAssertEqual(server.writes, total, "every row exactly once") + XCTAssertEqual(try cursor(.boards), 12) + XCTAssertEqual(try cursor(.alerts), 7) + XCTAssertEqual(try cursor(.telemetryFrames), 5) + + // A drained backlog falls back to the idle poll rather than sending an empty batch. + let requests = server.received.count + _ = await engine.runOnce() + XCTAssertEqual(server.received.count, requests, "a drained backlog must send nothing at all") + } + + /// A checkpoint that never landed must leave every row pending. The server holds them, so the + /// re-send is a no-op there — but the store cannot know that, and guessing costs the rows. + func testAFailedCheckpointLeavesEveryRowPending() async throws { + let store = store() + for index in 1...4 { try seedBoard("board-\(index)", syncSeq: Int64(index)) } + + // A database that went away between the response and the checkpoint. + let broken = SyncStore(generation: { 0 }, onPermanentFailure: { _, _ in }, database: { nil }) + XCTAssertThrowsError(try broken.commit([.boards: 4])) + + XCTAssertEqual(try cursor(.boards), 0) + XCTAssertEqual(store.pendingCount(), 4) + } +} diff --git a/modules/vescape-core/ios/sync/SyncTables.swift b/modules/vescape-core/ios/sync/SyncTables.swift new file mode 100644 index 000000000..c82d0c1ef --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncTables.swift @@ -0,0 +1,113 @@ +import Foundation + +/// Every table a Sync Batch can carry, in the order the server writes them: a Board-owned row +/// references its Board, so a batch carrying both has to put the Board first or the foreign key +/// refuses the whole batch. Delete Actions come last, so an action is judged against the Change +/// Timestamp the same batch just wrote. +/// +/// The batch builder walks this order and nothing else — never the size of a table's backlog, which +/// would produce a batch the server cannot apply. +/// +/// `cursorColumn` is what the scan runs on: an `AUTOINCREMENT` key for append-only tables, +/// `sync_seq` for mutable ones. Both are device-local counters that never cross the wire. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt `SyncTable` +enum SyncTable: String, CaseIterable { + case appSettings + case boards + case boardSettings + case boardWarnings + case alerts + case tuneProfiles + case tuneHistoryEntries + case privacyZones + case telemetryMarkers + case metricExclusionRanges + case diagnosticEvents + case telemetryFrames + case telemetryMinuteBuckets + case favorites + // Board-owned, so after `boards`; the Capture and its samples reference the Occurrence, so after + // it in turn. This chain is the one place the ordering rule bites twice in one batch. + case vescFaultOccurrences + case vescFaultCaptures + case vescFaultCaptureSamples + case deleteActions + + var wire: String { rawValue } + + var table: String { + switch self { + case .appSettings: return "app_settings" + case .boards: return "boards" + case .boardSettings: return "board_settings" + case .boardWarnings: return "board_warnings" + case .alerts: return "alerts" + case .tuneProfiles: return "tune_profiles" + case .tuneHistoryEntries: return "tune_history_entries" + case .privacyZones: return "privacy_zones" + case .telemetryMarkers: return "telemetry_markers" + case .metricExclusionRanges: return "metric_exclusion_ranges" + case .diagnosticEvents: return "diagnostic_events" + case .telemetryFrames: return "telemetry_frames" + case .telemetryMinuteBuckets: return "telemetry_minute_buckets" + case .favorites: return "favorites" + case .vescFaultOccurrences: return "vesc_fault_occurrences" + case .vescFaultCaptures: return "vesc_fault_captures" + case .vescFaultCaptureSamples: return "vesc_fault_capture_samples" + case .deleteActions: return "sync_actions" + } + } + + var cursorColumn: String { + switch self { + case .tuneHistoryEntries, .telemetryMarkers, .metricExclusionRanges, .diagnosticEvents, + .telemetryFrames, .vescFaultCaptureSamples, .deleteActions: + return syncRowIdColumn + default: + return syncSeqColumn + } + } + + /// `sync_sequences` key holding how far this table has been accepted. Distinct from the write + /// counters keyed on the bare table name, which hand out `sync_seq` positions. + /// + /// Sync Actions keep the key #282 already shipped, so the log's prune keeps reading the same row + /// the uploader commits. + var cursorKey: String { + self == .deleteActions ? syncActionsUploadedCursor : syncCursorPrefix + table + } +} + +internal let syncSeqColumn = "sync_seq" +internal let syncRowIdColumn = "id" +internal let syncCursorPrefix = "sync_cursor_" + +/// Rows accepted in one Sync Batch, total across every table. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt `MAX_SYNC_BATCH_ROWS` +let maxSyncBatchRows = 1_000 + +/// Actual compact UTF-8 JSON bytes accepted by `POST /api/sync`. Measured on the encoded request, +/// not estimated from object sizes — the server refuses on the byte count it actually receives. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt `MAX_SYNC_BATCH_BYTES` +let maxSyncBatchBytes = 1024 * 1024 + +/// Longest text one column of a server key may hold. Mirrored from the server so a row that cannot +/// be stored is refused here instead of wedging a batch. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncTables.kt `MAX_SYNC_KEY_LENGTH` +let maxSyncKeyLength = 128 + +/// Bounds of the Postgres `integer` columns the app's 32-bit values land in. +internal let syncInt32Min: Int64 = -2_147_483_648 +internal let syncInt32Max: Int64 = 2_147_483_647 + +/// `Number.MAX_SAFE_INTEGER`: past it `JSON.parse` rounds, so neither side could agree on the value. +internal let syncSafeIntMax: Int64 = 9_007_199_254_740_991 + +/// The five retained tables' cursor keys, named so cursor-gated retention reads what the uploader +/// commits. +internal let syncCursorFrames = "sync_cursor_telemetry_frames" +internal let syncCursorMarkers = "sync_cursor_telemetry_markers" +internal let syncCursorMinuteBuckets = "sync_cursor_telemetry_minute_buckets" +internal let syncCursorDiagnosticEvents = "sync_cursor_diagnostic_events" +internal let syncCursorExclusionRanges = "sync_cursor_metric_exclusion_ranges" diff --git a/modules/vescape-core/ios/sync/SyncWire.swift b/modules/vescape-core/ios/sync/SyncWire.swift new file mode 100644 index 000000000..4a3e7d680 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncWire.swift @@ -0,0 +1,374 @@ +import Foundation +import GRDB + +/// Local rows as the server reads them. +/// +/// Every encoder validates before transport, so a batch is refused here — with the row retained and +/// one metadata-only Diagnostic Event — rather than wedging against the server. The field sets +/// mirror `vescape-server` `src/sync/protocol.ts`; a column the server does not declare is not sent, +/// because an unknown field rejects the whole batch. +/// +/// Rows arrive as GRDB rows rather than typed structs: the iOS side stores telemetry in raw SQL, and +/// re-modelling fifteen tables here would add a second schema to keep in step with the first. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncWire.kt +/// @parity /modules/vescape-server/src/sync/protocol.ts +enum SyncWire { + static func encode(_ table: SyncTable, _ row: Row) throws -> String { + switch table { + case .appSettings: return try appSetting(row) + case .boards: return try board(row) + case .boardSettings: return try boardSetting(row) + case .boardWarnings: return try boardWarning(row) + case .alerts: return try alert(row) + case .tuneProfiles: return try tuneProfile(row) + case .tuneHistoryEntries: return try tuneHistoryEntry(row) + case .privacyZones: return try privacyZone(row) + case .telemetryMarkers: return try telemetryMarker(row) + case .metricExclusionRanges: return try metricExclusionRange(row) + case .diagnosticEvents: return try diagnosticEvent(row) + case .telemetryFrames: return try telemetryFrame(row) + case .telemetryMinuteBuckets: return try telemetryMinuteBucket(row) + case .favorites: return try favorite(row) + case .vescFaultOccurrences: return try vescFaultOccurrence(row) + case .vescFaultCaptures: return try vescFaultCapture(row) + case .vescFaultCaptureSamples: return try vescFaultCaptureSample(row) + case .deleteActions: return try deleteAction(row) + } + } + + static func appSetting(_ row: Row) throws -> String { + let writer = SyncRowWriter(.appSettings) + try writer.keyText("key", text(row, "key")) + writer.text("valueJson", row["value_json"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + /// `transport` is the one column only iOS stores on the Board itself — Android keeps it in board + /// settings and sends null there. The server declares the field for exactly this reason, so a + /// restored iPhone keeps the Board Link's selected transport instead of re-probing for it. + /// @platform-diff Android has no `boards.transport` column and sends null. + static func board(_ row: Row) throws -> String { + let writer = SyncRowWriter(.boards) + try writer.keyText("id", text(row, "id")) + writer.text("name", row["name"]) + writer.text("bleId", row["ble_id"]) + writer.text("transport", row["transport"]) + try writer.timestamp("createdAt", row["created_at"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + static func boardSetting(_ row: Row) throws -> String { + let writer = SyncRowWriter(.boardSettings) + try writer.keyText("boardId", text(row, "board_id")) + try writer.keyText("key", text(row, "key")) + writer.text("valueJson", row["value_json"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + static func boardWarning(_ row: Row) throws -> String { + let writer = SyncRowWriter(.boardWarnings) + try writer.keyText("boardId", text(row, "board_id")) + try writer.keyText("kind", text(row, "kind")) + writer.text("severity", row["severity"]) + try writer.timestamp("firstDetectedAt", row["first_detected_at"]) + try writer.timestamp("lastDetectedAt", row["last_detected_at"]) + writer.text("payloadJson", row["payload_json"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + /// An Alert Rule is not always "fire above this number": `thresholdKind` says how to read them, and + /// a config-relative rule follows the Refloat field in `configFieldId` with the offsets applied to + /// whatever that field holds. Restoring the numbers without the kind produces a rule that looks + /// configured and fires at the wrong point, which is worse than losing it. + static func alert(_ row: Row) throws -> String { + let writer = SyncRowWriter(.alerts) + try writer.keyText("boardId", text(row, "board_id")) + try writer.keyText("id", text(row, "id")) + try writer.keyText("controlId", text(row, "control_id")) + try writer.number("threshold", row["threshold"]) + try writer.number("thresholdMax", row["threshold_max"]) + try writer.keyText("thresholdKind", text(row, "threshold_kind")) + try writer.nullableKeyText("configFieldId", row["config_field_id"]) + try writer.number("thresholdOffset", row["threshold_offset"]) + try writer.number("thresholdMaxOffset", row["threshold_max_offset"]) + writer.bool("enabled", (row["enabled"] as Int64? ?? 0) != 0) + writer.text("soundType", row["sound_type"]) + try writer.timestamp("repeatEverySeconds", row["repeat_every_seconds"]) + try writer.count("beepCount", row["beep_count"]) + writer.text("source", row["source"]) + try writer.timestamp("createdAt", row["created_at"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + static func tuneProfile(_ row: Row) throws -> String { + let writer = SyncRowWriter(.tuneProfiles) + try writer.keyText("id", text(row, "id")) + try writer.keyText("boardId", text(row, "board_id")) + // May legitimately be empty: the app defaults an unknown Refloat package version to `''`. + try writer.derivedKeyText("refloatBaseVersion", row["refloat_base_version"]) + writer.text("name", row["name"]) + writer.text("icon", row["icon"]) + writer.text("color", row["color"]) + writer.text("fieldsJson", row["fields_json"]) + try writer.timestamp("createdAt", row["created_at"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + /// Carries no id: the local one restarts on a fresh install, so identity is `(profileId, createdAt)`. + static func tuneHistoryEntry(_ row: Row) throws -> String { + let writer = SyncRowWriter(.tuneHistoryEntries) + try writer.keyText("profileId", text(row, "profile_id")) + writer.text("fieldsJson", row["fields_json"]) + try writer.timestamp("createdAt", row["created_at"]) + return writer.build() + } + + static func privacyZone(_ row: Row) throws -> String { + let writer = SyncRowWriter(.privacyZones) + try writer.keyText("id", text(row, "id")) + writer.text("preset", row["preset"]) + writer.text("name", row["name"]) + writer.bool("enabled", (row["enabled"] as Int64? ?? 0) != 0) + try writer.int32("centerLatitudeE7", row["center_latitude_e7"]) + try writer.int32("centerLongitudeE7", row["center_longitude_e7"]) + try writer.int32("radiusMeters", row["radius_meters"]) + try writer.timestamp("createdAt", row["created_at"]) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + static func telemetryMarker(_ row: Row) throws -> String { + let writer = SyncRowWriter(.telemetryMarkers) + try writer.timestamp("occurredAtMs", row["occurred_at_ms"]) + try writer.timestamp("elapsedRealtimeMs", row["elapsed_realtime_ms"]) + try writer.keyText("type", text(row, "type")) + try writer.derivedKeyText("boardId", row["board_id"]) + writer.text("message", row["message"]) + try writer.timestamp("gapMs", row["gap_ms"]) + return writer.build() + } + + static func metricExclusionRange(_ row: Row) throws -> String { + let writer = SyncRowWriter(.metricExclusionRanges) + try writer.derivedKeyText("boardId", row["board_id"]) + writer.text("reason", row["reason"]) + try writer.timestamp("startMs", row["start_ms"]) + try writer.timestamp("endMs", row["end_ms"]) + try writer.count("sampleCount", row["sample_count"]) + return writer.build() + } + + static func diagnosticEvent(_ row: Row) throws -> String { + let writer = SyncRowWriter(.diagnosticEvents) + try writer.timestamp("occurredAtMs", row["occurred_at_ms"]) + try writer.timestamp("elapsedRealtimeMs", row["elapsed_realtime_ms"]) + try writer.keyText("eventName", text(row, "event_name")) + try writer.derivedKeyText("operation", row["operation"]) + try writer.derivedKeyText("phase", row["phase"]) + try writer.derivedKeyText("boardId", row["board_id"]) + writer.text("message", row["message"]) + writer.text("propertiesJson", row["properties_json"]) + return writer.build() + } + + /// A Telemetry Sample as recorded: still delta-encoded, carrying the Changed Masks. The local row + /// id and the per-row device columns never cross the wire — the Board reference replaces them + /// (ADR-0028) and a restored phone's full re-upload has to be an idempotent no-op. + /// + /// A frame that names no Board cannot be encoded; the scan never offers one. + static func telemetryFrame(_ row: Row) throws -> String { + let writer = SyncRowWriter(.telemetryFrames) + guard let boardId: String = row["board_id"] else { + throw SyncProtocolError(table: .telemetryFrames, field: "boardId", problem: "must name a Board") + } + try writer.keyText("boardId", boardId) + try writer.timestamp("capturedAtMs", row["captured_at_ms"]) + try writer.timestamp("elapsedRealtimeMs", row["elapsed_realtime_ms"]) + try writer.int32("canId", row["can_id"]) + try writer.count("flags", row["flags"]) + try writer.count("changedMask1", row["changed_mask_1"]) + try writer.count("changedMask2", row["changed_mask_2"]) + try writer.int32("speedCentiKmh", row["speed_centi_kmh"]) + try writer.int32("batteryVoltageMv", row["battery_voltage_mv"]) + try writer.int32("motorCurrentMa", row["motor_current_ma"]) + try writer.int32("batteryCurrentMa", row["battery_current_ma"]) + try writer.int32("dutyPermille", row["duty_permille"]) + try writer.int32("pitchCentiDeg", row["pitch_centi_deg"]) + try writer.int32("rollCentiDeg", row["roll_centi_deg"]) + try writer.int32("balancePitchCentiDeg", row["balance_pitch_centi_deg"]) + try writer.int32("balanceCurrentMa", row["balance_current_ma"]) + try writer.int32("erpm", row["erpm"]) + try writer.int32("state", row["state"]) + try writer.int32("switchState", row["switch_state"]) + try writer.int32("adc1Milli", row["adc1_milli"]) + try writer.int32("adc2Milli", row["adc2_milli"]) + try writer.int64("odometerCm", row["odometer_cm"]) + try writer.int32("tempMosfetDeciC", row["temp_mosfet_deci_c"]) + try writer.int32("tempMotorDeciC", row["temp_motor_deci_c"]) + try writer.int32("latitudeE7", row["latitude_e7"]) + try writer.int32("longitudeE7", row["longitude_e7"]) + try writer.int32("gpsSpeedCentiMps", row["gps_speed_centi_mps"]) + try writer.int32("bearingCentiDeg", row["bearing_centi_deg"]) + try writer.int32("accuracyCm", row["accuracy_cm"]) + try writer.int32("altitudeCm", row["altitude_cm"]) + try writer.timestamp("locationTimestampMs", row["location_timestamp_ms"]) + return writer.build() + } + + static func telemetryMinuteBucket(_ row: Row) throws -> String { + let writer = SyncRowWriter(.telemetryMinuteBuckets) + try writer.keyText("boardId", text(row, "board_id")) + try writer.timestamp("bucketStartMs", row["bucket_start_ms"]) + try writer.timestamp("updatedAt", row["updated_at"]) + try writer.count("sampleCount", row["sample_count"]) + try writer.timestamp("firstSampleAtMs", row["first_sample_at_ms"]) + try writer.timestamp("lastSampleAtMs", row["last_sample_at_ms"]) + try writer.int64("sumAbsSpeedCentiKmh", row["sum_abs_speed_centi_kmh"]) + try writer.count("movingSpeedSampleCount", row["moving_speed_sample_count"]) + try writer.int64("sumMovingAbsSpeedCentiKmh", row["sum_moving_abs_speed_centi_kmh"]) + try writer.int32("maxAbsSpeedCentiKmh", row["max_abs_speed_centi_kmh"]) + try writer.int32("minBatteryVoltageMv", row["min_battery_voltage_mv"]) + try writer.int32("maxMotorCurrentAbsMa", row["max_motor_current_abs_ma"]) + try writer.int32("maxBatteryCurrentAbsMa", row["max_battery_current_abs_ma"]) + try writer.int64("batteryUsedWhMilli", row["battery_used_wh_milli"]) + try writer.int64("batteryRegenWhMilli", row["battery_regen_wh_milli"]) + try writer.int32("maxDutyAbsPermille", row["max_duty_abs_permille"]) + try writer.int64("firstOdometerCm", row["first_odometer_cm"]) + try writer.int64("lastOdometerCm", row["last_odometer_cm"]) + try writer.count("gpsPointCount", row["gps_point_count"]) + try writer.count("preciseGpsPointCount", row["precise_gps_point_count"]) + try writer.int64("gpsDistanceCm", row["gps_distance_cm"]) + try writer.int32("maxGpsSpeedCentiMps", row["max_gps_speed_centi_mps"]) + try writer.int32("maxTempMosfetDeciC", row["max_temp_mosfet_deci_c"]) + try writer.int32("maxTempMotorDeciC", row["max_temp_motor_deci_c"]) + try writer.int32("firstLatitudeE7", row["first_latitude_e7"]) + try writer.int32("firstLongitudeE7", row["first_longitude_e7"]) + try writer.timestamp("firstMovingAtMs", row["first_moving_at_ms"]) + try writer.timestamp("lastMovingAtMs", row["last_moving_at_ms"]) + return writer.build() + } + + /// The Board name is resolved on read rather than snapshotted, so none crosses the wire. + static func favorite(_ row: Row) throws -> String { + let writer = SyncRowWriter(.favorites) + try writer.keyText("id", text(row, "id")) + try writer.nullableKeyText("boardId", row["board_id"]) + writer.text("name", row["name"]) + try writer.timestamp("startMs", row["start_ms"]) + try writer.timestamp("endMs", row["end_ms"]) + try writer.timestamp("createdAt", row["created_at"]) + try writer.timestamp("updatedAt", row["updated_at"]) + try writer.count("sampleCount", row["sample_count"]) + try writer.count("gpsPointCount", row["gps_point_count"]) + try writer.int64("distanceCm", row["distance_cm"]) + try writer.timestamp("movingDurationMs", row["moving_duration_ms"]) + try writer.int32("avgSpeedCentiKmh", row["avg_speed_centi_kmh"]) + try writer.int32("maxSpeedCentiKmh", row["max_speed_centi_kmh"]) + try writer.int64("batteryUsedWhMilli", row["battery_used_wh_milli"]) + return writer.build() + } + + /// One activation of a controller fault code on one Board (ADR-0037). A time series, so identity is + /// the app's minted id and never `(boardId, code)`. + /// + /// `code` crosses the wire raw: display has to tolerate a code the other side has never seen, so + /// the number is canonical and no interpretation of it travels. + static func vescFaultOccurrence(_ row: Row) throws -> String { + let writer = SyncRowWriter(.vescFaultOccurrences) + try writer.keyText("id", text(row, "id")) + try writer.keyText("boardId", text(row, "board_id")) + try writer.int32("code", row["code"]) + try writer.timestamp("occurredAtMs", row["occurred_at"]) + try writer.timestamp("lastObservedAtMs", row["last_observed_at"]) + try writer.timestamp("clearedAtMs", row["cleared_at"]) + writer.bool("dismissed", (row["dismissed"] as Int64? ?? 0) != 0) + try writer.timestamp("updatedAt", row["updated_at"]) + return writer.build() + } + + /// The window of decoded Board samples an Occurrence owns. Keyed by the Occurrence — one Occurrence + /// has at most one Capture — so the parent reference and the identity are the same field, and it + /// carries no Change Timestamp because it has no lifecycle to report. + static func vescFaultCapture(_ row: Row) throws -> String { + let writer = SyncRowWriter(.vescFaultCaptures) + try writer.keyText("occurrenceId", text(row, "occurrence_id")) + try writer.keyText("boardId", text(row, "board_id")) + try writer.timestamp("startedAtMs", row["started_at"]) + try writer.timestamp("openedAtMs", row["opened_at"]) + try writer.count("sampleCount", row["sample_count"]) + return writer.build() + } + + /// One decoded sample inside a Capture. The local row id stays home: it is an autoincrement that + /// restarts on a fresh install, so identity is `(occurrenceId, capturedAtMs)`, exactly as a Tune + /// History entry is keyed. + /// + /// Every value is nullable — a decoded sample carries whatever that Board Session actually + /// reported, and a field the firmware never sent is absent rather than zero. + static func vescFaultCaptureSample(_ row: Row) throws -> String { + let writer = SyncRowWriter(.vescFaultCaptureSamples) + try writer.keyText("occurrenceId", text(row, "occurrence_id")) + try writer.timestamp("capturedAtMs", row["captured_at"]) + try writer.reading("speed", row["speed"]) + try writer.reading("dutyCycle", row["duty_cycle"]) + try writer.reading("erpm", row["erpm"]) + try writer.reading("batteryVoltage", row["battery_voltage"]) + try writer.reading("batteryCurrent", row["battery_current"]) + try writer.reading("motorCurrent", row["motor_current"]) + try writer.reading("tempMosfet", row["temp_mosfet"]) + try writer.reading("tempMotor", row["temp_motor"]) + try writer.reading("pitch", row["pitch"]) + try writer.reading("roll", row["roll"]) + try writer.reading("balancePitch", row["balance_pitch"]) + try writer.reading("adc1", row["adc1"]) + try writer.reading("adc2", row["adc2"]) + try writer.int32("state", row["state"]) + return writer.build() + } + + /// One Sync Action, flat: the target, the identity within that target's scope, and when the Rider + /// removed it. The log's own `board_id`/`key` pair expands into the identity fields the server + /// declares for that target, so an action reads like the row it names. + static func deleteAction(_ row: Row) throws -> String { + let writer = SyncRowWriter(.deleteActions) + let target = text(row, "target") + let key = text(row, "key") + try writer.keyText("target", target) + switch target { + case "appSetting": try writer.keyText("key", key) + case "board": try writer.keyText("id", key) + case "boardSetting": + try writer.keyText("boardId", try board(row, target: target)) + try writer.keyText("key", key) + case "boardWarning": + try writer.keyText("boardId", try board(row, target: target)) + try writer.keyText("kind", key) + case "alert": + try writer.keyText("boardId", try board(row, target: target)) + try writer.keyText("id", key) + case "tuneProfile", "privacyZone", "favorite": try writer.keyText("id", key) + default: + throw SyncProtocolError(table: .deleteActions, field: "target", problem: "is not a known target") + } + try writer.timestamp("deletedAt", row["deleted_at"]) + return writer.build() + } + + private static func board(_ row: Row, target: String) throws -> String { + guard let boardId: String = row["board_id"] else { + throw SyncProtocolError(table: .deleteActions, field: "boardId", problem: "is missing for \(target)") + } + return boardId + } + + private static func text(_ row: Row, _ column: String) -> String { + (row[column] as String?) ?? "" + } +} diff --git a/modules/vescape-core/ios/sync/SyncWireTests.swift b/modules/vescape-core/ios/sync/SyncWireTests.swift new file mode 100644 index 000000000..98cc9c452 --- /dev/null +++ b/modules/vescape-core/ios/sync/SyncWireTests.swift @@ -0,0 +1,300 @@ +import XCTest +import GRDB +@testable import VescapeCore + +/// Wire encoding and the bounds it refuses on. The valid/invalid boundary cases mirror the server's +/// own schema (`vescape-server` `src/sync/protocol.ts`), so a row this side accepts is a row that +/// side can store — a batch is whole or refused, and a bad row must never reach transport. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/sync/SyncWireTest.kt +final class SyncWireTests: XCTestCase { + private func boardRow( + id: String = "board-1", + name: String = "Board", + transport: String? = nil + ) -> Row { + Row([ + "id": id, + "name": name, + "ble_id": nil, + "transport": transport, + "created_at": 10, + "updated_at": 20, + ]) + } + + private func frameRow(boardId: String? = "board-1", speed: Int64? = 100) -> Row { + var values: [String: DatabaseValueConvertible?] = [ + "id": 5, + "board_id": boardId, + "captured_at_ms": 1_000, + "elapsed_realtime_ms": 500, + "flags": 1, + "changed_mask_1": 3, + "changed_mask_2": 0, + "speed_centi_kmh": speed, + ] + for column in [ + "can_id", "battery_voltage_mv", "motor_current_ma", "battery_current_ma", "duty_permille", + "pitch_centi_deg", "roll_centi_deg", "balance_pitch_centi_deg", "balance_current_ma", "erpm", + "state", "switch_state", "adc1_milli", "adc2_milli", "odometer_cm", "temp_mosfet_deci_c", + "temp_motor_deci_c", "latitude_e7", "longitude_e7", "gps_speed_centi_mps", + "bearing_centi_deg", "accuracy_cm", "altitude_cm", "location_timestamp_ms", + ] { + values[column] = nil + } + return Row(values) + } + + func testABoardEncodesExactlyTheDeclaredFieldsNullsIncluded() throws { + XCTAssertEqual( + try SyncWire.board(boardRow()), + #"{"id":"board-1","name":"Board","bleId":null,"transport":null,"createdAt":10,"updatedAt":20}"# + ) + } + + /// iOS is the platform that stores Board Transport on the Board, and the server declares the field + /// for exactly that — dropping it would lose the Board Link's transport on restore. + func testABoardCarriesTheTransportOnlyIosStores() throws { + let encoded = try SyncWire.board(boardRow(transport: "direct")) + XCTAssertTrue(encoded.contains(#""transport":"direct""#)) + } + + /// "Cleared" and "not mentioned" are different intents, and only one survives a missing key. + func testNullableColumnsAreExplicitNullsNeverOmittedKeys() throws { + let encoded = try SyncWire.telemetryFrame(frameRow(speed: nil)) + XCTAssertTrue(encoded.contains(#""speedCentiKmh":null"#)) + } + + func testTextIsEscapedSoTheBodyStaysParseable() throws { + let encoded = try SyncWire.board(boardRow(name: "He said \"go\"\n")) + XCTAssertTrue(encoded.contains(#"\"go\""#)) + XCTAssertTrue(encoded.contains(#"\n"#)) + } + + func testAKeyAtTheLengthLimitIsValidAndOneOverIsRefused() throws { + _ = try SyncWire.board(boardRow(id: String(repeating: "b", count: maxSyncKeyLength))) + XCTAssertThrowsError( + try SyncWire.board(boardRow(id: String(repeating: "b", count: maxSyncKeyLength + 1))) + ) + } + + /// The server compiles `value.length <= 128`, which counts UTF-16 code units — so an emoji is two. + /// Swift's `count` would see 64 characters here and let a key through that the server refuses. + func testKeyLengthIsMeasuredInUtf16CodeUnitsLikeTheServer() { + let emoji = String(repeating: "🛹", count: 65) + XCTAssertEqual(emoji.count, 65) + XCTAssertEqual(emoji.utf16.count, 130) + XCTAssertThrowsError(try SyncWire.board(boardRow(id: emoji))) + } + + func testAnEmptyKeyIsRefusedWhereTheServerNamesIt() throws { + XCTAssertThrowsError(try SyncWire.board(boardRow(id: ""))) + _ = try SyncWire.appSetting( + Row(["key": "mapStyleKey", "value_json": "\"\"", "updated_at": 1]) + ) + } + + /// A sample that names no Board has nowhere to go on the server, so it never reaches transport. + func testAFrameWithoutABoardIsAProtocolError() { + XCTAssertThrowsError(try SyncWire.telemetryFrame(frameRow(boardId: nil))) { error in + XCTAssertEqual((error as? SyncProtocolError)?.field, "boardId") + } + } + + func testIntegerBoundsAreEnforcedAtTheEdge() throws { + _ = try SyncWire.telemetryFrame(frameRow(speed: Int64(Int32.max))) + XCTAssertThrowsError(try SyncRowWriter(.telemetryFrames).int32("speedCentiKmh", 2_147_483_648)) + } + + func testANonFiniteNumberIsRefusedBecauseJsonCannotExpressIt() { + XCTAssertThrowsError(try SyncRowWriter(.alerts).number("threshold", Double.nan)) { error in + XCTAssertEqual((error as? SyncProtocolError)?.field, "threshold") + } + } + + /// An action reads like the row it names: flat identity fields, not a nested envelope. + func testADeleteActionExpandsIntoTheIdentityItsTargetDeclares() throws { + XCTAssertEqual( + try SyncWire.deleteAction( + Row(["target": "boardSetting", "board_id": "board-1", "key": "transport", "deleted_at": 9]) + ), + #"{"target":"boardSetting","boardId":"board-1","key":"transport","deletedAt":9}"# + ) + XCTAssertEqual( + try SyncWire.deleteAction( + Row(["target": "tuneProfile", "board_id": nil, "key": "profile-1", "deleted_at": 4]) + ), + #"{"target":"tuneProfile","id":"profile-1","deletedAt":4}"# + ) + XCTAssertThrowsError( + try SyncWire.deleteAction( + Row(["target": "somethingElse", "board_id": nil, "key": "x", "deleted_at": 1]) + ) + ) + } + + // MARK: - Columns the server has dropped + + /// Both columns went away when VESC faults became Board-owned evidence in their own tables + /// (ADR-0037). The server's schema is strict, so a field it no longer declares does not get + /// ignored — it refuses the whole Sync Batch. + func testRetiredFaultFieldsAreNoLongerSent() throws { + XCTAssertFalse(try SyncWire.telemetryFrame(frameRow()).contains("faultCode")) + XCTAssertFalse(try SyncWire.telemetryMinuteBucket(bucketRow()).contains("faultCount")) + } + + private func bucketRow() -> Row { + var values: [String: DatabaseValueConvertible?] = [ + "board_id": "board-1", + "bucket_start_ms": 60_000, + "updated_at": 61_000, + "sample_count": 2, + "first_sample_at_ms": 60_000, + "last_sample_at_ms": 60_500, + "sum_abs_speed_centi_kmh": 10, + "max_abs_speed_centi_kmh": 9, + "max_motor_current_abs_ma": 1, + "max_battery_current_abs_ma": 1, + "battery_used_wh_milli": 1, + "battery_regen_wh_milli": 0, + "max_duty_abs_permille": 5, + "gps_point_count": 0, + "precise_gps_point_count": 0, + "gps_distance_cm": 0, + ] + for column in [ + "moving_speed_sample_count", "sum_moving_abs_speed_centi_kmh", "min_battery_voltage_mv", + "first_odometer_cm", "last_odometer_cm", "max_gps_speed_centi_mps", "max_temp_mosfet_deci_c", + "max_temp_motor_deci_c", "first_latitude_e7", "first_longitude_e7", "first_moving_at_ms", + "last_moving_at_ms", + ] { + values[column] = nil + } + return Row(values) + } + + // MARK: - Alert Rules + + /// A rule that travels without its kind and offsets comes back looking configured and firing at + /// the wrong point, which is worse than losing it. + func testAnAlertRuleCarriesTheWholeThresholdRuleNotJustTheNumber() throws { + let encoded = try SyncWire.alert( + Row([ + "board_id": "board-1", "id": "rule-1", "control_id": "duty", "threshold": 70.0, + "threshold_max": nil, "threshold_kind": "configRelative", "config_field_id": "tiltback_duty", + "threshold_offset": -5.0, "threshold_max_offset": nil, "enabled": 1, "sound_type": "beep", + "repeat_every_seconds": 30, "beep_count": 2, "source": "preset", "created_at": 1, + "updated_at": 2, + ]) + ) + + XCTAssertEqual( + encoded, + #"{"boardId":"board-1","id":"rule-1","controlId":"duty","threshold":70,"thresholdMax":null,"# + + #""thresholdKind":"configRelative","configFieldId":"tiltback_duty","thresholdOffset":-5,"# + + #""thresholdMaxOffset":null,"enabled":true,"soundType":"beep","repeatEverySeconds":30,"# + + #""beepCount":2,"source":"preset","createdAt":1,"updatedAt":2}"# + ) + } + + /// A Board Warning's detection time and its Change Timestamp answer different questions: a + /// re-triage that lowers a severity changes the row without the Board being seen in that state + /// again, and judging arrivals on the detection time drops exactly those edits. + func testABoardWarningCarriesItsChangeTimestampAsWellAsItsDetectionTime() throws { + XCTAssertEqual( + try SyncWire.boardWarning( + Row([ + "board_id": "board-1", "kind": "batteryImbalance", "severity": "warning", + "first_detected_at": 1, "last_detected_at": 2, "payload_json": "{}", "updated_at": 7, + ]) + ), + #"{"boardId":"board-1","kind":"batteryImbalance","severity":"warning","firstDetectedAt":1,"# + + #""lastDetectedAt":2,"payloadJson":"{}","updatedAt":7}"# + ) + } + + // MARK: - VESC Fault Evidence + + func testAFaultOccurrenceEncodesExactlyTheDeclaredFields() throws { + XCTAssertEqual( + try SyncWire.vescFaultOccurrence( + Row([ + "id": "fault-1", "board_id": "board-1", "code": 9, "occurred_at": 1_000, + "last_observed_at": 4_000, "cleared_at": nil, "dismissed": 1, "updated_at": 5_000, + "sync_seq": 3, + ]) + ), + #"{"id":"fault-1","boardId":"board-1","code":9,"occurredAtMs":1000,"lastObservedAtMs":4000,"# + + #""clearedAtMs":null,"dismissed":true,"updatedAt":5000}"# + ) + } + + func testAFaultCaptureEncodesExactlyTheDeclaredFields() throws { + XCTAssertEqual( + try SyncWire.vescFaultCapture( + Row([ + "occurrence_id": "fault-1", "board_id": "board-1", "started_at": 900, "opened_at": 1_000, + "sample_count": 3, "sync_seq": 2, + ]) + ), + #"{"occurrenceId":"fault-1","boardId":"board-1","startedAtMs":900,"openedAtMs":1000,"sampleCount":3}"# + ) + } + + /// The local autoincrement id restarts on a fresh install, so it never crosses the wire: identity + /// is `(occurrenceId, capturedAtMs)`. A field the firmware never sent is null, never zero. + func testAFaultCaptureSampleKeepsItsLocalIdHomeAndSendsAbsentFieldsAsNull() throws { + var values: [String: DatabaseValueConvertible?] = [ + "id": 41, + "occurrence_id": "fault-1", + "captured_at": 1_000, + "speed": 12.5, + "state": 4, + ] + for column in [ + "duty_cycle", "erpm", "battery_voltage", "battery_current", "motor_current", "temp_mosfet", + "temp_motor", "pitch", "roll", "balance_pitch", "adc1", "adc2", + ] { + values[column] = nil + } + + XCTAssertEqual( + try SyncWire.vescFaultCaptureSample(Row(values)), + #"{"occurrenceId":"fault-1","capturedAtMs":1000,"speed":12.5,"dutyCycle":null,"erpm":null,"# + + #""batteryVoltage":null,"batteryCurrent":null,"motorCurrent":null,"tempMosfet":null,"# + + #""tempMotor":null,"pitch":null,"roll":null,"balancePitch":null,"adc1":null,"adc2":null,"# + + #""state":4}"# + ) + } + + /// A decoded Board sample is the one thing on the wire the app did not author — it received it. + /// Refusing a non-finite float here would pause every table's backup on a permanent protocol + /// error that no retry can clear, over a reading the firmware itself could not express. Absent is + /// what these nullable columns already mean, so an unusable reading is absent too. + func testAnUnusableFirmwareReadingIsAbsentRatherThanAPermanentProtocolPause() throws { + var values: [String: DatabaseValueConvertible?] = [ + "id": 42, + "occurrence_id": "fault-1", + "captured_at": 1_000, + "speed": Double.nan, + "duty_cycle": Double.infinity, + "erpm": -Double.infinity, + "battery_voltage": 78.9, + "state": 4, + ] + for column in [ + "battery_current", "motor_current", "temp_mosfet", "temp_motor", "pitch", "roll", + "balance_pitch", "adc1", "adc2", + ] { + values[column] = nil + } + + let encoded = try SyncWire.vescFaultCaptureSample(Row(values)) + + XCTAssertTrue(encoded.contains(#""speed":null"#)) + XCTAssertTrue(encoded.contains(#""dutyCycle":null"#)) + XCTAssertTrue(encoded.contains(#""erpm":null"#)) + XCTAssertTrue(encoded.contains(#""batteryVoltage":78.9"#), "a usable reading beside an unusable one still lands") + } +} diff --git a/modules/vescape-core/ios/telemetry/AppDataRepository.swift b/modules/vescape-core/ios/telemetry/AppDataRepository.swift index 4905cf4ba..a9e3f4457 100644 --- a/modules/vescape-core/ios/telemetry/AppDataRepository.swift +++ b/modules/vescape-core/ios/telemetry/AppDataRepository.swift @@ -17,6 +17,47 @@ enum AppDataScope: String { case settings } +/// App settings that name *this phone* rather than the Rider, and so never leave it: restoring them +/// onto a second phone would overwrite that phone's own identity or session state. Enforced at the +/// write path — `writeAppSetting` leaves their `sync_seq` at 0, which is below every Sync Cursor, so +/// no upload scan ever sees the row. +/// +/// Rider Name and Rider Color live in `app_settings` by design, so that Group Ride keeps working +/// signed-out; that placement is what makes them phone-local rather than Account-scoped. See #277. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `NOT_SYNCED_SETTING_KEYS` +internal let notSyncedSettingKeys: [String] = [ + // Rider identity — a second phone in the same Group Ride must not become the same Rider. + "riderId", + "riderName", + "riderColor", + // Device/session state — names this phone's current session, not the Rider's configuration. + "selectedBoardId", + "lastGpsLatitude", + "lastGpsLongitude", + "directionPointLatitude", + "directionPointLongitude", + // Connection and companion behaviour — phone-side BLE and foreground policy. + "autoConnect", + "companionPresenceEnabled", + "companionPresenceCooldownMinutes", + "connectionSoundsEnabled", + "autoCloseEnabled", + "autoCloseDelayMinutes", + // Wear pairing — the watch is paired to one phone. + "wearMirrorIntervalMs", + "wearAutoLaunchOnConnect", + // The backup master switch is per phone, and deliberately does not travel through the mechanism + // it turns off: a restored snapshot must never be able to switch backup back on. + "syncEnabled", + // The backup choice is per phone: the expensive first upload belongs to the phone that holds the + // backlog, so a restore onto a second phone asks that Rider again rather than deciding for them. + "syncBackupChoiceMade", + // So is the data-plan choice, and for the same reason the other two are: it answers what this + // phone's connection costs, not what the Rider prefers. Travelling would let a restore onto a + // cellular-only phone inherit the other phone's answer and upload a ride over metered data. + "syncWifiOnly", +] + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/AppDataRepository.kt final class AppDataRepository { static let shared = AppDataRepository() @@ -84,7 +125,7 @@ final class AppDataRepository { let boards = try Row.fetchAll( db, sql: """ - SELECT id, name, ble_id, transport, created_at, deleted_at FROM boards + SELECT id, name, ble_id, transport, created_at, updated_at, deleted_at FROM boards WHERE deleted_at IS NULL ORDER BY created_at ASC """ ) @@ -106,7 +147,7 @@ final class AppDataRepository { guard let board = try Row.fetchOne( db, sql: """ - SELECT id, name, ble_id, transport, created_at, deleted_at FROM boards + SELECT id, name, ble_id, transport, created_at, updated_at, deleted_at FROM boards WHERE id = ? LIMIT 1 """, arguments: [id] @@ -140,28 +181,44 @@ final class AppDataRepository { // Legal Mode changes only through the dedicated native intent. ] + linkSettings.filter { $0.0 != "transport" } let transport = linkSettings.first { $0.0 == "transport" }?.1 as? String + // One stamp for the board row's last-write-wins timestamp and every board-setting row written + // below. Native stamps it rather than trusting the bridge value: it must come from the device + // clock that already writes `created_at` and must move on every upsert, including partial edits. let updatedAt = nowMs() write { db in + // Read-modify-write rather than an `ON CONFLICT` fold: `INSERT OR REPLACE` deletes the old row + // before inserting, so the ratchet has no `excluded`-style handle on the value it replaces. + let previous = try Int64.fetchOne(db, sql: "SELECT updated_at FROM boards WHERE id = ?", arguments: [id]) // An existing tombstone survives the write, so an ordinary upsert can never resurrect a // deleted Board — deletion is terminal (ADR 0027). Only `deleteBoard` stamps a new one. let deletedAt = try Int64.fetchOne(db, sql: "SELECT deleted_at FROM boards WHERE id = ?", arguments: [id]) try db.execute( sql: """ - INSERT OR REPLACE INTO boards (id, name, ble_id, transport, created_at, deleted_at) - VALUES (?, ?, ?, ?, ?, ?) + INSERT OR REPLACE INTO boards (id, name, ble_id, transport, created_at, updated_at, sync_seq, deleted_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) """, - arguments: [id, name, bleId, transport, createdAt, deletedAt] + arguments: [ + id, name, bleId, transport, createdAt, + ratchetUpdatedAt(previous, updatedAt), try nextSyncSeq(db, syncSeqBoards), deletedAt, + ] ) for (key, value) in settings { guard let value, let json = Self.encodeJson(value) else { - try db.execute(sql: "DELETE FROM board_settings WHERE board_id = ? AND key = ?", arguments: [id, key]) + // Semantic removal: a Board edit that drops a key is the Rider clearing that setting, so a + // restore must not resurrect the old value (#282). + try deleteForSync( + db, + target: .boardSetting, + boardId: id, + key: key, + whereClause: "board_id = ? AND key = ?", + keys: [id, key], + now: updatedAt + ) continue } - try db.execute( - sql: "INSERT OR REPLACE INTO board_settings (board_id, key, value_json, updated_at) VALUES (?, ?, ?, ?)", - arguments: [id, key, json, updatedAt] - ) + try Self.writeBoardSetting(db, boardId: id, key: key, json: json, now: updatedAt) } } notifyDataChanged(.boards) @@ -170,18 +227,37 @@ final class AppDataRepository { /// The Rider-facing delete: configuration goes, the Board row stays as a tombstone (ADR 0027). /// Telemetry and Tune Profiles are untouched — both outlive the Board. /// - /// A Board that is not there (or already deleted) is left alone. + /// The tombstone is an ordinary write, so it moves both sync columns like any other edit. A Board + /// that is not there (or already deleted) is left alone. /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `deleteBoardWithSettings` func deleteBoard(_ id: String) { let deletedAt = nowMs() write { db in + guard let row = try Row.fetchOne( + db, + sql: "SELECT updated_at, deleted_at FROM boards WHERE id = ?", + arguments: [id] + ), row["deleted_at"] as Int64? == nil else { return } + // The children are raw deletes — the Board's own Sync Action covers the whole cascade, so an + // upsert never quietly deletes rows in three other tables (#282). try db.execute(sql: "DELETE FROM board_settings WHERE board_id = ?", arguments: [id]) try db.execute(sql: "DELETE FROM board_warnings WHERE board_id = ?", arguments: [id]) // Alert Rules are Board-owned (#254) — drop them with the Board so no orphan rows survive. try db.execute(sql: "DELETE FROM alerts WHERE board_id = ?", arguments: [id]) + // The action and the tombstone share one timestamp, the newly ratcheted `updated_at`, so the + // server judges both against the same moment. + let tombstonedAt = ratchetUpdatedAt(row["updated_at"] as Int64?, deletedAt) + try appendDeleteAction( + db, + target: .board, + boardId: nil, + key: id, + rowStamp: tombstonedAt, + now: tombstonedAt + ) try db.execute( - sql: "UPDATE boards SET deleted_at = ? WHERE id = ? AND deleted_at IS NULL", - arguments: [deletedAt, id] + sql: "UPDATE boards SET deleted_at = ?, updated_at = ?, sync_seq = ? WHERE id = ?", + arguments: [tombstonedAt, tombstonedAt, try nextSyncSeq(db, syncSeqBoards), id] ) } BoardConfigStore.shared.clear(boardId: id) @@ -195,10 +271,7 @@ final class AppDataRepository { let value: [String: Any] = ["percent": percent, "voltage": voltage ?? NSNull(), "at": atMs] guard let json = Self.encodeJson(value) else { return } write { db in - try db.execute( - sql: "INSERT OR REPLACE INTO board_settings (board_id, key, value_json, updated_at) VALUES (?, ?, ?, ?)", - arguments: [boardId, "lastBattery", json, atMs] - ) + try Self.writeBoardSetting(db, boardId: boardId, key: "lastBattery", json: json, now: atMs) } notifyDataChanged(.boards) } @@ -208,10 +281,7 @@ final class AppDataRepository { func updateLegalMode(boardId: String, enabled: Bool) { guard let json = Self.encodeJson(["enabled": enabled]) else { return } write { db in - try db.execute( - sql: "INSERT OR REPLACE INTO board_settings (board_id, key, value_json, updated_at) VALUES (?, ?, ?, ?)", - arguments: [boardId, "legalMode", json, self.nowMs()] - ) + try Self.writeBoardSetting(db, boardId: boardId, key: "legalMode", json: json, now: self.nowMs()) } notifyDataChanged(.boards) } @@ -241,6 +311,7 @@ final class AppDataRepository { "matchBoardConfig": values["matchBoardConfig"] ?? nil, "legalMode": values["legalMode"] ?? ["enabled": false], "link": link, + "updatedAt": row["updated_at"] as Int64, "deletedAt": row["deleted_at"] as Int64?, ] } @@ -334,6 +405,7 @@ final class AppDataRepository { "repeatEverySeconds": row["repeat_every_seconds"] as Int64?, "beepCount": row["beep_count"] as Int? ?? alertBeepCountDefault, "source": row["source"] as String?, + "updatedAt": row["updated_at"] as Int64, ] } } @@ -365,7 +437,8 @@ final class AppDataRepository { createdAt: row["created_at"] as Int64, repeatEverySeconds: row["repeat_every_seconds"] as Int64?, beepCount: row["beep_count"] as Int? ?? alertBeepCountDefault, - source: row["source"] as String? + source: row["source"] as String?, + updatedAt: row["updated_at"] as Int64 ) } } @@ -385,34 +458,69 @@ final class AppDataRepository { let repeatEverySeconds = normalizedAlertRepeatSeconds(Self.doubleValue(rule["repeatEverySeconds"] ?? nil)) let beepCount = normalizedAlertBeepCount(Self.longValue(rule["beepCount"] ?? nil).map { Int($0) }) let source = rule["source"] as? String + // Native stamps the last-write-wins timestamp rather than trusting the bridge value: it must come + // from the device clock that already writes `created_at` and must move on every upsert. + let updatedAt = nowMs() let thresholdRule = rule["thresholdRule"] as? [String: Any] let thresholdKind = thresholdRule?["kind"] as? String ?? "fixed" let configFieldId = thresholdRule?["fieldId"] as? String let thresholdOffset = Self.doubleValue(thresholdRule?["thresholdOffset"]) let thresholdMaxOffset = Self.doubleValue(thresholdRule?["thresholdMaxOffset"]) write { db in + // See `upsertBoard` for why the ratchet reads the old value instead of folding it on conflict. + let previous = try Int64.fetchOne( + db, + sql: "SELECT updated_at FROM alerts WHERE board_id = ? AND id = ?", + arguments: [boardId, id] + ) try db.execute( sql: """ - INSERT OR REPLACE INTO alerts (board_id, id, control_id, threshold, threshold_max, enabled, sound_type, created_at, repeat_every_seconds, beep_count, source, threshold_kind, config_field_id, threshold_offset, threshold_max_offset) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + INSERT OR REPLACE INTO alerts (board_id, id, control_id, threshold, threshold_max, enabled, sound_type, created_at, repeat_every_seconds, beep_count, source, threshold_kind, config_field_id, threshold_offset, threshold_max_offset, updated_at, sync_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, - arguments: [boardId, id, controlId, threshold, thresholdMax, enabled ? 1 : 0, soundType, createdAt, repeatEverySeconds, beepCount, source, thresholdKind, configFieldId, thresholdOffset, thresholdMaxOffset] + arguments: [ + boardId, id, controlId, threshold, thresholdMax, enabled ? 1 : 0, soundType, createdAt, + repeatEverySeconds, beepCount, source, thresholdKind, configFieldId, thresholdOffset, + thresholdMaxOffset, + ratchetUpdatedAt(previous, updatedAt), try nextSyncSeq(db, syncSeqAlerts), + ] ) } } + /// Targeted toggle. Unlike `upsertAlertRule` it never rewrites the whole row, so both sync columns + /// have to move here explicitly — without them, toggling a rule leaves it invisible to the upload + /// scan and the change never reaches the server. + /// + /// The `MAX(updated_at + 1, ?)` fold is the same ratchet `ratchetUpdatedAt` applies, expressed in + /// SQL because the row is already being read by the `WHERE`. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `setAlertRuleEnabled` func setAlertRuleEnabled(_ boardId: String, _ id: String, _ enabled: Bool) { + let updatedAt = nowMs() write { db in try db.execute( - sql: "UPDATE alerts SET enabled = ? WHERE board_id = ? AND id = ?", - arguments: [enabled ? 1 : 0, boardId, id] + sql: """ + UPDATE alerts SET enabled = ?, updated_at = MAX(updated_at + 1, ?), sync_seq = ? + WHERE board_id = ? AND id = ? + """, + arguments: [enabled ? 1 : 0, updatedAt, try nextSyncSeq(db, syncSeqAlerts), boardId, id] ) } } + /// Semantic removal, and the path preset regeneration takes too: JS regenerates a Board's preset + /// rules by deleting the old ones and writing new ones, and the deleted ones have to disappear + /// server-side as well (#282). func deleteAlertRule(_ boardId: String, _ id: String) { write { db in - try db.execute(sql: "DELETE FROM alerts WHERE board_id = ? AND id = ?", arguments: [boardId, id]) + try deleteForSync( + db, + target: .alert, + boardId: boardId, + key: id, + whereClause: "board_id = ? AND id = ?", + keys: [boardId, id] + ) } } @@ -450,29 +558,52 @@ final class AppDataRepository { let createdAt = Self.longValue(zone["createdAt"] ?? nil) ?? now let updatedAt = Self.longValue(zone["updatedAt"] ?? nil) ?? now write { db in + let stamp = try stampSyncColumns( + db, + table: "privacy_zones", + sequence: syncSeqPrivacyZones, + whereClause: "id = ?", + keys: [id], + now: updatedAt + ) try db.execute( sql: """ INSERT OR REPLACE INTO privacy_zones - (id, preset, name, enabled, center_latitude_e7, center_longitude_e7, radius_meters, created_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + (id, preset, name, enabled, center_latitude_e7, center_longitude_e7, radius_meters, created_at, updated_at, sync_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, - arguments: [id, preset, name, enabled ? 1 : 0, latitude.toE7, longitude.toE7, radius, createdAt, updatedAt] + arguments: [ + id, preset, name, enabled ? 1 : 0, latitude.toE7, longitude.toE7, radius, createdAt, + stamp.updatedAt, stamp.syncSeq, + ] ) } } + /// Targeted toggle that bypasses the upsert, so it moves both sync columns itself; see + /// `setAlertRuleEnabled`. func setPrivacyZoneEnabled(_ id: String, _ enabled: Bool) { let updatedAt = nowMs() write { db in try db.execute( - sql: "UPDATE privacy_zones SET enabled = ?, updated_at = ? WHERE id = ?", - arguments: [enabled ? 1 : 0, updatedAt, id] + sql: "UPDATE privacy_zones SET enabled = ?, updated_at = MAX(updated_at + 1, ?), sync_seq = ? WHERE id = ?", + arguments: [enabled ? 1 : 0, updatedAt, try nextSyncSeq(db, syncSeqPrivacyZones), id] ) } } + /// Semantic removal: the Rider deleted the zone, so the server has to lose it too (#282). func deletePrivacyZone(_ id: String) { - write { db in try db.execute(sql: "DELETE FROM privacy_zones WHERE id = ?", arguments: [id]) } + write { db in + try deleteForSync( + db, + target: .privacyZone, + boardId: nil, + key: id, + whereClause: "id = ?", + keys: [id] + ) + } } // MARK: - Direction point @@ -595,7 +726,7 @@ final class AppDataRepository { else { return } let updatedAt = nowMs() guard let rawValue, !(rawValue is NSNull) else { - write { db in try db.execute(sql: "DELETE FROM app_settings WHERE key = ?", arguments: [key]) } + write { db in try Self.deleteAppSetting(db, key: key, now: updatedAt) } notifyDataChanged(.settings) return } @@ -612,6 +743,11 @@ final class AppDataRepository { } else if key == "satelliteImagerySaturation" { guard let saturation = Self.satelliteImagerySaturation(rawValue) else { return } value = saturation + } else if key == "syncEnabled" || key == "syncWifiOnly" || key == "syncBackupChoiceMade" { + // Strict Bool (Android rejects non-Boolean too): the backup switch must never persist a + // malformed value that reads back truthy. + guard let flag = rawValue as? Bool else { return } + value = flag } else if key == "themeMode" { guard let mode = Self.themeMode(rawValue) else { return } value = mode @@ -638,11 +774,12 @@ final class AppDataRepository { } guard let json = Self.encodeJson(value) else { return } write { db in - try db.execute( - sql: "INSERT OR REPLACE INTO app_settings (key, value_json, updated_at) VALUES (?, ?, ?)", - arguments: [key, json, updatedAt] - ) + try Self.writeAppSetting(db, key: key, json: json, now: updatedAt) } + // The uploader reads the Wi-Fi switch from native truth, not from a JS call, so a write from any + // source — the settings row, the one-time choice, a restored backup — reaches it the same way. + if key == "syncEnabled" { SyncCoordinator.shared.setEnabled(value as? Bool ?? false) } + if key == "syncWifiOnly" { SyncCoordinator.shared.setWifiOnly(value as? Bool ?? false) } notifyDataChanged(.settings) } @@ -653,17 +790,82 @@ final class AppDataRepository { let value = code.flatMap { $0.count == 2 ? ["jurisdictionCode": $0] : nil } write { db in guard let value, let json = Self.encodeJson(value) else { - try db.execute(sql: "DELETE FROM app_settings WHERE key = 'legalPolicy'") + try Self.deleteAppSetting(db, key: "legalPolicy", now: self.nowMs()) return } - try db.execute( - sql: "INSERT OR REPLACE INTO app_settings (key, value_json, updated_at) VALUES (?, ?, ?)", - arguments: ["legalPolicy", json, self.nowMs()] - ) + try Self.writeAppSetting(db, key: "legalPolicy", json: json, now: self.nowMs()) } notifyDataChanged(.settings) } + /// Stamps both sync columns; see `upsertBoard`. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `upsertBoardSetting` + private static func writeBoardSetting( + _ db: Database, + boardId: String, + key: String, + json: String, + now: Int64 + ) throws { + let stamp = try stampSyncColumns( + db, + table: "board_settings", + sequence: syncSeqBoardSettings, + whereClause: "board_id = ? AND key = ?", + keys: [boardId, key], + now: now + ) + try db.execute( + sql: """ + INSERT OR REPLACE INTO board_settings (board_id, key, value_json, updated_at, sync_seq) + VALUES (?, ?, ?, ?, ?) + """, + arguments: [boardId, key, json, stamp.updatedAt, stamp.syncSeq] + ) + } + + /// Semantic removal of a stored app setting. Every caller means the same thing — the stored + /// override is gone: an edit back to the default, and `legalPolicy` resolving to nothing. + /// + /// Phone-local keys never reach the server (they carry `sync_seq = 0`), so removing one records no + /// action either — an action for a row the server never held would delete nothing and say nothing. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `deleteAppSetting` + internal static func deleteAppSetting(_ db: Database, key: String, now: Int64) throws { + guard !notSyncedSettingKeys.contains(key) else { + try db.execute(sql: "DELETE FROM app_settings WHERE key = ?", arguments: [key]) + return + } + try deleteForSync( + db, + target: .appSetting, + boardId: nil, + key: key, + whereClause: "key = ?", + keys: [key], + now: now + ) + } + + /// Stamps both sync columns like `upsertBoard`, except for the phone-local keys in + /// [notSyncedSettingKeys]: those keep `sync_seq` at 0, which sits below every Sync Cursor, so the + /// upload scan never picks the row up and the key stays on this phone (#277). + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `upsertAppSetting` + private static func writeAppSetting(_ db: Database, key: String, json: String, now: Int64) throws { + let previous = try Int64.fetchOne( + db, + sql: "SELECT updated_at FROM app_settings WHERE key = ?", + arguments: [key] + ) + let syncSeq = notSyncedSettingKeys.contains(key) ? 0 : try nextSyncSeq(db, syncSeqAppSettings) + try db.execute( + sql: """ + INSERT OR REPLACE INTO app_settings (key, value_json, updated_at, sync_seq) + VALUES (?, ?, ?, ?) + """, + arguments: [key, json, ratchetUpdatedAt(previous, now), syncSeq] + ) + } + // MARK: - Shared pure helpers (also used by VescapeCoreModule bridge glue) /// Durable app-scoped settings shape. A TS/Android/iOS parity triangle — the container tag covers @@ -682,6 +884,12 @@ final class AppDataRepository { // the keys exist here only so getSettings() returns the full settings shape. "autoCloseEnabled": false, "autoCloseDelayMinutes": 15, + // Backup master switch. Off by default: the uploader does nothing until the Rider turns it on. + "syncEnabled": false, + // Nothing uploads on a metered connection while this is on — mid-ride included. + "syncWifiOnly": false, + // The one-time backup choice has been offered on this phone and answered. + "syncBackupChoiceMade": false, "selectedBoardId": NSNull(), "riderId": NSNull(), "riderName": NSNull(), diff --git a/modules/vescape-core/ios/telemetry/BoardTombstoneTests.swift b/modules/vescape-core/ios/telemetry/BoardTombstoneTests.swift index 5325b17c6..9443a9793 100644 --- a/modules/vescape-core/ios/telemetry/BoardTombstoneTests.swift +++ b/modules/vescape-core/ios/telemetry/BoardTombstoneTests.swift @@ -138,4 +138,21 @@ final class BoardTombstoneTests: XCTestCase { XCTAssertNotNil(try deletedAt("board-1"), "an upsert cleared the tombstone") XCTAssertTrue(repo.getBoards().isEmpty, "a resurrected Board came back to the list") } + + /// A tombstone is an ordinary write: the server only keeps it if it arrives with a newer stamp + /// and the upload scan only sees it if its Sync Cursor moved. + func testDeleteMovesBothSyncColumns() throws { + seedBoard() + let before = try queue.read { db in + try Row.fetchOne(db, sql: "SELECT updated_at, sync_seq FROM boards WHERE id = 'board-1'")! + } + + repo.deleteBoard("board-1") + + let after = try queue.read { db in + try Row.fetchOne(db, sql: "SELECT updated_at, sync_seq FROM boards WHERE id = 'board-1'")! + } + XCTAssertGreaterThan(after["updated_at"] as Int64, before["updated_at"] as Int64) + XCTAssertGreaterThan(after["sync_seq"] as Int64, before["sync_seq"] as Int64) + } } diff --git a/modules/vescape-core/ios/telemetry/DatabaseBackupManager.swift b/modules/vescape-core/ios/telemetry/DatabaseBackupManager.swift index 89e0a6260..c279ab393 100644 --- a/modules/vescape-core/ios/telemetry/DatabaseBackupManager.swift +++ b/modules/vescape-core/ios/telemetry/DatabaseBackupManager.swift @@ -7,7 +7,7 @@ import GRDB /// `TelemetryDatabase.migrator`. /// /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `TELEMETRY_DATABASE_VERSION` -internal let TELEMETRY_SCHEMA_VERSION = 42 +internal let TELEMETRY_SCHEMA_VERSION = 48 private let MANIFEST_ENTRY = "manifest.json" private let DATABASE_ENTRY = "db.sqlite" diff --git a/modules/vescape-core/ios/telemetry/FavoriteStore.swift b/modules/vescape-core/ios/telemetry/FavoriteStore.swift index 4d89dfc49..bca5970ba 100644 --- a/modules/vescape-core/ios/telemetry/FavoriteStore.swift +++ b/modules/vescape-core/ios/telemetry/FavoriteStore.swift @@ -154,11 +154,15 @@ struct FavoriteStore { moving_duration_ms INTEGER NOT NULL, avg_speed_centi_kmh INTEGER NOT NULL, max_speed_centi_kmh INTEGER NOT NULL, - battery_used_wh_milli INTEGER NOT NULL + battery_used_wh_milli INTEGER NOT NULL, + sync_seq INTEGER NOT NULL DEFAULT 0 ) """) try db.execute(sql: "CREATE INDEX index_favorites_start_ms_end_ms ON favorites(start_ms, end_ms)") try db.execute(sql: "CREATE INDEX index_favorites_board_id ON favorites(board_id)") + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_favorites_sync_seq ON favorites(sync_seq)") + try createSyncSequencesTable(db) + try createSyncActionsTable(db) } // MARK: - Reads @@ -183,8 +187,8 @@ struct FavoriteStore { INSERT INTO favorites ( id, board_id, name, start_ms, end_ms, created_at, updated_at, sample_count, gps_point_count, distance_cm, moving_duration_ms, - avg_speed_centi_kmh, max_speed_centi_kmh, battery_used_wh_milli - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + avg_speed_centi_kmh, max_speed_centi_kmh, battery_used_wh_milli, sync_seq + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, arguments: [ favorite.id, favorite.boardId, favorite.name, @@ -192,6 +196,7 @@ struct FavoriteStore { favorite.summary.sampleCount, favorite.summary.gpsPointCount, favorite.summary.distanceCm, favorite.summary.movingDurationMs, favorite.summary.avgSpeedCentiKmh, favorite.summary.maxSpeedCentiKmh, favorite.summary.batteryUsedWhMilli, + try nextSyncSeq(db, syncSeqFavorites), ] ) } @@ -208,13 +213,14 @@ struct FavoriteStore { try db.execute( sql: """ UPDATE favorites SET - name = ?, start_ms = ?, end_ms = ?, updated_at = ?, + name = ?, start_ms = ?, end_ms = ?, updated_at = MAX(updated_at + 1, ?), sync_seq = ?, sample_count = ?, gps_point_count = ?, distance_cm = ?, moving_duration_ms = ?, avg_speed_centi_kmh = ?, max_speed_centi_kmh = ?, battery_used_wh_milli = ? WHERE id = ? """, arguments: [ favorite.name, favorite.startMs, favorite.endMs, favorite.updatedAtMs, + try nextSyncSeq(db, syncSeqFavorites), favorite.summary.sampleCount, favorite.summary.gpsPointCount, favorite.summary.distanceCm, favorite.summary.movingDurationMs, favorite.summary.avgSpeedCentiKmh, favorite.summary.maxSpeedCentiKmh, @@ -233,15 +239,23 @@ struct FavoriteStore { } /// Unpin one Favorite. Telemetry inside its range is untouched and becomes deletable again. - /// Favorite Media rows are parent-covered and raw-deleted in the same transaction (ADR 0030); + /// Emits one Sync Action for the Favorite; its Favorite Media rows emit none, because the parent + /// action covers them. Favorite Media rows are parent-covered and raw-deleted in the same + /// transaction (ADR 0030); /// filesystem cleanup is best-effort in the repository after this commit succeeds. @discardableResult func delete(_ id: String) -> Bool { guard let writer = resolveWriter() else { return false } return (try? writer.write { db in try db.execute(sql: "DELETE FROM favorite_media WHERE favorite_id = ?", arguments: [id]) - try db.execute(sql: "DELETE FROM favorites WHERE id = ?", arguments: [id]) - return db.changesCount > 0 + return try deleteForSync( + db, + target: .favorite, + boardId: nil, + key: id, + whereClause: "id = ?", + keys: [id] + ) }) ?? false } diff --git a/modules/vescape-core/ios/telemetry/SyncActionLog.swift b/modules/vescape-core/ios/telemetry/SyncActionLog.swift new file mode 100644 index 000000000..eda2d2bcd --- /dev/null +++ b/modules/vescape-core/ios/telemetry/SyncActionLog.swift @@ -0,0 +1,188 @@ +import Foundation +import GRDB + +/// What a Sync Action can name — and, by omission, what it cannot. +/// +/// Every case is configuration or current state a Rider edits directly. Ride History is absent on +/// purpose: Telemetry Samples, markers, minute buckets, exclusion ranges and diagnostic events are +/// pruned on a retention rule, and an action naming one of those would make the server delete +/// exactly the rides the backup exists to preserve. Leaving them unnameable makes that boundary +/// structural rather than a rule someone has to remember (server ADR-0004). +/// +/// `table` is the local table the case removes from, so a test can assert no retained table is ever +/// given a case. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `DeleteTarget` +/// @parity /modules/vescape-core/src/index.ts `DeleteTarget` +enum DeleteTarget: String, CaseIterable { + case appSetting + case board + case boardSetting + case boardWarning + case alert + case tuneProfile + case privacyZone + case favorite + + var table: String { + switch self { + case .appSetting: return "app_settings" + case .board: return "boards" + case .boardSetting: return "board_settings" + case .boardWarning: return "board_warnings" + case .alert: return "alerts" + case .tuneProfile: return "tune_profiles" + case .privacyZone: return "privacy_zones" + case .favorite: return "favorites" + } + } +} + +/// The only Sync Action type today. Named rather than implied so a later intent needs no second log. +internal let syncActionTypeDelete = "delete" + +/// `sync_sequences` key holding the highest action cursor the server has accepted. +internal let syncActionsUploadedCursor = "sync_actions_uploaded" + +/// The Sync Action log: an append-only record that something was semantically removed. A deleted row +/// cannot carry a Change Timestamp saying it is gone, so this log is the only signal the server can +/// apply the same durable state transition from. +/// +/// Its cursor is `id` — `AUTOINCREMENT`, which SQLite guarantees monotonic and never reused — so the +/// log needs no `sync_seq` of its own. Rows are transport state, not durable truth: they are pruned +/// once the server has accepted them. +/// +/// Idempotent, and called both from the migration that introduced it and from the store-level +/// `createTables` seams tests build their schema from. No database trigger writes here — intent +/// cannot be inferred from SQL alone. +/// +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `SyncActionEntity` +internal func createSyncActionsTable(_ db: Database) throws { + try db.execute( + sql: """ + CREATE TABLE IF NOT EXISTS sync_actions ( + id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, + type TEXT NOT NULL, + target TEXT NOT NULL, + board_id TEXT, + key TEXT NOT NULL, + deleted_at INTEGER NOT NULL + ) + """ + ) + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_sync_actions_target ON sync_actions(target)") +} + +/// Record that [target] identified by `boardId`/`key` was semantically removed. +/// +/// `rowStamp` is the removed row's own change timestamp, read before the delete: the action is +/// stamped `max(now, rowStamp)` so a rewound device clock cannot produce an action the server reads +/// as older than the row it names — that action would be dropped as a no-op, and the phone could not +/// self-heal by re-sending, because the row is gone. +/// +/// A nil `rowStamp` means there was no row to remove, so no intent to record either. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `appendDeleteAction` +internal func appendDeleteAction( + _ db: Database, + target: DeleteTarget, + boardId: String?, + key: String, + rowStamp: Int64?, + now: Int64 = telemetryNowMs() +) throws { + guard let rowStamp else { return } + try db.execute( + sql: """ + INSERT INTO sync_actions (type, target, board_id, key, deleted_at) + VALUES (?, ?, ?, ?, ?) + """, + arguments: [syncActionTypeDelete, target.rawValue, boardId, key, max(now, rowStamp)] + ) +} + +/// The one semantic-removal primitive: read the row's change timestamp, append its action, delete +/// the row — all inside the caller's transaction, so the action and the delete commit together or +/// not at all. +/// +/// `stampColumn` is the row's own change clock: `updated_at` everywhere except Board Warnings, whose +/// `last_detected_at` is what their `updated_at` was written from. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `appendDeleteAction` +@discardableResult +internal func deleteForSync( + _ db: Database, + target: DeleteTarget, + boardId: String?, + key: String, + whereClause: String, + keys: StatementArguments, + stampColumn: String = "updated_at", + now: Int64 = telemetryNowMs() +) throws -> Bool { + let stamp = try Int64.fetchOne( + db, + sql: "SELECT \(stampColumn) FROM \(target.table) WHERE \(whereClause)", + arguments: keys + ) + try appendDeleteAction(db, target: target, boardId: boardId, key: key, rowStamp: stamp, now: now) + try db.execute(sql: "DELETE FROM \(target.table) WHERE \(whereClause)", arguments: keys) + return db.changesCount > 0 +} + +/// One Sync Action as it leaves the phone. The uploader (#284) owns the batching; this is the read +/// shape it pages through. +struct SyncAction { + let id: Int64 + let type: String + let target: String + let boardId: String? + let key: String + let deletedAt: Int64 +} + +/// The next page of actions to upload, in cursor order. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `getSyncActionsAfter` +internal func syncActionsAfter(_ db: Database, _ afterId: Int64, limit: Int) throws -> [SyncAction] { + try Row.fetchAll( + db, + sql: "SELECT * FROM sync_actions WHERE id > ? ORDER BY id ASC LIMIT ?", + arguments: [afterId, limit] + ).map { row in + SyncAction( + id: row["id"], + type: row["type"], + target: row["target"], + boardId: row["board_id"], + key: row["key"], + deletedAt: row["deleted_at"] + ) + } +} + +/// Checkpoint the highest action cursor the server has accepted, in its own transaction, committed +/// before `pruneUploadedSyncActions` runs: a crash between the two leaves rows that will be sent +/// again — harmless, since applying an action twice is a no-op — whereas pruning first would drop an +/// action nobody has accepted. Never moves backwards. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `commitSyncActionCursor` +internal func commitSyncActionCursor(_ db: Database, throughId: Int64) throws { + try db.execute( + sql: """ + INSERT OR REPLACE INTO sync_sequences (name, last_value) + VALUES (?, MAX(?, COALESCE((SELECT last_value FROM sync_sequences WHERE name = ?), 0))) + """, + arguments: [syncActionsUploadedCursor, throughId, syncActionsUploadedCursor] + ) +} + +/// Drop what the server has already accepted. Gated on the committed cursor rather than a caller's +/// number, so pruning structurally cannot outrun the checkpoint. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `pruneUploadedSyncActions` +@discardableResult +internal func pruneUploadedSyncActions(_ db: Database) throws -> Int { + guard let accepted = try Int64.fetchOne( + db, + sql: "SELECT last_value FROM sync_sequences WHERE name = ?", + arguments: [syncActionsUploadedCursor] + ) else { return 0 } + try db.execute(sql: "DELETE FROM sync_actions WHERE id <= ?", arguments: [accepted]) + return db.changesCount +} diff --git a/modules/vescape-core/ios/telemetry/SyncActionLogTests.swift b/modules/vescape-core/ios/telemetry/SyncActionLogTests.swift new file mode 100644 index 000000000..33bda926a --- /dev/null +++ b/modules/vescape-core/ios/telemetry/SyncActionLogTests.swift @@ -0,0 +1,385 @@ +import XCTest +import GRDB +@testable import VescapeCore + +/// The Sync Action log (#282): an append-only record of semantic removals, which no surviving row +/// can express. A deleted row cannot carry a Change Timestamp saying it is gone. +/// +/// Runs the real migrator and the real repositories/stores against an in-memory database. The +/// Android peer asserts the same classification against the DAO source, because Room keeps its SQL +/// out of reach of a JVM unit test. +/// +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncActionLogTest.kt +final class SyncActionLogTests: XCTestCase { + private var queue: DatabaseQueue! + private var repo: AppDataRepository! + + override func setUpWithError() throws { + queue = try DatabaseQueue() + try TelemetryDatabase.migrator.migrate(queue) + repo = AppDataRepository.forTesting(dbWriter: queue) + } + + override func tearDownWithError() throws { + repo = nil + queue = nil + } + + // MARK: Helpers + + private func actions() throws -> [SyncAction] { + try queue.read { db in try syncActionsAfter(db, 0, limit: 100) } + } + + private func seedBoard(_ id: String = "board-1") { + repo.upsertBoard([ + "id": id, + "name": "ADV", + "createdAt": Int64(1000), + "description": "trail board", + "link": ["bleId": "AA:BB", "transport": "direct"] as [String: Any?], + ]) + } + + private func seedFavorite(_ id: String = "fav-1") { + FavoriteStore(dbWriter: queue).insert( + Favorite( + id: id, + boardId: "board-1", + name: "commute", + startMs: 1000, + endMs: 2000, + createdAtMs: 1000, + updatedAtMs: 1000, + summary: FavoriteSummary() + ) + ) + } + + private func seedWarning(kind: String, lastDetectedAt: Int64) { + BoardWarningStore(dbWriter: queue).upsert( + BoardWarning( + boardId: "board-1", + kind: kind, + severity: "warn", + firstDetectedAtMs: 500, + lastDetectedAtMs: lastDetectedAt, + payloadJson: "{}" + ) + ) + } + + // MARK: The seven Rider-facing removals + + func testDeletingAnAlertRuleEmitsOneAction() throws { + seedBoard() + repo.upsertAlertRule(["boardId": "board-1", "id": "rule-1", "controlId": "speed"]) + + repo.deleteAlertRule("board-1", "rule-1") + + let actions = try self.actions() + XCTAssertEqual(actions.count, 1) + XCTAssertEqual(actions.first?.type, "delete") + XCTAssertEqual(actions.first?.target, DeleteTarget.alert.rawValue) + XCTAssertEqual(actions.first?.boardId, "board-1") + XCTAssertEqual(actions.first?.key, "rule-1") + } + + func testDeletingAPrivacyZoneEmitsOneAction() throws { + repo.upsertPrivacyZone([ + "id": "zone-1", "name": "home", "centerLatitude": 52.0, "centerLongitude": 21.0, + "radiusMeters": Int64(100), + ]) + + repo.deletePrivacyZone("zone-1") + + XCTAssertEqual(try actions().map { ($0.target, $0.key) }.map { "\($0.0):\($0.1)" }, ["privacyZone:zone-1"]) + } + + func testResettingAnAppSettingToItsDefaultEmitsAnAction() throws { + repo.updateSetting("telemetryPollRateHz", rawValue: 50) + + repo.updateSetting("telemetryPollRateHz", rawValue: nil) + + let actions = try self.actions() + XCTAssertEqual(actions.count, 1) + XCTAssertEqual(actions.first?.target, DeleteTarget.appSetting.rawValue) + XCTAssertNil(actions.first?.boardId, "an app setting is not Board-owned") + XCTAssertEqual(actions.first?.key, "telemetryPollRateHz") + } + + /// A phone-local key never reaches the server, so its removal has nothing to tell the server about. + func testRemovingAPhoneLocalSettingEmitsNoAction() throws { + repo.updateSetting("selectedBoardId", rawValue: "board-1") + + repo.updateSetting("selectedBoardId", rawValue: nil) + + XCTAssertEqual(try actions().count, 0) + XCTAssertNil(try queue.read { db in + try String.fetchOne(db, sql: "SELECT value_json FROM app_settings WHERE key = 'selectedBoardId'") + }) + } + + func testClearingLegalPolicyEmitsAnAppSettingAction() throws { + repo.updateLegalPolicy(jurisdictionCode: "PL") + + repo.updateLegalPolicy(jurisdictionCode: nil) + + XCTAssertEqual(try actions().map(\.key), ["legalPolicy"]) + XCTAssertEqual(try actions().map(\.target), [DeleteTarget.appSetting.rawValue]) + } + + /// A Board edit that drops a key is the Rider clearing that setting. + func testDroppingABoardSettingKeyEmitsAnAction() throws { + seedBoard() + + repo.upsertBoard([ + "id": "board-1", + "name": "ADV", + "createdAt": Int64(1000), + "description": "", + "link": ["bleId": "AA:BB", "transport": "direct"] as [String: Any?], + ]) + + let actions = try self.actions() + XCTAssertEqual(actions.map(\.target), [DeleteTarget.boardSetting.rawValue]) + XCTAssertEqual(actions.first?.boardId, "board-1") + XCTAssertEqual(actions.first?.key, "description") + } + + func testDeletingATuneProfileEmitsOneActionAndItsHistoryNone() throws { + let store = TuneProfileStore(dbWriter: queue) + _ = try store.createProfile( + boardId: "board-1", name: "keep", icon: "sliders-horizontal", color: "purple", + fields: [:], refloatBaseVersion: "1.3.0" + ) + let doomed = try store.createProfile( + boardId: "board-1", name: "drop", icon: "sliders-horizontal", color: "purple", + fields: [:], refloatBaseVersion: "1.3.0" + ) + let id = doomed["id"] as! String + + try store.deleteProfile(profileId: id) + + let actions = try self.actions() + XCTAssertEqual(actions.count, 1, "Tune History is parent-covered") + XCTAssertEqual(actions.first?.target, DeleteTarget.tuneProfile.rawValue) + XCTAssertEqual(actions.first?.key, id) + } + + func testDeletingAFavoriteEmitsOneActionAndItsMediaNone() throws { + seedFavorite() + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO favorite_media (id, favorite_id, captured_at, mime_type, media_kind, byte_count, content_hash, created_at) + VALUES ('media-1', 'fav-1', NULL, 'image/jpeg', 'photo', 10, 'hash', 1000) + """ + ) + } + + XCTAssertTrue(FavoriteStore(dbWriter: queue).delete("fav-1")) + + let actions = try self.actions() + XCTAssertEqual(actions.count, 1, "the Favorite's action covers its manifest rows") + XCTAssertEqual(actions.first?.target, DeleteTarget.favorite.rawValue) + XCTAssertEqual(actions.first?.key, "fav-1") + } + + /// An automatic clear after a clean detector evaluation is still a durable state transition. + func testClearingABoardWarningEmitsAnActionStampedFromItsDetection() throws { + seedWarning(kind: "cell-spread", lastDetectedAt: 9_000_000_000_000) + + XCTAssertTrue(BoardWarningStore(dbWriter: queue).delete("board-1", "cell-spread")) + + let actions = try self.actions() + XCTAssertEqual(actions.count, 1) + XCTAssertEqual(actions.first?.target, DeleteTarget.boardWarning.rawValue) + XCTAssertEqual(actions.first?.boardId, "board-1") + XCTAssertEqual(actions.first?.key, "cell-spread") + XCTAssertEqual( + actions.first?.deletedAt, 9_000_000_000_000, + "a detection in the future outranks the wall clock, or the server drops the action" + ) + } + + /// Clearing every warning on a Board is one action per row — each row is its own current state. + func testClearingAllWarningsForABoardEmitsOneActionPerRow() throws { + seedWarning(kind: "cell-spread", lastDetectedAt: 1_000) + seedWarning(kind: "footpad-disabled", lastDetectedAt: 2_000) + + XCTAssertTrue(BoardWarningStore(dbWriter: queue).deleteForBoard("board-1")) + + XCTAssertEqual(try actions().map(\.key).sorted(), ["cell-spread", "footpad-disabled"]) + } + + // MARK: The Board tombstone + + func testDeletingABoardEmitsOneActionAndNoneForItsCascade() throws { + seedBoard() + repo.upsertAlertRule(["boardId": "board-1", "id": "rule-1", "controlId": "speed"]) + seedWarning(kind: "cell-spread", lastDetectedAt: 1_000) + + repo.deleteBoard("board-1") + + let actions = try self.actions() + XCTAssertEqual(actions.map(\.target), [DeleteTarget.board.rawValue]) + XCTAssertEqual(actions.first?.key, "board-1") + XCTAssertNil(actions.first?.boardId, "a Board is Account-owned; it names itself in `key`") + + let row = try queue.read { db in + try Row.fetchOne(db, sql: "SELECT deleted_at, updated_at FROM boards WHERE id = 'board-1'") + } + XCTAssertEqual(row?["deleted_at"] as Int64?, actions.first?.deletedAt) + XCTAssertEqual(row?["updated_at"] as Int64?, actions.first?.deletedAt) + } + + // MARK: Stamping + + /// A rewound clock would otherwise stamp the action below the row the server already holds, and + /// the action would be dropped as a no-op with no row left to re-send. + func testDeletionStampNeverFallsBelowTheRemovedRow() throws { + let future: Int64 = 9_000_000_000_000 + try queue.write { db in + try db.execute( + sql: """ + INSERT INTO privacy_zones + (id, preset, name, enabled, center_latitude_e7, center_longitude_e7, radius_meters, created_at, updated_at, sync_seq) + VALUES ('zone-1', 'custom', 'home', 1, 0, 0, 100, 0, ?, 1) + """, + arguments: [future] + ) + } + + repo.deletePrivacyZone("zone-1") + + XCTAssertEqual(try actions().first?.deletedAt, future) + } + + /// Nothing removed, nothing to say. + func testDeletingAMissingRowEmitsNoAction() throws { + repo.deletePrivacyZone("does-not-exist") + repo.deleteAlertRule("board-1", "missing") + + XCTAssertEqual(try actions().count, 0) + } + + // MARK: Retention + + /// The whole reason Ride History has no target: a retention sweep must not reach this log. + func testRetentionWritesNoActions() throws { + try queue.write { db in + try db.execute( + sql: "INSERT INTO telemetry_markers (occurred_at_ms, elapsed_realtime_ms, type) VALUES (1, 1, 'start')" + ) + try db.execute(sql: "DELETE FROM telemetry_markers WHERE occurred_at_ms < 100") + } + + XCTAssertEqual(try actions().count, 0) + } + + // MARK: Cursor and pruning + + /// The accepted cursor commits first; pruning reads it back rather than trusting a caller. A crash + /// between the two re-sends an action, which is a no-op — pruning first would lose one. + func testPruningOnlyRemovesWhatTheCursorHasAccepted() throws { + seedFavorite("fav-1") + seedFavorite("fav-2") + FavoriteStore(dbWriter: queue).delete("fav-1") + FavoriteStore(dbWriter: queue).delete("fav-2") + let logged = try actions() + XCTAssertEqual(logged.count, 2) + + XCTAssertEqual(try queue.write { db in try pruneUploadedSyncActions(db) }, 0) + XCTAssertEqual(try actions().count, 2, "nothing is accepted yet") + + try queue.write { db in try commitSyncActionCursor(db, throughId: logged[0].id) } + XCTAssertEqual(try queue.write { db in try pruneUploadedSyncActions(db) }, 1) + XCTAssertEqual(try actions().map(\.id), [logged[1].id]) + } + + func testTheAcceptedCursorNeverMovesBackwards() throws { + try queue.write { db in + try commitSyncActionCursor(db, throughId: 10) + try commitSyncActionCursor(db, throughId: 3) + } + + let cursor = try queue.read { db in + try Int64.fetchOne( + db, + sql: "SELECT last_value FROM sync_sequences WHERE name = ?", + arguments: [syncActionsUploadedCursor] + ) + } + XCTAssertEqual(cursor, 10) + } + + // MARK: Schema + + func testTheLogIsKeyedOnAnAutoincrementCursor() throws { + let sql = try queue.read { db in + try String.fetchOne(db, sql: "SELECT sql FROM sqlite_master WHERE name = 'sync_actions'") + } + XCTAssertTrue(sql?.contains("AUTOINCREMENT") ?? false, sql ?? "missing sync_actions") + + let triggers = try queue.read { db in + try String.fetchAll(db, sql: "SELECT name FROM sqlite_master WHERE type = 'trigger'") + } + XCTAssertEqual(triggers, [], "intent cannot be inferred from SQL — no trigger writes the log") + } + + func testMigrationIsANoOpOnReRun() throws { + XCTAssertNoThrow(try TelemetryDatabase.migrator.migrate(queue)) + } + + /// The retention boundary, made structural: a target can only name configuration or current state. + /// Mirrors the server's `DELETE_ACTION_TARGETS` test. + func testNoRetainedTableCanBeNamedByADeleteTarget() { + let retained: Set = [ + "telemetry_frames", + "telemetry_markers", + "telemetry_minute_buckets", + "metric_exclusion_ranges", + "diagnostic_events", + "tune_history_entries", + "favorite_media", + "sync_actions", + "sync_sequences", + ] + let named = Set(DeleteTarget.allCases.map(\.table)) + + XCTAssertEqual(named.intersection(retained), []) + XCTAssertEqual(named.count, DeleteTarget.allCases.count) + } + + /// Every raw `DELETE FROM` against a syncable table has to sit in a file that is allowed to write + /// one — the delete-owning stores, where each statement is either parent-covered or maintenance. + /// A new raw delete elsewhere fails here rather than silently skipping the log. + func testEverySyncableDeleteLivesInADeleteOwningStore() throws { + let root = URL(fileURLWithPath: #filePath).deletingLastPathComponent().deletingLastPathComponent() + let owners: Set = [ + "AppDataRepository.swift", + "TuneProfileStore.swift", + "FavoriteStore.swift", + "BoardWarningStore.swift", + "SyncActionLog.swift", + // Schema migrations are maintenance: they rewrite what a table holds, never Rider intent. + "TelemetryDatabase.swift", + ] + let syncable = Set(DeleteTarget.allCases.map(\.table)) + + let files = FileManager.default.enumerator(at: root, includingPropertiesForKeys: nil)? + .compactMap { $0 as? URL } + .filter { $0.pathExtension == "swift" && !$0.lastPathComponent.hasSuffix("Tests.swift") } ?? [] + + for file in files where !owners.contains(file.lastPathComponent) { + let source = try String(contentsOf: file, encoding: .utf8) + for table in syncable { + XCTAssertFalse( + source.contains("DELETE FROM \(table)"), + "\(file.lastPathComponent) deletes from \(table) outside a Sync Action-owning store" + ) + } + } + } +} diff --git a/modules/vescape-core/ios/telemetry/SyncCursorMigrationTests.swift b/modules/vescape-core/ios/telemetry/SyncCursorMigrationTests.swift new file mode 100644 index 000000000..af92b0e41 --- /dev/null +++ b/modules/vescape-core/ios/telemetry/SyncCursorMigrationTests.swift @@ -0,0 +1,563 @@ +import XCTest +import GRDB +@testable import VescapeCore + +/// Incremental-sync cursors: the `v43_sync_cursors` migration adds `updated_at` to `boards`, +/// `alerts` and `telemetry_minute_buckets`, backfills it from each table's best evidence of last +/// change, and indexes it. `v44_sync_seq` then splits the two jobs that column was doing — `sync_seq` +/// carries the Sync Cursor, `updated_at` stays the last-write-wins timestamp. Every write path has to +/// move both. +/// +/// Runs the real migrator against an in-memory database, stopping at v27 to seed pre-migration rows. +/// @parity /modules/vescape-core/android/src/test/java/expo/modules/vescapecore/telemetry/SyncCursorMigrationTest.kt +final class SyncCursorMigrationTests: XCTestCase { + private var queue: DatabaseQueue! + + override func setUpWithError() throws { + queue = try DatabaseQueue() + } + + override func tearDownWithError() throws { + queue = nil + } + + /// Migrate up to (and including) the last pre-cursor migration, so the seeded rows look exactly + /// like an installed app's rows before it upgrades. + private func migrateToV27() throws { + try TelemetryDatabase.migrator.migrate(queue, upTo: "v27_alert_board_id") + } + + private func migrateToLatest() throws { + try TelemetryDatabase.migrator.migrate(queue) + } + + private func columnNames(_ table: String) throws -> [String] { + try queue.read { db in try db.columns(in: table).map(\.name) } + } + + private func indexNames(_ table: String) throws -> [String] { + try queue.read { db in try db.indexes(on: table).map(\.name) } + } + + func testV27HasNoCursorColumns() throws { + try migrateToV27() + + for table in ["boards", "alerts", "telemetry_minute_buckets"] { + XCTAssertFalse(try columnNames(table).contains("updated_at"), "\(table) already has a cursor") + } + } + + func testMigrationAddsCursorColumnAndIndexToEverySyncedTable() throws { + try migrateToLatest() + + for table in ["boards", "alerts", "telemetry_minute_buckets"] { + XCTAssertTrue(try columnNames(table).contains("updated_at"), "\(table) is missing updated_at") + XCTAssertTrue( + try indexNames(table).contains("index_\(table)_updated_at"), + "\(table) is missing its updated_at index" + ) + } + } + + /// The backfill is the whole point of shipping this as a migration rather than a plain column add: + /// a row left at the `DEFAULT 0` would report epoch zero to the server and get re-synced forever. + func testBackfillCarriesExistingRowsInsteadOfLeavingThemAtZero() throws { + try migrateToV27() + try queue.write { db in + try db.execute( + sql: "INSERT INTO boards (id, name, ble_id, transport, created_at) VALUES (?, ?, NULL, NULL, ?)", + arguments: ["board-1", "ADV", 1_000] + ) + try db.execute( + sql: """ + INSERT INTO alerts (board_id, id, control_id, threshold, threshold_max, enabled, sound_type, created_at, source) + VALUES (?, ?, ?, ?, NULL, 1, ?, ?, NULL) + """, + arguments: ["board-1", "rule-1", "duty", 70.0, "default", 2_000] + ) + try db.execute( + sql: """ + INSERT INTO telemetry_minute_buckets ( + bucket_start_ms, device_id, device_name, sample_count, first_sample_at_ms, last_sample_at_ms, + sum_abs_speed_centi_kmh, moving_speed_sample_count, sum_moving_abs_speed_centi_kmh, + max_abs_speed_centi_kmh, min_battery_voltage_mv, max_motor_current_abs_ma, + max_battery_current_abs_ma, battery_used_wh_milli, battery_regen_wh_milli, + max_duty_abs_permille, first_odometer_cm, last_odometer_cm, + gps_point_count, precise_gps_point_count, gps_distance_cm, max_gps_speed_centi_mps + ) VALUES (?, ?, NULL, 1, ?, ?, 0, 0, 0, 0, NULL, 0, 0, 0, 0, 0, NULL, NULL, 0, 0, 0, NULL) + """, + arguments: [60_000, "board-1", 60_000, 3_000] + ) + } + + try migrateToLatest() + + try queue.read { db in + XCTAssertEqual(try Int64.fetchOne(db, sql: "SELECT updated_at FROM boards"), 1_000) + XCTAssertEqual(try Int64.fetchOne(db, sql: "SELECT updated_at FROM alerts"), 2_000) + // Buckets have no `created_at`; `last_sample_at_ms` is the closest record of last change. + XCTAssertEqual( + try Int64.fetchOne(db, sql: "SELECT updated_at FROM telemetry_minute_buckets"), + 3_000 + ) + } + } + + // MARK: - Write paths + + private func makeRepository() throws -> AppDataRepository { + try migrateToLatest() + return AppDataRepository.forTesting(dbWriter: queue) + } + + private func alertCursor() throws -> Int64? { + try queue.read { db in try Int64.fetchOne(db, sql: "SELECT updated_at FROM alerts") } + } + + func testUpsertsStampTheCursor() throws { + let repo = try makeRepository() + + repo.upsertBoard(["id": "board-1", "name": "ADV", "createdAt": 1_000]) + repo.upsertAlertRule([ + "boardId": "board-1", "id": "rule-1", "controlId": "duty", "threshold": 70.0, + "enabled": true, "createdAt": 1_000, + ]) + + let boardCursor = try queue.read { db in + try Int64.fetchOne(db, sql: "SELECT updated_at FROM boards") + } + // Stamped from the device clock, not from the bridge-supplied `createdAt`. + XCTAssertGreaterThan(boardCursor ?? 0, 1_000) + XCTAssertGreaterThan(try alertCursor() ?? 0, 1_000) + } + + /// The regression this whole change exists to prevent. `setAlertRuleEnabled` is a targeted UPDATE + /// rather than a whole-row rewrite, so it is the one write path that can silently skip the cursor + /// — toggling an alert would then never reach the server. + func testSetAlertRuleEnabledBumpsTheCursor() throws { + let repo = try makeRepository() + repo.upsertAlertRule([ + "boardId": "board-1", "id": "rule-1", "controlId": "duty", "threshold": 70.0, + "enabled": true, "createdAt": 1_000, + ]) + let before = try XCTUnwrap(alertCursor()) + + // The cursor is millisecond-resolution wall clock, so force a tick we can observe. + Thread.sleep(forTimeInterval: 0.005) + repo.setAlertRuleEnabled("board-1", "rule-1", false) + + let after = try XCTUnwrap(alertCursor()) + XCTAssertGreaterThan(after, before) + XCTAssertEqual( + try queue.read { db in try Int64.fetchOne(db, sql: "SELECT enabled FROM alerts") }, + 0 + ) + } + + /// Buckets are append-and-merge targets: a later append has to move the cursor even though it + /// leaves most aggregate columns folded into the existing row. + func testBucketUpsertAdvancesTheCursorOnMerge() throws { + try migrateToLatest() + var bucket = TelemetryBucket(bucketStartMs: 60_000, boardId: "board-1") + bucket.firstSampleAtMs = 60_000 + bucket.lastSampleAtMs = 60_500 + bucket.sampleCount = 1 + + try queue.write { db in try upsertBucket(db, bucket, now: 1_000) } + try queue.write { db in try upsertBucket(db, bucket, now: 5_000) } + + let (cursor, samples) = try queue.read { db -> (Int64?, Int64?) in + ( + try Int64.fetchOne(db, sql: "SELECT updated_at FROM telemetry_minute_buckets"), + try Int64.fetchOne(db, sql: "SELECT sample_count FROM telemetry_minute_buckets") + ) + } + XCTAssertEqual(cursor, 5_000) + // Sanity: the second write merged into the same row rather than inserting a new one. + XCTAssertEqual(samples, 2) + } + + /// A device clock that steps backwards must not merely freeze the stamp: the server guards this + /// table with `stored.updated_at < EXCLUDED.updated_at` like every other mutable table, so a + /// frozen stamp is scanned, sent, and silently dropped. The merge ratchets strictly past it (#281). + func testBucketCursorRatchetsPastAStampTheClockCannotBeat() throws { + try migrateToLatest() + var bucket = TelemetryBucket(bucketStartMs: 60_000, boardId: "board-1") + bucket.firstSampleAtMs = 60_000 + bucket.lastSampleAtMs = 60_500 + + try queue.write { db in try upsertBucket(db, bucket, now: 5_000) } + try queue.write { db in try upsertBucket(db, bucket, now: 1_000) } + + XCTAssertEqual( + try queue.read { db in try Int64.fetchOne(db, sql: "SELECT updated_at FROM telemetry_minute_buckets") }, + 5_001 + ) + } + + // MARK: - Sync Cursor sequence (#275) + + private func syncSeq(_ table: String) throws -> Int64? { + try queue.read { db in try Int64.fetchOne(db, sql: "SELECT sync_seq FROM \(table)") } + } + + private func counter(_ name: String) throws -> Int64? { + try queue.read { db in + try Int64.fetchOne(db, sql: "SELECT last_value FROM sync_sequences WHERE name = ?", arguments: [name]) + } + } + + func testMigrationAddsSyncSeqColumnAndIndexToEverySyncedTable() throws { + try migrateToLatest() + + for table in syncSeqTables { + XCTAssertTrue(try columnNames(table).contains("sync_seq"), "\(table) is missing sync_seq") + XCTAssertTrue( + try indexNames(table).contains("index_\(table)_sync_seq"), + "\(table) is missing its sync_seq index" + ) + } + } + + /// Pre-29 rows need distinct, increasing positions, and the counter has to resume above all of + /// them — otherwise the first writes after upgrade reuse numbers the scan would order wrongly. + func testMigrationBackfillsSyncSeqAndResumesTheCounterAboveIt() throws { + try migrateToV27() + try queue.write { db in + for (index, id) in ["board-1", "board-2", "board-3"].enumerated() { + try db.execute( + sql: "INSERT INTO boards (id, name, ble_id, transport, created_at) VALUES (?, ?, NULL, NULL, ?)", + arguments: [id, "ADV", 1_000 + index] + ) + } + } + + try migrateToLatest() + + let seqs = try queue.read { db in + try Int64.fetchAll(db, sql: "SELECT sync_seq FROM boards ORDER BY sync_seq") + } + XCTAssertEqual(seqs.count, 3) + XCTAssertEqual(Set(seqs).count, 3, "backfilled positions collide") + XCTAssertEqual(try counter(syncSeqBoards), seqs.max()) + } + + func testUpsertsAdvanceTheSyncSeq() throws { + let repo = try makeRepository() + + repo.upsertBoard(["id": "board-1", "name": "ADV", "createdAt": 1_000]) + let first = try XCTUnwrap(syncSeq(syncSeqBoards)) + repo.upsertBoard(["id": "board-1", "name": "Renamed", "createdAt": 1_000]) + + XCTAssertGreaterThan(try XCTUnwrap(syncSeq(syncSeqBoards)), first) + } + + /// The reason the counter lives in its own table instead of being derived as `MAX(sync_seq) + 1`: + /// deleting the highest row would hand its number out again, and the reused row would land below a + /// cursor the phone had already advanced past. + func testSyncSeqIsNotReusedAfterTheHighestRowIsDeleted() throws { + let repo = try makeRepository() + repo.upsertBoard(["id": "board-1", "name": "ADV", "createdAt": 1_000]) + let deleted = try XCTUnwrap(syncSeq(syncSeqBoards)) + + repo.deleteBoard("board-1") + repo.upsertBoard(["id": "board-2", "name": "GT", "createdAt": 1_000]) + + XCTAssertGreaterThan(try XCTUnwrap(syncSeq(syncSeqBoards)), deleted) + } + + func testBucketMergeAdvancesTheSyncSeq() throws { + try migrateToLatest() + var bucket = TelemetryBucket(bucketStartMs: 60_000, boardId: "board-1") + bucket.firstSampleAtMs = 60_000 + bucket.lastSampleAtMs = 60_500 + + try queue.write { db in try upsertBucket(db, bucket, now: 1_000) } + let first = try XCTUnwrap(syncSeq(syncSeqMinuteBuckets)) + // A merge rewrites a row the scan may already have passed, so it needs a fresh position too. + try queue.write { db in try upsertBucket(db, bucket, now: 2_000) } + + XCTAssertGreaterThan(try XCTUnwrap(syncSeq(syncSeqMinuteBuckets)), first) + } + + // MARK: - Last-write-wins ratchet (#275) + + func testRatchetStepsPastAStampTheClockCannotBeat() throws { + XCTAssertEqual(ratchetUpdatedAt(nil, 1_000), 1_000) + // Clock ahead of the stored row: truthful wall clock, no inflation. + XCTAssertEqual(ratchetUpdatedAt(1_000, 5_000), 5_000) + // Clock rewound below it: strictly above, so the server's `stored < incoming` guard accepts it. + XCTAssertEqual(ratchetUpdatedAt(5_000, 1_000), 5_001) + XCTAssertEqual(ratchetUpdatedAt(5_000, 5_000), 5_001) + } + + /// A rewound clock must not leave the row stamped at or below the copy the server already holds — + /// the upsert guard there keeps the stored row unless the incoming stamp is strictly newer, so a + /// frozen stamp is a silently dropped edit. + func testBoardUpsertNeverStampsAtOrBelowTheStoredValue() throws { + let repo = try makeRepository() + repo.upsertBoard(["id": "board-1", "name": "ADV", "createdAt": 1_000]) + // Stand in for a rewind by putting the stored row far beyond any clock the write can read. + let ahead = Int64(Date().timeIntervalSince1970 * 1000) + 3_600_000 + try queue.write { db in + try db.execute(sql: "UPDATE boards SET updated_at = ?", arguments: [ahead]) + } + + repo.upsertBoard(["id": "board-1", "name": "Renamed", "createdAt": 1_000]) + + XCTAssertEqual(try queue.read { db in try Int64.fetchOne(db, sql: "SELECT updated_at FROM boards") }, ahead + 1) + } + + func testSetAlertRuleEnabledNeverStampsAtOrBelowTheStoredValue() throws { + let repo = try makeRepository() + repo.upsertAlertRule([ + "boardId": "board-1", "id": "rule-1", "controlId": "duty", "threshold": 70.0, + "enabled": true, "createdAt": 1_000, + ]) + let ahead = Int64(Date().timeIntervalSince1970 * 1000) + 3_600_000 + try queue.write { db in + try db.execute(sql: "UPDATE alerts SET updated_at = ?", arguments: [ahead]) + } + let seqBefore = try XCTUnwrap(syncSeq(syncSeqAlerts)) + + repo.setAlertRuleEnabled("board-1", "rule-1", false) + + XCTAssertEqual(try alertCursor(), ahead + 1) + XCTAssertGreaterThan(try XCTUnwrap(syncSeq(syncSeqAlerts)), seqBefore) + } + + // MARK: - The six remaining mutable tables (#281) + + private func rowSyncSeq(_ table: String, _ whereClause: String, _ args: StatementArguments) throws -> Int64? { + try queue.read { db in + try Int64.fetchOne(db, sql: "SELECT sync_seq FROM \(table) WHERE \(whereClause)", arguments: args) + } + } + + func testRemainingMutableTablesCarryACursorAndItsIndex() throws { + try migrateToLatest() + + for table in syncSeqTablesV45 { + XCTAssertTrue(try columnNames(table).contains("sync_seq"), "\(table) is missing sync_seq") + XCTAssertTrue( + try indexNames(table).contains("index_\(table)_sync_seq"), + "\(table) is missing its sync_seq index" + ) + } + XCTAssertTrue(try columnNames("board_warnings").contains("updated_at")) + } + + /// Append-only tables key on `INTEGER PRIMARY KEY AUTOINCREMENT`, which SQLite guarantees + /// monotonic and never reused — that key already is their cursor, so a second one would be dead + /// weight the write paths would have to keep in step. + func testAppendOnlyTablesGainNoCursor() throws { + try migrateToLatest() + + for table in ["telemetry_frames", "telemetry_markers", "diagnostic_events", + "metric_exclusion_ranges", "tune_history_entries"] { + XCTAssertFalse(try columnNames(table).contains("sync_seq"), "\(table) should not carry sync_seq") + } + } + + func testMigrationBackfillsRemainingTablesWithDistinctPositions() throws { + try TelemetryDatabase.migrator.migrate(queue, upTo: "v42_telemetry_board_id") + try queue.write { db in + for (index, id) in ["zone-1", "zone-2", "zone-3"].enumerated() { + try db.execute( + sql: """ + INSERT INTO privacy_zones + (id, preset, name, enabled, center_latitude_e7, center_longitude_e7, radius_meters, created_at, updated_at) + VALUES (?, 'home', 'Home', 1, 0, 0, 100, ?, ?) + """, + arguments: [id, 1_000 + index, 1_000 + index] + ) + } + } + + try migrateToLatest() + + let seqs = try queue.read { db in + try Int64.fetchAll(db, sql: "SELECT sync_seq FROM privacy_zones ORDER BY sync_seq") + } + XCTAssertEqual(Set(seqs).count, 3, "backfilled positions collide") + XCTAssertFalse(seqs.contains(0), "a backfilled row sits at the seed value") + XCTAssertEqual(try counter(syncSeqPrivacyZones), seqs.max()) + } + + /// Re-running the migrator must not renumber rows or re-seed the counter below positions already + /// handed out. + func testRemainingTablesMigrationIsANoOpOnReRun() throws { + try migrateToLatest() + let repo = try makeRepository() + repo.upsertPrivacyZone(["id": "zone-1", "preset": "home", "name": "Home", + "centerLatitude": 0.0, "centerLongitude": 0.0, "radiusMeters": 100]) + let before = try counter(syncSeqPrivacyZones) + + try migrateToLatest() + + XCTAssertEqual(try counter(syncSeqPrivacyZones), before) + } + + func testPrivacyZoneToggleMovesBothColumns() throws { + try migrateToLatest() + let repo = try makeRepository() + repo.upsertPrivacyZone(["id": "zone-1", "preset": "home", "name": "Home", + "centerLatitude": 0.0, "centerLongitude": 0.0, "radiusMeters": 100]) + let first = try XCTUnwrap(syncSeq("privacy_zones")) + let stamp = try queue.read { db in try Int64.fetchOne(db, sql: "SELECT updated_at FROM privacy_zones") } + + repo.setPrivacyZoneEnabled("zone-1", false) + + XCTAssertGreaterThan(try XCTUnwrap(syncSeq("privacy_zones")), first) + XCTAssertGreaterThan( + try XCTUnwrap(queue.read { db in try Int64.fetchOne(db, sql: "SELECT updated_at FROM privacy_zones") }), + try XCTUnwrap(stamp) + ) + } + + /// Rider identity and this phone's session state live in `app_settings` but name the phone, not + /// the Rider (#277). They are excluded by never being given a cursor position: 0 sits below every + /// Sync Cursor, so no scan sees the row. + func testPhoneLocalSettingsKeepNoCursorPosition() throws { + try migrateToLatest() + let repo = try makeRepository() + + repo.updateSetting("riderName", rawValue: "Kacper") + repo.updateSetting("liveHistoryLimit", rawValue: 10) + + XCTAssertEqual(try rowSyncSeq("app_settings", "key = ?", ["riderName"]), 0) + XCTAssertGreaterThan(try XCTUnwrap(rowSyncSeq("app_settings", "key = ?", ["liveHistoryLimit"])), 0) + } + + /// The backfill numbers every existing row from `rowid`, which would hand a phone-local key a + /// position and ship it exactly once on the first upload after upgrade. + func testMigrationStripsCursorsFromPhoneLocalSettings() throws { + try TelemetryDatabase.migrator.migrate(queue, upTo: "v42_telemetry_board_id") + try queue.write { db in + for key in ["riderName", "liveHistoryLimit"] { + try db.execute( + sql: "INSERT INTO app_settings (key, value_json, updated_at) VALUES (?, ?, ?)", + arguments: [key, "1", 1_000] + ) + } + } + + try migrateToLatest() + + XCTAssertEqual(try rowSyncSeq("app_settings", "key = ?", ["riderName"]), 0) + XCTAssertGreaterThan(try XCTUnwrap(rowSyncSeq("app_settings", "key = ?", ["liveHistoryLimit"])), 0) + } + + // MARK: - VESC Fault Evidence (#430) + + /// Seed fault evidence the way an installed app holds it just before the upgrade. + /// + /// The tables are rebuilt in their pre-v48 shape first: `VescFaultStore.createTables` is the + /// current schema, so migrating from scratch produces the columns a phone installed on v40..v47 + /// does not have — and only that phone's shape exercises the backfill. + private func seedFaultEvidenceBeforeV48() throws { + try TelemetryDatabase.migrator.migrate(queue, upTo: "v47_sync_binding") + try queue.write { db in + try db.execute(sql: "DROP TABLE vesc_fault_occurrences") + try db.execute(sql: "DROP TABLE vesc_fault_captures") + try db.execute(sql: """ + CREATE TABLE vesc_fault_occurrences ( + id TEXT NOT NULL PRIMARY KEY, + board_id TEXT NOT NULL, + code INTEGER NOT NULL, + occurred_at INTEGER NOT NULL, + last_observed_at INTEGER NOT NULL, + cleared_at INTEGER, + dismissed INTEGER NOT NULL + ) + """) + try db.execute(sql: """ + CREATE TABLE vesc_fault_captures ( + occurrence_id TEXT NOT NULL PRIMARY KEY, + board_id TEXT NOT NULL, + started_at INTEGER NOT NULL, + opened_at INTEGER NOT NULL, + sample_count INTEGER NOT NULL + ) + """) + } + try queue.write { db in + for (index, id) in ["fault-1", "fault-2"].enumerated() { + try db.execute( + sql: """ + INSERT INTO vesc_fault_occurrences + (id, board_id, code, occurred_at, last_observed_at, cleared_at, dismissed) + VALUES (?, 'board-1', 9, ?, ?, NULL, 0) + """, + arguments: [id, 1_000 + index, 5_000 + index] + ) + try db.execute( + sql: """ + INSERT INTO vesc_fault_captures + (occurrence_id, board_id, started_at, opened_at, sample_count) + VALUES (?, 'board-1', ?, ?, 3) + """, + arguments: [id, 900 + index, 1_000 + index] + ) + } + } + } + + func testFaultEvidenceGainsItsCursorColumnsAndIndexes() throws { + try migrateToLatest() + + XCTAssertTrue(try columnNames("vesc_fault_occurrences").contains("updated_at")) + for table in syncSeqTablesV48 { + XCTAssertTrue(try columnNames(table).contains("sync_seq"), "\(table) is missing sync_seq") + XCTAssertTrue( + try indexNames(table).contains("index_\(table)_sync_seq"), + "\(table) is missing its sync_seq index" + ) + } + // Append-only on an `AUTOINCREMENT` key, which already is its cursor. + XCTAssertFalse(try columnNames("vesc_fault_capture_samples").contains("sync_seq")) + } + + /// Epoch zero is not a truthful change timestamp: the server keeps the stored row unless the + /// incoming stamp is strictly newer, so a whole existing fault record would arrive unrestorable. + func testOccurrenceChangeTimestampIsBackfilledFromTheLastObservation() throws { + try seedFaultEvidenceBeforeV48() + + try migrateToLatest() + + let stamps = try queue.read { db in + try Int64.fetchAll(db, sql: "SELECT updated_at FROM vesc_fault_occurrences ORDER BY id") + } + XCTAssertEqual(stamps, [5_000, 5_001]) + } + + func testFaultEvidenceBackfillsDistinctPositionsAndSeedsItsCounters() throws { + try seedFaultEvidenceBeforeV48() + + try migrateToLatest() + + for (table, counterName) in [ + ("vesc_fault_occurrences", syncSeqVescFaultOccurrences), + ("vesc_fault_captures", syncSeqVescFaultCaptures), + ] { + let seqs = try queue.read { db in + try Int64.fetchAll(db, sql: "SELECT sync_seq FROM \(table) ORDER BY sync_seq") + } + XCTAssertEqual(Set(seqs).count, 2, "\(table) backfilled positions collide") + XCTAssertFalse(seqs.contains(0), "\(table) has a row at the seed value") + XCTAssertEqual(try counter(counterName), seqs.max()) + } + } + + /// Re-running the migrator must not renumber rows or re-seed a counter below positions already + /// handed out. + func testFaultEvidenceMigrationIsANoOpOnReRun() throws { + try seedFaultEvidenceBeforeV48() + try migrateToLatest() + let before = try counter(syncSeqVescFaultOccurrences) + + try migrateToLatest() + + XCTAssertEqual(try counter(syncSeqVescFaultOccurrences), before) + } +} diff --git a/modules/vescape-core/ios/telemetry/TelemetryDao.swift b/modules/vescape-core/ios/telemetry/TelemetryDao.swift index 59c630b84..d649503c4 100644 --- a/modules/vescape-core/ios/telemetry/TelemetryDao.swift +++ b/modules/vescape-core/ios/telemetry/TelemetryDao.swift @@ -32,7 +32,126 @@ internal func insertFrame(_ db: Database, _ state: FullTelemetryState) throws { ) } -internal func upsertBucket(_ db: Database, _ b: TelemetryBucket) throws { +/// Table names carrying a `sync_seq`, and the keys their counters use in `sync_sequences`. +internal let syncSeqBoards = "boards" +internal let syncSeqAlerts = "alerts" +internal let syncSeqMinuteBuckets = "telemetry_minute_buckets" +internal let syncSeqAppSettings = "app_settings" +internal let syncSeqBoardSettings = "board_settings" +internal let syncSeqBoardWarnings = "board_warnings" +internal let syncSeqPrivacyZones = "privacy_zones" +internal let syncSeqTuneProfiles = "tune_profiles" +internal let syncSeqFavorites = "favorites" +internal let syncSeqVescFaultOccurrences = "vesc_fault_occurrences" +internal let syncSeqVescFaultCaptures = "vesc_fault_captures" + +/// The three tables the `v44_sync_seq` migration gave a `sync_seq`, frozen at the set that existed +/// then. A migration iterates the tables it actually shipped with, never the current +/// [syncSeqTables] — growing that list must not retroactively change an older migration step. +internal let syncSeqTablesV44 = [syncSeqBoards, syncSeqAlerts, syncSeqMinuteBuckets] + +/// The six remaining mutable tables, given a `sync_seq` by `v45_sync_seq_remaining` (#281). +internal let syncSeqTablesV45 = [ + syncSeqAppSettings, + syncSeqBoardSettings, + syncSeqBoardWarnings, + syncSeqPrivacyZones, + syncSeqTuneProfiles, + syncSeqFavorites, +] + +/// The two mutable VESC Fault Evidence tables, given a `sync_seq` by `v48_fault_sync` (#430). +/// `vesc_fault_capture_samples` is deliberately absent: it is append-only on an `AUTOINCREMENT` key. +internal let syncSeqTablesV48 = [syncSeqVescFaultOccurrences, syncSeqVescFaultCaptures] + +/// Every table carrying a `sync_seq`. Append-only tables are deliberately absent: they declare +/// `INTEGER PRIMARY KEY AUTOINCREMENT`, which SQLite guarantees monotonic and never reused, so their +/// key already *is* their cursor. +internal let syncSeqTables = syncSeqTablesV44 + syncSeqTablesV45 + syncSeqTablesV48 + +/// The Sync Cursor counter table. Idempotent, and called both from the migration that introduced it +/// and from the store-level `createTables` seams tests build their schema from — a table whose write +/// path allocates a cursor cannot be created without it. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `SyncSequenceEntity` +internal func createSyncSequencesTable(_ db: Database) throws { + try db.execute( + sql: """ + CREATE TABLE IF NOT EXISTS sync_sequences ( + name TEXT NOT NULL PRIMARY KEY, + last_value INTEGER NOT NULL + ) + """ + ) +} + +/// Hands out the next Sync Cursor position for [name]. +/// +/// The Sync Cursor is the phone's own record of how far it has uploaded and never crosses the wire, +/// which is what lets the upload scan run on a counter instead of a clock: a device clock that steps +/// backwards makes an `updated_at >= watermark` scan skip the write entirely, because the row lands +/// below a cursor the phone already passed. A counter cannot regress. +/// +/// Bump-then-read rather than read-then-bump so two writes racing inside the same database can never +/// be handed the same number; both statements run in the caller's transaction. The counter lives in +/// its own table rather than being derived as `MAX(sync_seq) + 1`, which would hand the same number +/// out twice after the highest row is deleted. Seeded on demand because a database created fresh +/// never runs the migration that inserts the row. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `nextSyncSeq` +internal func nextSyncSeq(_ db: Database, _ name: String) throws -> Int64 { + try db.execute( + sql: "INSERT OR IGNORE INTO sync_sequences (name, last_value) VALUES (?, 0)", + arguments: [name] + ) + try db.execute( + sql: "UPDATE sync_sequences SET last_value = last_value + 1 WHERE name = ?", + arguments: [name] + ) + return try Int64.fetchOne(db, sql: "SELECT last_value FROM sync_sequences WHERE name = ?", arguments: [name]) ?? 0 +} + +/// The write-time fold behind `updated_at` on `boards` and `alerts`: never below the value already +/// stored, and strictly above it whenever the clock fails to be. +/// +/// `+ 1` rather than a plain `max` because the server keeps the stored row unless the incoming stamp +/// is strictly newer — freezing at the old value would satisfy the scan and still lose the edit. Per +/// row, so the inflation is bounded by the rewind and disappears once the wall clock passes it. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `ratchetUpdatedAt` +internal func ratchetUpdatedAt(_ previous: Int64?, _ now: Int64) -> Int64 { + guard let previous else { return now } + return max(previous + 1, now) +} + +/// Stamps `updated_at` and `sync_seq` on a row that `INSERT OR REPLACE` is about to rewrite. +/// +/// Read-modify-write rather than an `ON CONFLICT` fold: `INSERT OR REPLACE` deletes the old row +/// before inserting, so the ratchet has no `excluded`-style handle on the value it replaces. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `upsertBoardSetting` +internal func stampSyncColumns( + _ db: Database, + table: String, + sequence: String, + whereClause: String, + keys: StatementArguments, + now: Int64 +) throws -> (updatedAt: Int64, syncSeq: Int64) { + let previous = try Int64.fetchOne( + db, + sql: "SELECT updated_at FROM \(table) WHERE \(whereClause)", + arguments: keys + ) + return (ratchetUpdatedAt(previous, now), try nextSyncSeq(db, sequence)) +} + +/// [now] is the last-write-wins timestamp stamped on the row, ratcheted on conflict exactly as +/// boards and alerts are: the server guards this table with `WHERE stored.updated_at < +/// EXCLUDED.updated_at` like every other mutable table, so a stamp frozen at the stored value would +/// satisfy the scan and still be dropped server-side. +/// +/// Completeness is `sync_seq`'s job, and it moves on every write including a merge into a row the +/// scan may already have passed. +/// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryBucketBuilder.kt `toEntity` +internal func upsertBucket(_ db: Database, _ b: TelemetryBucket, now: Int64 = telemetryNowMs()) throws { + let syncSeq = try nextSyncSeq(db, syncSeqMinuteBuckets) try db.execute( sql: """ INSERT INTO telemetry_minute_buckets ( @@ -42,8 +161,9 @@ internal func upsertBucket(_ db: Database, _ b: TelemetryBucket) throws { max_battery_current_abs_ma, battery_used_wh_milli, battery_regen_wh_milli, max_duty_abs_permille, first_odometer_cm, last_odometer_cm, gps_point_count, precise_gps_point_count, gps_distance_cm, max_gps_speed_centi_mps, max_temp_mosfet_deci_c, max_temp_motor_deci_c, - first_latitude_e7, first_longitude_e7, first_moving_at_ms, last_moving_at_ms - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?, ?, ?, ?, ?, ?, ?) + first_latitude_e7, first_longitude_e7, first_moving_at_ms, last_moving_at_ms, updated_at, + sync_seq + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(bucket_start_ms, board_id) DO UPDATE SET sample_count=telemetry_minute_buckets.sample_count + excluded.sample_count, last_sample_at_ms=MAX(telemetry_minute_buckets.last_sample_at_ms, excluded.last_sample_at_ms), @@ -64,7 +184,9 @@ internal func upsertBucket(_ db: Database, _ b: TelemetryBucket) throws { max_temp_mosfet_deci_c=MAX(telemetry_minute_buckets.max_temp_mosfet_deci_c, excluded.max_temp_mosfet_deci_c), max_temp_motor_deci_c=MAX(telemetry_minute_buckets.max_temp_motor_deci_c, excluded.max_temp_motor_deci_c), first_moving_at_ms=MIN(telemetry_minute_buckets.first_moving_at_ms, excluded.first_moving_at_ms), - last_moving_at_ms=MAX(telemetry_minute_buckets.last_moving_at_ms, excluded.last_moving_at_ms) + last_moving_at_ms=MAX(telemetry_minute_buckets.last_moving_at_ms, excluded.last_moving_at_ms), + updated_at=MAX(telemetry_minute_buckets.updated_at + 1, excluded.updated_at), + sync_seq=excluded.sync_seq """, arguments: [ b.bucketStartMs, b.boardId, b.sampleCount, b.firstSampleAtMs, b.lastSampleAtMs, @@ -72,7 +194,8 @@ internal func upsertBucket(_ db: Database, _ b: TelemetryBucket) throws { b.minBatteryVoltageMv, b.maxMotorCurrentAbsMa, b.maxBatteryCurrentAbsMa, b.batteryUsedWhMilli, b.batteryRegenWhMilli, b.maxDutyAbsPermille, b.firstOdometerCm, b.lastOdometerCm, b.gpsPointCount, b.preciseGpsPointCount, b.maxGpsSpeedCentiMps, b.maxTempMosfetDeciC, - b.maxTempMotorDeciC, b.firstLatitudeE7, b.firstLongitudeE7, b.firstMovingAtMs, b.lastMovingAtMs, + b.maxTempMotorDeciC, b.firstLatitudeE7, b.firstLongitudeE7, b.firstMovingAtMs, b.lastMovingAtMs, now, + syncSeq, ] ) } diff --git a/modules/vescape-core/ios/telemetry/TelemetryDatabase.swift b/modules/vescape-core/ios/telemetry/TelemetryDatabase.swift index 1bd060f69..6395d6f92 100644 --- a/modules/vescape-core/ios/telemetry/TelemetryDatabase.swift +++ b/modules/vescape-core/ios/telemetry/TelemetryDatabase.swift @@ -177,6 +177,31 @@ enum TelemetryDatabase { } } + /// Replace the app-data database with an empty one, taking the Sync Cursors, the pending Sync + /// Actions and the Account binding with it (#284). + /// + /// Deleting the file rather than clearing tables is what makes the Account change safe: nothing + /// can survive with a cursor position or a binding that belonged to the previous Account. The wipe + /// is local maintenance and emits no Sync Actions to either Account — the log is part of what + /// goes. + /// + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/DatabaseBackupManager.kt `replaceWithFreshDatabase` + static func replaceWithFreshDatabase() throws { + guard let target = databaseURL else { throw CocoaError(.fileNoSuchFile) } + if let reopened { try? reopened.close() } + else if case let .success(pool) = poolResult { try? pool.close() } + + let fm = FileManager.default + for suffix in ["", "-wal", "-shm"] { + try? fm.removeItem(at: URL(fileURLWithPath: target.path + suffix)) + } + + // Migrating rebuilds the schema, so the new database starts unbound with no cursors. + let pool = try DatabasePool(path: target.path) + try migrator.migrate(pool) + reopened = pool + } + /// Internal, not private, so migration tests can run the real migrator against an in-memory /// database and stop at a chosen version with `migrate(_:upTo:)`. internal static var migrator: DatabaseMigrator { @@ -689,6 +714,161 @@ enum TelemetryDatabase { try db.execute(sql: "DROP TABLE IF EXISTS \(DEVICE_BOARD_MAP)") } + // Change Timestamps for the three tables that had none (#275): a rename, a toggle or a bucket + // still filling was invisible to an "everything changed since T" scan. Additive, and existing + // rows are backfilled rather than left at the `DEFAULT 0` a scan would re-send. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `MIGRATION_42_43` + migrator.registerMigration("v43_sync_cursors") { db in + let backfillSource = [ + "boards": "created_at", + "alerts": "created_at", + "telemetry_minute_buckets": "last_sample_at_ms", + ] + for (table, source) in backfillSource.sorted(by: { $0.key < $1.key }) { + let hasUpdatedAt = try db.columns(in: table).contains { $0.name == "updated_at" } + if !hasUpdatedAt { + try db.execute(sql: "ALTER TABLE \(table) ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + try db.execute(sql: "UPDATE \(table) SET updated_at = \(source)") + } + try db.execute( + sql: "CREATE INDEX IF NOT EXISTS index_\(table)_updated_at ON \(table)(updated_at)" + ) + } + } + + // Splits the device-local Sync Cursor from the wall-clock last-write-wins timestamp. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `MIGRATION_43_44` + migrator.registerMigration("v44_sync_seq") { db in + try db.execute( + sql: """ + CREATE TABLE IF NOT EXISTS sync_sequences ( + name TEXT NOT NULL PRIMARY KEY, + last_value INTEGER NOT NULL + ) + """ + ) + for table in syncSeqTablesV44 { + let hasSyncSeq = try db.columns(in: table).contains { $0.name == "sync_seq" } + if !hasSyncSeq { + try db.execute(sql: "ALTER TABLE \(table) ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0") + try db.execute(sql: "UPDATE \(table) SET sync_seq = rowid") + } + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_\(table)_sync_seq ON \(table)(sync_seq)") + try db.execute( + sql: """ + INSERT OR REPLACE INTO sync_sequences (name, last_value) + VALUES (?, (SELECT COALESCE(MAX(sync_seq), 0) FROM \(table))) + """, + arguments: [table] + ) + } + } + + // Sync Cursors for the six remaining mutable tables (#281). `board_warnings` also gains the + // wall-clock `updated_at` every other mutable table already carries, backfilled from its newest + // detection. + // + // Existing rows are backfilled from `rowid` — distinct and non-zero, so no two rows share a + // cursor position and none of them sit at the seed value — and each table's sequence is seeded + // past the highest value handed out. Every step is guarded, so a re-run is a no-op. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `MIGRATION_44_45` + migrator.registerMigration("v45_sync_seq_remaining") { db in + let hasWarningUpdatedAt = try db.columns(in: "board_warnings").contains { $0.name == "updated_at" } + if !hasWarningUpdatedAt { + try db.execute(sql: "ALTER TABLE board_warnings ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0") + try db.execute(sql: "UPDATE board_warnings SET updated_at = last_detected_at") + } + + for table in syncSeqTablesV45 { + let hasSyncSeq = try db.columns(in: table).contains { $0.name == "sync_seq" } + if !hasSyncSeq { + try db.execute(sql: "ALTER TABLE \(table) ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0") + try db.execute(sql: "UPDATE \(table) SET sync_seq = rowid") + } + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_\(table)_sync_seq ON \(table)(sync_seq)") + try db.execute( + sql: """ + INSERT OR REPLACE INTO sync_sequences (name, last_value) + VALUES (?, (SELECT COALESCE(MAX(sync_seq), 0) FROM \(table))) + """, + arguments: [table] + ) + } + + // Phone-local keys are defined by their absence from the scan, so the backfill above has to be + // undone for them: an uploader would otherwise ship whatever this phone happened to hold at + // upgrade time, exactly once. See `notSyncedSettingKeys`. + let placeholders = notSyncedSettingKeys.map { _ in "?" }.joined(separator: ",") + try db.execute( + sql: "UPDATE app_settings SET sync_seq = 0 WHERE key IN (\(placeholders))", + arguments: StatementArguments(notSyncedSettingKeys) + ) + } + + // The Sync Action log (#282): an append-only record of semantic removals, which no surviving row + // can express. Additive — a new table only — and guarded, so a re-run is a no-op. + // + // The log is keyed on its own `AUTOINCREMENT` cursor and carries no `sync_seq`: SQLite + // guarantees that key monotonic and never reused, so it already *is* the cursor. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `MIGRATION_45_46` + migrator.registerMigration("v46_sync_actions") { db in + try createSyncActionsTable(db) + } + + // The Account binding (#284): which Vescape Account this local database belongs to. Additive and + // guarded, and deliberately left empty — an existing install is unbound until an Account signs + // in and claims it, which is also what keeps the current age-only retention behaviour until + // then. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `MIGRATION_46_47` + migrator.registerMigration("v47_sync_binding") { db in + try createSyncBindingTable(db) + } + + // VESC Fault Evidence joins the backup (#430). The two mutable fault tables gain the Sync Cursor + // every other mutable table carries, and the Occurrence gains the wall-clock `updated_at` the + // server keys its upsert on. + // + // `updated_at` is backfilled from `last_observed_at` rather than left at `DEFAULT 0`: an + // existing occurrence has a truthful moment it last changed, and epoch zero would be a lie the + // server's last-write-wins guard reads as "older than anything". + // + // It cannot simply be derived from `last_observed_at` on every read either — a Rider dismissing + // an occurrence changes the row without the fault being observed again, and that edit is + // precisely the one a restore has to preserve. + // + // `vesc_fault_capture_samples` gets nothing: it is append-only on an `INTEGER PRIMARY KEY + // AUTOINCREMENT`, which already is its cursor. + // + // Backfill and seeding follow `v45_sync_seq_remaining` exactly, and every step is guarded, so a + // re-run is a no-op. + // @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDatabase.kt `MIGRATION_47_48` + migrator.registerMigration("v48_fault_sync") { db in + let hasOccurrenceUpdatedAt = try db.columns(in: "vesc_fault_occurrences") + .contains { $0.name == "updated_at" } + if !hasOccurrenceUpdatedAt { + try db.execute( + sql: "ALTER TABLE vesc_fault_occurrences ADD COLUMN updated_at INTEGER NOT NULL DEFAULT 0" + ) + try db.execute(sql: "UPDATE vesc_fault_occurrences SET updated_at = last_observed_at") + } + + for table in syncSeqTablesV48 { + let hasSyncSeq = try db.columns(in: table).contains { $0.name == "sync_seq" } + if !hasSyncSeq { + try db.execute(sql: "ALTER TABLE \(table) ADD COLUMN sync_seq INTEGER NOT NULL DEFAULT 0") + try db.execute(sql: "UPDATE \(table) SET sync_seq = rowid") + } + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_\(table)_sync_seq ON \(table)(sync_seq)") + try db.execute( + sql: """ + INSERT OR REPLACE INTO sync_sequences (name, last_value) + VALUES (?, (SELECT COALESCE(MAX(sync_seq), 0) FROM \(table))) + """, + arguments: [table] + ) + } + } + return migrator } } diff --git a/modules/vescape-core/ios/telemetry/TelemetryRepository.swift b/modules/vescape-core/ios/telemetry/TelemetryRepository.swift index 8c0bc3c09..5b0292cd7 100644 --- a/modules/vescape-core/ios/telemetry/TelemetryRepository.swift +++ b/modules/vescape-core/ios/telemetry/TelemetryRepository.swift @@ -310,6 +310,7 @@ internal final class TelemetryRepository { summary: summary ) guard FavoriteStore.shared.insert(favorite) else { return nil } + SyncCoordinator.shared.notifyRiderEdit() return favorite.toMap( boardName: favorite.boardId.flatMap { Self.boardNamesById()[$0] }, routePoints: favoriteRoutePoints(favorite) @@ -375,6 +376,7 @@ internal final class TelemetryRepository { summary: Self.favoriteSummary(points, config: config) ) guard let stored = FavoriteStore.shared.update(updated) else { return nil } + SyncCoordinator.shared.notifyRiderEdit() return stored.toMap( boardName: stored.boardId.flatMap { Self.boardNamesById()[$0] }, routePoints: favoriteRoutePoints(stored) @@ -385,7 +387,10 @@ internal final class TelemetryRepository { /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryRepository.kt `deleteFavorite` func deleteFavorite(_ id: String) -> Bool { let deleted = FavoriteStore.shared.delete(id) - if deleted { FavoriteMediaStore.shared.deleteDirectory(favoriteId: id) } + if deleted { + FavoriteMediaStore.shared.deleteDirectory(favoriteId: id) + SyncCoordinator.shared.notifyRiderEdit() + } return deleted } @@ -439,15 +444,13 @@ internal final class TelemetryRepository { return buildFavoriteSummary(buildTelemetryBuckets(sanitized)) } + /// Retention sweep. Age-only while this database has never been bound to an Account, and age plus + /// the accepted Sync Cursor once it has — cleanup must not remove a row the uploader has not + /// delivered (#284). func deleteBefore(_ beforeMs: Int64) -> Int { guard let pool else { return 0 } return (try? pool.write { db in - let count = try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM telemetry_frames WHERE captured_at_ms < ?", arguments: [beforeMs]) ?? 0 - try db.execute(sql: "DELETE FROM telemetry_frames WHERE captured_at_ms < ?", arguments: [beforeMs]) - try db.execute(sql: "DELETE FROM telemetry_minute_buckets WHERE bucket_start_ms < ?", arguments: [beforeMs]) - try db.execute(sql: "DELETE FROM telemetry_markers WHERE occurred_at_ms < ?", arguments: [beforeMs]) - try db.execute(sql: "DELETE FROM metric_exclusion_ranges WHERE end_ms < ?", arguments: [beforeMs]) - return count + try deleteBeforeGated(db, beforeMs: beforeMs) }) ?? 0 } @@ -616,6 +619,9 @@ internal final class TelemetryRepository { for marker in markers { try insertMarker(db, marker) } for range in sanitization.exclusions { try insertExclusion(db, range) } } + // Samples are actually being produced, which is what the uploader's ride cadence follows — Idle + // Pause halts production without ending the Board Session. + SyncCoordinator.shared.notifySamplesPersisted() } private func marker(type: String, capture: TelemetryCapture, gapMs: Int64?) -> [String: Any?] { diff --git a/modules/vescape-core/ios/telemetry/TuneProfileStore.swift b/modules/vescape-core/ios/telemetry/TuneProfileStore.swift index eeaffa140..1b2787ce2 100644 --- a/modules/vescape-core/ios/telemetry/TuneProfileStore.swift +++ b/modules/vescape-core/ios/telemetry/TuneProfileStore.swift @@ -78,10 +78,14 @@ struct TuneProfileStore { color TEXT NOT NULL DEFAULT 'purple', fields_json TEXT NOT NULL, created_at INTEGER NOT NULL, - updated_at INTEGER NOT NULL + updated_at INTEGER NOT NULL, + sync_seq INTEGER NOT NULL DEFAULT 0 ) """) try db.execute(sql: "CREATE INDEX index_tune_profiles_board_id ON tune_profiles(board_id)") + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_tune_profiles_sync_seq ON tune_profiles(sync_seq)") + try createSyncSequencesTable(db) + try createSyncActionsTable(db) try db.execute(sql: "CREATE INDEX index_tune_profiles_board_id_refloat_base_version ON tune_profiles(board_id, refloat_base_version)") try db.execute(sql: """ @@ -153,10 +157,13 @@ struct TuneProfileStore { return try inWrite { db in try db.execute( sql: """ - INSERT INTO tune_profiles (id, board_id, refloat_base_version, name, icon, color, fields_json, created_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + INSERT INTO tune_profiles (id, board_id, refloat_base_version, name, icon, color, fields_json, created_at, updated_at, sync_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, - arguments: [id, boardId, compatibility, name, icon, color, fieldsJson, now, now] + arguments: [ + id, boardId, compatibility, name, icon, color, fieldsJson, now, now, + try nextSyncSeq(db, syncSeqTuneProfiles), + ] ) try Self.insertHistory(db, profileId: id, fieldsJson: fieldsJson, createdAt: now) return try Self.requireProfileMap(db, id) @@ -173,8 +180,11 @@ struct TuneProfileStore { let now = Self.nowMs() return try inWrite { db in try db.execute( - sql: "UPDATE tune_profiles SET name = ?, icon = ?, color = ?, updated_at = ? WHERE id = ?", - arguments: [name, icon, color, now, profileId] + sql: """ + UPDATE tune_profiles SET name = ?, icon = ?, color = ?, + updated_at = MAX(updated_at + 1, ?), sync_seq = ? WHERE id = ? + """, + arguments: [name, icon, color, now, try nextSyncSeq(db, syncSeqTuneProfiles), profileId] ) guard let map = try Self.fetchProfileMap(db, profileId) else { throw TuneProfileError.profileNotFound(profileId) @@ -197,8 +207,17 @@ struct TuneProfileStore { arguments: [boardId, row["refloat_base_version"] as String] ) ?? 0 if count <= 1 { throw TuneProfileError.cannotDeleteLast } + // Tune History is a parent-covered cascade: raw, because the profile's own Sync Action + // covers it (#282). try db.execute(sql: "DELETE FROM tune_history_entries WHERE profile_id = ?", arguments: [profileId]) - try db.execute(sql: "DELETE FROM tune_profiles WHERE id = ?", arguments: [profileId]) + try deleteForSync( + db, + target: .tuneProfile, + boardId: nil, + key: profileId, + whereClause: "id = ?", + keys: [profileId] + ) return true } } @@ -224,8 +243,11 @@ struct TuneProfileStore { try Self.insertHistory(db, profileId: profileId, fieldsJson: profile["fields_json"], createdAt: now) try db.execute( - sql: "UPDATE tune_profiles SET fields_json = ?, updated_at = ? WHERE id = ?", - arguments: [entry["fields_json"] as String, now, profileId] + sql: """ + UPDATE tune_profiles SET fields_json = ?, updated_at = MAX(updated_at + 1, ?), sync_seq = ? + WHERE id = ? + """, + arguments: [entry["fields_json"] as String, now, try nextSyncSeq(db, syncSeqTuneProfiles), profileId] ) guard let map = try Self.fetchProfileMap(db, profileId) else { throw TuneProfileError.disappearedDuringRollback(profileId) @@ -248,10 +270,13 @@ struct TuneProfileStore { let color: String = source["color"] try db.execute( sql: """ - INSERT INTO tune_profiles (id, board_id, refloat_base_version, name, icon, color, fields_json, created_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + INSERT INTO tune_profiles (id, board_id, refloat_base_version, name, icon, color, fields_json, created_at, updated_at, sync_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, - arguments: [copyId, targetBoardId, source["refloat_base_version"] as String, newName, icon, color, fieldsJson, now, now] + arguments: [ + copyId, targetBoardId, source["refloat_base_version"] as String, newName, icon, color, + fieldsJson, now, now, try nextSyncSeq(db, syncSeqTuneProfiles), + ] ) try Self.insertHistory(db, profileId: copyId, fieldsJson: fieldsJson, createdAt: now) return try Self.requireProfileMap(db, copyId) @@ -270,8 +295,11 @@ struct TuneProfileStore { } try Self.insertHistory(db, profileId: profileId, fieldsJson: current["fields_json"], createdAt: now) try db.execute( - sql: "UPDATE tune_profiles SET fields_json = ?, updated_at = ? WHERE id = ?", - arguments: [fieldsJson, now, profileId] + sql: """ + UPDATE tune_profiles SET fields_json = ?, updated_at = MAX(updated_at + 1, ?), sync_seq = ? + WHERE id = ? + """, + arguments: [fieldsJson, now, try nextSyncSeq(db, syncSeqTuneProfiles), profileId] ) guard let map = try Self.fetchProfileMap(db, profileId) else { throw TuneProfileError.disappearedDuringSave(profileId) diff --git a/modules/vescape-core/ios/warnings/BoardWarningStore.swift b/modules/vescape-core/ios/warnings/BoardWarningStore.swift index afc493792..33006edde 100644 --- a/modules/vescape-core/ios/warnings/BoardWarningStore.swift +++ b/modules/vescape-core/ios/warnings/BoardWarningStore.swift @@ -63,10 +63,15 @@ struct BoardWarningStore { first_detected_at INTEGER NOT NULL, last_detected_at INTEGER NOT NULL, payload_json TEXT NOT NULL, + updated_at INTEGER NOT NULL DEFAULT 0, + sync_seq INTEGER NOT NULL DEFAULT 0, PRIMARY KEY (board_id, kind) ) """) try db.execute(sql: "CREATE INDEX index_board_warnings_board_id ON board_warnings(board_id)") + try db.execute(sql: "CREATE INDEX IF NOT EXISTS index_board_warnings_sync_seq ON board_warnings(sync_seq)") + try createSyncSequencesTable(db) + try createSyncActionsTable(db) } /// The shared pool failed to open — findings are dropped / reads come back empty, so leave the @@ -149,16 +154,20 @@ struct BoardWarningStore { try db.execute( sql: """ INSERT INTO board_warnings - (board_id, kind, severity, first_detected_at, last_detected_at, payload_json) - VALUES (?, ?, ?, ?, ?, ?) + (board_id, kind, severity, first_detected_at, last_detected_at, payload_json, + updated_at, sync_seq) + VALUES (?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(board_id, kind) DO UPDATE SET severity = excluded.severity, last_detected_at = excluded.last_detected_at, - payload_json = excluded.payload_json + payload_json = excluded.payload_json, + updated_at = MAX(board_warnings.updated_at + 1, excluded.updated_at), + sync_seq = excluded.sync_seq """, arguments: [ warning.boardId, warning.kind, warning.severity, warning.firstDetectedAtMs, warning.lastDetectedAtMs, warning.payloadJson, + warning.lastDetectedAtMs, try nextSyncSeq(db, syncSeqBoardWarnings), ] ) } @@ -167,6 +176,13 @@ struct BoardWarningStore { } } + /// Semantic removal, whether the Rider cleared the warning or a detector evaluated the kind with + /// real data and found the condition gone — an automatic clear is still a durable state transition + /// the server has to make (#282). + /// + /// Stamped from `last_detected_at` rather than `updated_at`: it is the warning's own change clock, + /// and it is what the row's `updated_at` was written from. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `deleteBoardWarning` @discardableResult func delete(_ boardId: String, _ kind: String) -> Bool { guard let writer = resolveWriter() else { @@ -175,11 +191,15 @@ struct BoardWarningStore { } do { return try writer.write { db in - try db.execute( - sql: "DELETE FROM board_warnings WHERE board_id = ? AND kind = ?", - arguments: [boardId, kind] + try deleteForSync( + db, + target: .boardWarning, + boardId: boardId, + key: kind, + whereClause: "board_id = ? AND kind = ?", + keys: [boardId, kind], + stampColumn: "last_detected_at" ) - return db.changesCount > 0 } } catch { BoardWarningFailureReporter.shared.report(site: "store_delete", error: error) @@ -187,6 +207,10 @@ struct BoardWarningStore { } } + /// The Rider cleared every warning on one Board: one action per removed row, because each row is + /// a separate piece of current state. Distinct from the Board delete's cascade, which is raw and + /// covered by the Board's own action. + /// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryDao.kt `deleteBoardWarnings` @discardableResult func deleteForBoard(_ boardId: String) -> Bool { guard let writer = resolveWriter() else { @@ -195,8 +219,24 @@ struct BoardWarningStore { } do { return try writer.write { db in - try db.execute(sql: "DELETE FROM board_warnings WHERE board_id = ?", arguments: [boardId]) - return db.changesCount > 0 + let kinds = try String.fetchAll( + db, + sql: "SELECT kind FROM board_warnings WHERE board_id = ?", + arguments: [boardId] + ) + var removed = false + for kind in kinds { + removed = try deleteForSync( + db, + target: .boardWarning, + boardId: boardId, + key: kind, + whereClause: "board_id = ? AND kind = ?", + keys: [boardId, kind], + stampColumn: "last_detected_at" + ) || removed + } + return removed } } catch { BoardWarningFailureReporter.shared.report(site: "store_delete_for_board", error: error) diff --git a/modules/vescape-core/src/e2eFake.ts b/modules/vescape-core/src/e2eFake.ts index 613de88a5..c89be7466 100644 --- a/modules/vescape-core/src/e2eFake.ts +++ b/modules/vescape-core/src/e2eFake.ts @@ -85,6 +85,9 @@ const e2eSettings: AppSettings = { companionPresenceCooldownMinutes: 60, autoCloseEnabled: false, autoCloseDelayMinutes: 15, + syncEnabled: false, + syncWifiOnly: false, + syncBackupChoiceMade: false, telemetryPollRateHz: 20, wearPushRateHz: 4, wearAutoLaunchOnConnect: true, @@ -749,10 +752,12 @@ export const e2eFake = { }, upsertBoard(board: BoardInput): void { + // Stand in for native: the sync cursor is stamped by the store on every write, never by the caller. const index = e2eBoards.findIndex((b) => b.id === board.id) // A tombstone survives an upsert, like native — only a delete stamps one. const stored: Board = { ...board, + updatedAt: Date.now(), deletedAt: index >= 0 ? e2eBoards[index].deletedAt : null, } if (index >= 0) { @@ -795,13 +800,17 @@ export const e2eFake = { }, seedE2EData(flow: string): void { + // Seeded rows stand in for freshly inserted ones, where the sync cursor equals `createdAt`. + const seededAt = Date.now() + if (flow === 'connect-board') { const boardId = 'e2e-board-1' const board: Board = { id: boardId, name: 'E2E Board', description: 'Seeded by Maestro', - createdAt: Date.now(), + createdAt: seededAt, + updatedAt: seededAt, deletedAt: null, batteryConfig: { mode: 'preset', @@ -823,7 +832,8 @@ export const e2eFake = { id: boardId, name: 'E2E History Board', description: 'Seeded by Maestro', - createdAt: Date.now(), + createdAt: seededAt, + updatedAt: seededAt, deletedAt: null, batteryConfig: { mode: 'preset', @@ -846,7 +856,8 @@ export const e2eFake = { id: boardId, name: 'E2E Privacy Board', description: 'Seeded by Maestro', - createdAt: Date.now(), + createdAt: seededAt, + updatedAt: seededAt, deletedAt: null, batteryConfig: { mode: 'preset', diff --git a/modules/vescape-core/src/index.ts b/modules/vescape-core/src/index.ts index 17a209ef2..ea325233f 100644 --- a/modules/vescape-core/src/index.ts +++ b/modules/vescape-core/src/index.ts @@ -183,16 +183,24 @@ export interface BoardLink { refloatBaseVersion?: string } +// @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `BoardEntity` +// @parity /modules/vescape-core/ios/telemetry/AppDataRepository.swift `composeBoard` export interface Board { id: string name: string description: string | null createdAt: number + /** + * Incremental-sync cursor: epoch ms of the last write to this board, from the same clock as + * {@link createdAt}. Native stamps it on every upsert (including partial edits), so a value sent + * from JS is ignored — read it, do not author it. + */ + updatedAt: number /** * Tombstone stamp: epoch ms of the rider's delete, `null` while the Board is alive. A deleted * Board keeps its row so Ride History can still name it (ADR 0027) — {@link getBoards} filters - * tombstones, {@link getBoard} deliberately does not. Native-owned: deletion goes through - * {@link deleteBoard}, never through an upsert. + * tombstones, {@link getBoard} deliberately does not. Native-owned like {@link updatedAt}: + * deletion goes through {@link deleteBoard}, never through an upsert. */ deletedAt: number | null batteryConfig: BatteryConfig | null @@ -241,10 +249,12 @@ export interface Board { } /** - * Write shape for {@link upsertBoard}. A tombstone is stamped by {@link deleteBoard} alone, and an - * upsert never clears the one already on the row, so callers never author `deletedAt`. + * Write shape for {@link upsertBoard}. Native stamps `updatedAt` from its own clock on every write, + * so callers never author it — a board that has never been persisted has no cursor yet. `deletedAt` + * is out for the same reason: a tombstone is stamped by {@link deleteBoard} alone, and an upsert + * never clears the one already on the row. */ -export type BoardInput = Omit +export type BoardInput = Omit export interface LastBattery { percent: number @@ -331,6 +341,12 @@ export interface AlertRule { * preset sounds only — a text-to-speech rule speaks once regardless. */ beepCount: number + /** + * Incremental-sync cursor: epoch ms of the last write to this rule, from the same clock as + * {@link createdAt}. Native stamps it on every upsert and on the enable/disable toggle, so a + * value sent from JS is ignored — read it, do not author it. + */ + updatedAt: number /** * Provenance tag. `manual` (or absent) = rider-authored. `preset` rules are generated + owned * by JS orchestration and regenerated wholesale; native persists the string opaquely. @@ -359,6 +375,12 @@ export const ALERT_BEEP_COUNT_RANGE = { min: 1, max: 5 } as const /** Beeps per announcement when nothing says otherwise — matches the pre-`beepCount` behavior. */ export const ALERT_BEEP_COUNT_DEFAULT = 3 +/** + * Write shape for {@link upsertAlertRule}. Native stamps `updatedAt` from its own clock on every + * write, so callers never author it — a rule that has never been persisted has no cursor yet. + */ +export type AlertRuleInput = Omit + export type PrivacyZonePreset = 'home' | 'work' | 'custom' export interface PrivacyZone { @@ -1132,6 +1154,22 @@ export interface AppSettings { autoCloseEnabled: boolean /** Minutes without a board connection before auto close fires. UI offers 1–480; native accepts up to 1440. */ autoCloseDelayMinutes: number + /** + * Backup master switch, off by default. Off means the uploader does nothing at all: no scan, no + * request, no retry, no notification. Phone-local, and deliberately not synced — a restored + * snapshot must never be able to switch backup back on. + */ + syncEnabled: boolean + /** + * Nothing uploads on a metered connection while this is on — mid-ride included. No row classes, + * no backlog thresholds, no partial exceptions. + */ + syncWifiOnly: boolean + /** + * The one-time backup choice has been offered on this phone and answered. Phone-local: the + * expensive first upload belongs to the phone that holds the backlog. + */ + syncBackupChoiceMade: boolean /** * Max telemetry poll rate in Hz, applied as a minimum spacing floor between * requests. Polling stays response-paced (the next request is only sent once @@ -1412,6 +1450,28 @@ export interface AppDataChangedEvent { scope: 'boards' | 'settings' } +/** + * What a Sync Action can name — and, by omission, what it cannot. A deleted row cannot carry a + * Change Timestamp saying it is gone, so native appends a Sync Action for every semantic removal and + * the server replays it against the Rider's backup. + * + * Every case is configuration or current state a Rider edits directly. Ride History is absent on + * purpose: telemetry is pruned locally on a retention rule, and an action naming it would delete + * exactly the rides the backup exists to preserve. The log is native-owned — JS never writes it — + * and this union exists so the two native definitions cannot drift apart unnoticed. + * @parity /modules/vescape-core/ios/telemetry/SyncActionLog.swift `DeleteTarget` + * @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/TelemetryEntities.kt `DeleteTarget` + */ +export type DeleteTarget = + | 'appSetting' + | 'board' + | 'boardSetting' + | 'boardWarning' + | 'alert' + | 'tuneProfile' + | 'privacyZone' + | 'favorite' + /** * Two-level Board Warning severity, fixed at detection time. * @parity /modules/vescape-core/ios/warnings/BoardWarningKind.swift `BoardWarningSeverity` @@ -2029,8 +2089,62 @@ export interface DeviceCredentialStatus { state: DeviceCredentialState accountId: string | null expiresAt: string | null + /** + * A different Vescape Account signed in on a phone whose local database already belongs to another + * one. Native refuses to bind — resetting the Sync Cursors over the existing rows would upload the + * previous Account's data to the new one — so the credential is not stored until the Rider + * confirms through `confirmSyncAccountReset` that all local app data is erased. + */ + accountChangeRequiresReset?: boolean } +/** + * Why the uploader stopped. A paused engine is not woken by ordinary timer or connectivity kicks. + * + * @parity /modules/vescape-core/ios/sync/SyncPolicy.swift `SyncPauseReason` + * @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt `SyncPauseReason` + */ +export type SyncPauseReason = 'authentication' | 'protocol' | 'rowTooLarge' + +/** + * The backup state the Rider is shown, derived natively from the same state the uploader decides on. + * + * @parity /modules/vescape-core/ios/sync/SyncPolicy.swift `SyncActivity` + * @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncPolicy.kt `SyncActivity` + */ +export type SyncActivity = + | 'disabled' + | 'signedOut' + | 'upToDate' + | 'syncing' + | 'waitingForWifi' + | 'offline' + | 'paused' + +/** + * Native-owned backup state. JS renders it and never infers one of its own. + * + * @parity /modules/vescape-core/ios/sync/SyncCoordinator.swift `SyncStatus` + * @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/sync/SyncCoordinator.kt `SyncStatus` + */ +export interface SyncStatus { + /** The Account this local database is bound to, or null while it has never been claimed. */ + accountId: string | null + pendingRows: number + activity: SyncActivity + /** Which permanent failure stopped the uploader, when `activity` is `paused`. */ + pause: SyncPauseReason | null + lastUploadAtMs: number | null +} + +/** + * Backup state changed. Emitted on every transition and replayed on subscribe, so a late listener is + * immediately consistent. + * @parity /modules/vescape-core/ios/VescapeCoreModule.swift `sendSyncStatus` + * @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/VescapeCoreModule.kt `onSyncStatus` + */ +export type SyncStatusEvent = SyncStatus + export type CriticalRideNotificationPermissionStatus = | 'not-determined' | 'denied' @@ -2093,6 +2207,7 @@ type VescapeCoreEvents = { onRouteProgress: (event: RouteProgressEvent) => void /** Native forecast, on every successful refresh and on subscribe. */ onWeather: (event: WeatherEvent) => void + onSyncStatus: (event: SyncStatusEvent) => void } interface NativeEventEmitter void>> { @@ -2172,6 +2287,12 @@ type VescapeCoreNativeModule = NativeEventEmitter & { getDeviceCredentialState(): DeviceCredentialStatus revokeDeviceCredential(): Promise clearDeviceCredential(): void + confirmSyncAccountReset( + serverUrl: string, + deviceToken: string, + accountId: string, + ): Promise + getSyncStatus(): Promise openAppUpdate(): void getRemoteTiltState(): RemoteTiltState | null setSelectedBoard(boardId: string | null): void @@ -2266,7 +2387,7 @@ type VescapeCoreNativeModule = NativeEventEmitter & { upsertBoard(board: BoardInput): Promise deleteBoard(id: string): Promise getAlertRules(boardId: string): Promise - upsertAlertRule(rule: AlertRule): Promise + upsertAlertRule(rule: AlertRuleInput): Promise setAlertRuleEnabled(boardId: string, id: string, enabled: boolean): Promise deleteAlertRule(boardId: string, id: string): Promise getPrivacyZones(): Promise @@ -2703,6 +2824,25 @@ export function getDeviceCredentialState(): DeviceCredentialStatus { return native.getDeviceCredentialState() } +/** + * Erase all local app data and hand the fresh database to a different Account. Destructive, and only + * ever called after the Rider confirms — cloud restore does not exist in this version, so what is + * erased is gone. + */ +export async function confirmSyncAccountReset( + serverUrl: string, + deviceToken: string, + accountId: string, +): Promise { + return native.confirmSyncAccountReset(serverUrl, deviceToken, accountId) +} + +/** Read native-owned backup state: what is bound, what is pending, and why it stopped. */ +export async function getSyncStatus(): Promise { + return native.getSyncStatus() +} + +/** Back up over Wi-Fi only. Native waits for Wi-Fi rather than failing on a metered connection. */ export async function revokeDeviceCredential(): Promise { return native.revokeDeviceCredential() } @@ -3174,15 +3314,15 @@ export async function getAlertRules(boardId: string): Promise { * @parity /modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/AppDataRepository.kt `toAlertRuleEntity` * @parity /modules/vescape-core/ios/telemetry/AppDataRepository.swift `upsertAlertRule` */ -function bridgeableAlertRule(rule: AlertRule): AlertRule { +function bridgeableAlertRule(rule: AlertRuleInput): AlertRuleInput { const thresholdRule = rule.thresholdRule if (!thresholdRule || thresholdRule.kind !== 'config-relative') return rule const { thresholdMaxOffset, ...rest } = thresholdRule if (thresholdMaxOffset != null) return rule - return { ...rule, thresholdRule: rest as AlertRule['thresholdRule'] } + return { ...rule, thresholdRule: rest as AlertRuleInput['thresholdRule'] } } -export async function upsertAlertRule(rule: AlertRule): Promise { +export async function upsertAlertRule(rule: AlertRuleInput): Promise { return native.upsertAlertRule(bridgeableAlertRule(rule)) } @@ -3442,6 +3582,10 @@ export function addRouteProgressListener( return emitter.addListener('onRouteProgress', cb) } +export function addSyncStatusListener(cb: (event: SyncStatusEvent) => void): EventSubscription { + return emitter.addListener('onSyncStatus', cb) +} + export function addLiveStateListener(cb: (event: LiveStateEvent) => void): EventSubscription { if (E2E_ENABLED) { return e2eFake.addLiveStateListener(cb) diff --git a/src/app/_layout.tsx b/src/app/_layout.tsx index 9625c2416..b06f96978 100644 --- a/src/app/_layout.tsx +++ b/src/app/_layout.tsx @@ -29,6 +29,7 @@ import { startBoardConfigChangeNoticeSync } from '@/modules/board/store/boardCon import { BoardConfigChangeNoticeModal } from '@/modules/board/components/BoardConfigChangeNoticeModal' import { startTuneSnapshotSessionSync } from '@/modules/tune/store/tuneSnapshotStore' import { startBoardWarningsSync } from '@/modules/board/store/boardWarningsStore' +import { startSyncStatusSync } from '@/modules/profile/store/syncStatusStore' import { startVescFaultsSync } from '@/modules/board/store/vescFaultsStore' import { useGroupRideStore } from '@/modules/group-ride/store/groupRideStore' import { useRiderStore } from '@/modules/group-ride/store/riderStore' @@ -105,6 +106,7 @@ function RootLayout() { const stopAppStatusSync = startAppStatusSync() const stopNavigationSync = startNavigationSync() const stopWeatherSync = startWeatherSync() + const stopSyncStatusSync = startSyncStatusSync() return () => { useGroupRideStore.getState().stopObserving() stopAppDataSync() @@ -119,6 +121,7 @@ function RootLayout() { stopAppStatusSync() stopNavigationSync() stopWeatherSync() + stopSyncStatusSync() } }, [fixturesReady]) @@ -208,6 +211,7 @@ function RootLayout() { + router.push(routes.settingsMap)} /> + router.push(routes.settingsSync)} + /> {Platform.OS === 'android' && ( diff --git a/src/app/settings/components/widgets.tsx b/src/app/settings/components/widgets.tsx index e5ed60869..953f1ea3f 100644 --- a/src/app/settings/components/widgets.tsx +++ b/src/app/settings/components/widgets.tsx @@ -9,6 +9,7 @@ import { ExpandingWidgetShowcase, DialWidgetShowcase, SwitchWidgetShowcase, + BackupStatusLineShowcase, } from '@/screens/showcase/widgets/DisplayWidgetShowcases' import { InputWidgetShowcase, @@ -33,6 +34,7 @@ export default function WidgetsPage() { + ) diff --git a/src/app/settings/database.tsx b/src/app/settings/database.tsx index e2172e622..58ba25067 100644 --- a/src/app/settings/database.tsx +++ b/src/app/settings/database.tsx @@ -13,6 +13,7 @@ import { theme } from '@/constants/theme' import { SettingsCard } from '@/components/settings/SettingsCard' import { SettingsRow } from '@/components/settings/SettingsRow' import { Button } from '@/components/base/Button' +import { ProgressBar } from '@/components/base/ProgressBar' import { ConfirmModal } from '@/components/modals/ConfirmModal' import { useSettingsDatabaseOps } from '@/modules/settings/hooks/useSettingsDatabaseOps' import { IconHero } from '@/components/settings/IconHero' @@ -57,19 +58,10 @@ export default function DatabaseSettingsScreen() { > {db.rebuildState === 'running' && ( - - - - {db.rebuildProgressLabel ? ( - {db.rebuildProgressLabel} - ) : null} + )} @@ -153,28 +145,7 @@ const styles = StyleSheet.create({ fontWeight: '700', }, rebuildProgress: { - flexDirection: 'row', - alignItems: 'center', - gap: 8, marginHorizontal: 14, marginBottom: 12, }, - rebuildProgressTrack: { - flex: 1, - height: 3, - backgroundColor: theme.neutral.surfaceDeep, - borderRadius: 999, - overflow: 'hidden', - }, - rebuildProgressFill: { - height: '100%', - backgroundColor: theme.status.warning.color, - }, - rebuildProgressText: { - minWidth: 44, - color: theme.neutral.textMuted, - fontSize: 11, - fontWeight: '700', - textAlign: 'right', - }, }) diff --git a/src/app/settings/sync.tsx b/src/app/settings/sync.tsx new file mode 100644 index 000000000..a72935083 --- /dev/null +++ b/src/app/settings/sync.tsx @@ -0,0 +1,88 @@ +import { ScrollView, StyleSheet, Switch, View } from 'react-native' +import { SafeAreaView } from 'react-native-safe-area-context' +import { CloudArrowUpIcon, WifiHighIcon } from 'phosphor-react-native' + +import { Text } from '@/components/base/Text' +import { SettingsCard } from '@/components/settings/SettingsCard' +import { SettingsRow } from '@/components/settings/SettingsRow' +import { IconHero } from '@/components/settings/IconHero' +import { theme } from '@/constants/theme' +import { BackupStatusLine } from '@/modules/profile/components/BackupStatusLine' +import { useSettingsStore } from '@/modules/settings/store/settingsStore' + +export default function SyncSettingsScreen() { + const syncEnabled = useSettingsStore((s) => s.syncEnabled) + const syncWifiOnly = useSettingsStore((s) => s.syncWifiOnly) + const set = useSettingsStore((s) => s.set) + + return ( + + + + + + void set('syncEnabled', v)} + trackColor={{ false: theme.palette.slate.border, true: theme.palette.sky.border }} + thumbColor={syncEnabled ? theme.palette.sky.color : theme.palette.slate.textMuted} + /> + } + > + + + + + + {syncEnabled ? ( + void set('syncWifiOnly', v)} + trackColor={{ false: theme.palette.slate.border, true: theme.palette.sky.border }} + thumbColor={ + syncWifiOnly ? theme.palette.sky.color : theme.palette.slate.textMuted + } + /> + } + /> + ) : null} + + + + + Backup is off by default and stays on this phone until you turn it on. Switching it off + stops the uploader immediately; nothing already uploaded is deleted, and this switch is + never restored from a backup. + + + + + ) +} + +const styles = StyleSheet.create({ + container: { flex: 1, backgroundColor: theme.palette.slate.bg }, + content: { padding: 12, gap: 12, paddingBottom: 40 }, + status: { paddingHorizontal: 14, paddingBottom: 14 }, + note: { paddingHorizontal: 4 }, + noteText: { + color: theme.palette.slate.textMuted, + fontSize: 12, + lineHeight: 17, + }, +}) diff --git a/src/components/base/ProgressBar.tsx b/src/components/base/ProgressBar.tsx new file mode 100644 index 000000000..80993bf0a --- /dev/null +++ b/src/components/base/ProgressBar.tsx @@ -0,0 +1,64 @@ +import { StyleSheet, View } from 'react-native' + +import { Text } from '@/components/base/Text' +import { theme } from '@/constants/theme' + +export interface ProgressBarProps { + /** Units finished so far. */ + current: number + /** Units the run started with. A total of zero draws an empty track and no readout. */ + total: number + color?: string +} + +/** + * Determinate progress for work the Rider is waiting on: a rebuild, a backup drain. The counts are + * the contract, and the bar owns how they read — every place that shows progress shows it the same. + */ +export function ProgressBar({ + current, + total, + color = theme.status.warning.color, +}: ProgressBarProps) { + const fraction = total > 0 ? Math.min(1, Math.max(0, current / total)) : 0 + + return ( + + + + + {total > 0 ? ( + + {current}/{total} + + ) : null} + + ) +} + +const styles = StyleSheet.create({ + row: { + flexDirection: 'row', + alignItems: 'center', + gap: 8, + }, + track: { + flex: 1, + height: 3, + backgroundColor: theme.palette.slate.surfaceDeep, + borderRadius: 999, + overflow: 'hidden', + }, + fill: { + height: '100%', + }, + label: { + minWidth: 44, + color: theme.palette.slate.textMuted, + fontSize: 11, + fontWeight: '700', + textAlign: 'right', + }, +}) diff --git a/src/modules/alerts/lib/customAlertRules.ts b/src/modules/alerts/lib/customAlertRules.ts index 7a7da61dd..f590415a8 100644 --- a/src/modules/alerts/lib/customAlertRules.ts +++ b/src/modules/alerts/lib/customAlertRules.ts @@ -17,7 +17,7 @@ import { * {@link DraftAlertRule} is that shape; the live adapter maps its store rules down to it * and the wizard holds them in memory until `save()` stamps the new Board's id on. */ -export type DraftAlertRule = Omit +export type DraftAlertRule = Omit /** * Take ownership of a level: expand it exactly as the preset generator would, then hand the diff --git a/src/modules/alerts/store/alertPresetStore.test.ts b/src/modules/alerts/store/alertPresetStore.test.ts index 6919c7b25..f82598b4c 100644 --- a/src/modules/alerts/store/alertPresetStore.test.ts +++ b/src/modules/alerts/store/alertPresetStore.test.ts @@ -33,6 +33,7 @@ function makeBoard(overrides?: { name: 'Board', description: null, createdAt: 1, + updatedAt: 1, deletedAt: null, // Honor an explicit `null` (invalid config) — `??` would swallow it back to the valid default. batteryConfig: @@ -134,6 +135,7 @@ test('manual rules and other metrics survive a preset regeneration', async () => createdAt: 1, repeatEverySeconds: null, beepCount: ALERT_BEEP_COUNT_DEFAULT, + updatedAt: 1, source: 'manual', } const otherPreset: AlertRule = { @@ -147,6 +149,7 @@ test('manual rules and other metrics survive a preset regeneration', async () => createdAt: 1, repeatEverySeconds: null, beepCount: ALERT_BEEP_COUNT_DEFAULT, + updatedAt: 1, source: 'preset', } const { useAlertsStore, useAlertPresetStore } = await setup({ seedRules: [manual, otherPreset] }) @@ -197,6 +200,7 @@ test('editing an inactive board regenerates only that board rules', async () => createdAt: 1, repeatEverySeconds: null, beepCount: ALERT_BEEP_COUNT_DEFAULT, + updatedAt: 1, source: 'preset', } getAlertRules.mockImplementation(async (boardId: string) => diff --git a/src/modules/alerts/store/alertsStore.ts b/src/modules/alerts/store/alertsStore.ts index 37f6fe95e..540a69eea 100644 --- a/src/modules/alerts/store/alertsStore.ts +++ b/src/modules/alerts/store/alertsStore.ts @@ -4,12 +4,20 @@ import { getAlertRules, setAlertRuleEnabled, type AlertRule, + type AlertRuleInput, type AlertSoundType, upsertAlertRule, } from 'vescape-core' import { generateId } from '@/helpers/id' -export type { AlertSoundType } from 'vescape-core' +export type { AlertRule, AlertRuleInput, AlertSoundType } from 'vescape-core' + +/** + * Native owns `updatedAt` (the incremental-sync cursor) and stamps it from its own clock. Rules + * mirrored into local state before that write lands carry this optimistic value until the next + * `load()` replaces them with the persisted rows. + */ +const withLocalCursor = (rule: AlertRuleInput): AlertRule => ({ ...rule, updatedAt: Date.now() }) interface AlertsState { /** @@ -40,7 +48,7 @@ interface AlertsActions { load(boardId: string | null): Promise add(controlId: string, draft: AlertRuleDraft): void update(id: string, draft: AlertRuleDraft): void - upsert(rule: AlertRule): Promise + upsert(rule: AlertRuleInput): Promise setEnabled(id: string, enabled: boolean): Promise toggle(id: string): Promise remove(id: string): Promise @@ -71,14 +79,14 @@ export const useAlertsStore = create((set, get) => add(controlId, draft) { const boardId = get().boardId if (!boardId) return - const rule: AlertRule = { + const rule = withLocalCursor({ boardId, id: generateId(), controlId, enabled: true, createdAt: Date.now(), ...draft, - } + }) set((s) => ({ rules: [...s.rules, rule] })) void upsertAlertRule(rule) }, @@ -94,10 +102,11 @@ export const useAlertsStore = create((set, get) => async upsert(rule) { // Only reflect the rule locally when it belongs to the bound Board; always persist natively. if (rule.boardId === get().boardId) { + const local = withLocalCursor(rule) set((s) => { const exists = s.rules.some((r) => r.id === rule.id) return { - rules: exists ? s.rules.map((r) => (r.id === rule.id ? rule : r)) : [...s.rules, rule], + rules: exists ? s.rules.map((r) => (r.id === rule.id ? local : r)) : [...s.rules, local], } }) } diff --git a/src/modules/board/store/boardStore.test.ts b/src/modules/board/store/boardStore.test.ts index 048c6f28d..0d1c5897f 100644 --- a/src/modules/board/store/boardStore.test.ts +++ b/src/modules/board/store/boardStore.test.ts @@ -111,6 +111,7 @@ test('stored Board Link survives a store reload from native boards', async () => name: 'ADV', description: null, createdAt: 1, + updatedAt: 1, deletedAt: null, batteryConfig: null, link: null, @@ -141,6 +142,7 @@ test('updated battery config survives a store reload from native boards', async name: 'ADV', description: null, createdAt: 1, + updatedAt: 1, deletedAt: null, batteryConfig: { mode: 'preset', diff --git a/src/modules/board/store/boardStore.ts b/src/modules/board/store/boardStore.ts index f49648844..1d7c20db0 100644 --- a/src/modules/board/store/boardStore.ts +++ b/src/modules/board/store/boardStore.ts @@ -87,11 +87,15 @@ export const useBoardStore = create((set, get) => ({ alertPreset, alertPresetsOnboarded, }) { + const now = Date.now() const board: Board = { id: id ?? generateId(), name, description: description ?? null, - createdAt: Date.now(), + createdAt: now, + // Native owns `updatedAt` (the incremental-sync cursor) and stamps it from its own clock on + // the upsert below; this optimistic value only holds until the next load() replaces the row. + updatedAt: now, deletedAt: null, batteryConfig: batteryConfig ?? DEFAULT_BATTERY_CONFIG, topSpeedKmh, diff --git a/src/modules/moduleBoundaries.test.ts b/src/modules/moduleBoundaries.test.ts index 3a0004fa2..ef0df4d16 100644 --- a/src/modules/moduleBoundaries.test.ts +++ b/src/modules/moduleBoundaries.test.ts @@ -24,6 +24,8 @@ const ALLOWED_EDGES = new Set([ 'legal -> settings', // release reads/persists dismissed Community Message IDs through App Settings 'release -> settings', + // the backup choice reads/persists the Wi-Fi-only switch through App Settings + 'profile -> settings', // settings defaults sourced from owning domains 'settings -> alerts', 'settings -> history', diff --git a/src/modules/profile/components/BackupChoiceModal.tsx b/src/modules/profile/components/BackupChoiceModal.tsx new file mode 100644 index 000000000..2bb187b94 --- /dev/null +++ b/src/modules/profile/components/BackupChoiceModal.tsx @@ -0,0 +1,85 @@ +import { StyleSheet, View } from 'react-native' +import { CloudArrowUpIcon } from 'phosphor-react-native' + +import { Button } from '@/components/base/Button' +import { Text } from '@/components/base/Text' +import { FadeCardModal } from '@/components/modals/FadeCardModal' +import { theme } from '@/constants/theme' +import { useSettingsStore } from '@/modules/settings/store/settingsStore' +import { useSyncStatusStore } from '@/modules/profile/store/syncStatusStore' + +interface BackupChoiceModalProps { + /** Render regardless of the stored choice, with a fixed pending volume — the showcase. */ + preview?: { pendingRows: number } +} + +/** + * The one expensive moment in this feature's life — the first upload on a phone with months of + * history — offered as a decision rather than something that happened to the Rider. + * + * Shown once, with the pending volume, in the flow where backup is turned on. Both answers set the + * same App Setting the ordinary settings row does; afterwards this never appears again. + */ +export function BackupChoiceModal({ preview }: BackupChoiceModalProps) { + const status = useSyncStatusStore((state) => state.status) + const loaded = useSettingsStore((state) => state.loaded) + const syncEnabled = useSettingsStore((state) => state.syncEnabled) + const choiceMade = useSettingsStore((state) => state.syncBackupChoiceMade) + const set = useSettingsStore((state) => state.set) + + // Only once backup is actually on: a phone with the master switch off, or signed out, has nothing + // to decide about yet. + const visible = + preview != null || + (loaded && syncEnabled && !choiceMade && status.accountId !== null && status.pause === null) + const pendingRows = preview?.pendingRows ?? status.pendingRows + + const choose = (wifiOnly: boolean) => { + void set('syncWifiOnly', wifiOnly) + void set('syncBackupChoiceMade', true) + } + + return ( + +