You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The uploader. Native scans each table forward from its Sync Cursor, sends small Sync Batches, and advances only what the server accepted.
It runs in vescape-core on both platforms, inside the window the app already keeps alive: the foreground service while a Board Session or GPS is active on Android, the existing background modes on iOS. No WorkManager and no BGTaskScheduler — a ride that ends offline on a phone that is then never opened waits for the next app open or the next ride, and that is accepted.
Runtime shape:
RIDE_INTERVAL 30s while a Board Session is producing samples
IDLE_INTERVAL 5min no-op when nothing is pending
BATCH_ROW_CAP 1000 total across every table
BATCH_BYTE_CAP 1 MiB of compact UTF-8 JSON
BACKOFF 30s, doubling, capped at 15min, reset on success
Immediate kicks on connectivity regained, ride end and sign-in. On a 200 with rows still pending it sends again straight away rather than waiting for the next tick, so a long backlog drains instead of trickling.
Two pieces carry all the interesting behaviour and both are pure, with no database, clock or network:
Batch building. Given per-table pending readers, current cursors, total-row and exact encoded-byte caps, return a batch and its cursor advance set. It walks the tables in the order the server writes them, where parents precede children, and stops at whichever cap comes first. It measures compact UTF-8 wire JSON rather than estimating object size. It must never order by backlog size.
Policy. Given whether a ride is producing samples, the network, the Wi-Fi-only setting, the pending count, the last result and the time, return send now, wait until, or paused.
cursors are written after the response, in their own transaction, never alongside the rows. A cursor advanced past rows the server did not take is unrecoverable; a cursor left behind is a re-send the server upserts idempotently. Always fail toward re-sending.
cursors are scoped to the one Account bound to the whole local database. Same-Account sign-in keeps them. A confirmed different-Account switch replaces the database and therefore starts with fresh cursors; there are no per-Account rows or cursor partitions.
Retention safety
A retention cutoff is only a candidate cutoff once the database belongs to an Account. Cleanup must not remove a row the uploader has not yet delivered.
never-bound database -> current age-only cleanup
bound database -> age cutoff AND row cursor <= accepted Sync Cursor
confirmed switch -> replace whole database; bypass retention intentionally
For append-only telemetry_frames, telemetry_markers, diagnostic_events and metric_exclusion_ranges, the row cursor is the existing AUTOINCREMENT id. For mutable telemetry_minute_buckets, it is sync_seq; this protects an old bucket that was rewritten after its earlier version uploaded. A missing cursor is treated as 0, meaning no rows in that table are safe to prune.
Each retention sweep reads its table cursor and deletes inside one database transaction. Racing an upload therefore fails safe: before cursor commit it retains the rows, after cursor commit the server has accepted them. Retention emits no Sync Actions. Sign-out does not remove the Account binding, so data recorded while signed out remains protected for the same Account.
Account change reset
The Device Token exchange returns a stable server Account id. The first Account claims the existing local database. Signing back into that same Account keeps all local data and cursors.
A different Account does not reset cursors over the existing rows — that would upload the previous Account's Boards, Ride History, locations and settings to the new Account. Native instead reports accountChangeRequiresReset; JS shows an explicit destructive warning that cloud restore is not available in this version. Only after confirmation does native perform one ordered transition:
stop uploader
-> invalidate/cancel in-flight work
-> replace app-data database with a fresh database
-> clear Sync Cursors + pending Sync Actions
-> bind fresh database to new Account id
-> install new Device Token
-> start uploader
The wipe is local maintenance, emits no Sync Actions, and must not race an old response. A generation captured by each request is checked before cursor commit, so a response from the previous Account becomes a no-op. Cancelling the warning leaves the old database and Account binding untouched.
Wire validation and poison batches
Native builds strongly typed wire DTOs and validates the constraints mirrored from KacperKozak/vescape-server#14: required and unknown fields, enum/target names, key lengths, integer bounds, finite numbers, total row count and actual compact UTF-8 bytes. Contract fixtures include valid and invalid boundary rows and run against Kotlin, Swift and the server decoder.
A batch is whole or refused. Native never advances past, skips or silently quarantines a bad row in this version. A single row that cannot encode or exceeds 1 MiB is a permanent local protocol error and pauses the engine with that row retained.
Transport/result policy:
200 accepted
response schema valid AND every table count == submitted count
-> commit cursor advance set
400 invalid-request / 409 dependency-conflict / 422 / other unknown 4xx
-> permanent pause, no cursor movement
401
-> dead credential, pause for sign-in
413
-> retry with a smaller byte target
-> if a single row still fails, permanent pause
429
-> wait for Retry-After
5xx / network / timeout
-> transient exponential backoff
App Status gate responses keep their existing Online Capability handling. A malformed or unexpected 2xx response is a permanent protocol failure. A timeout can happen after the server committed, so the unchanged batch is resent idempotently.
A permanent pause writes one coalesced Diagnostic Event keyed by failure class, table and cursor, then stops automatic retries; ordinary timer/connectivity/sign-in kicks do not bypass it. The event contains only error code, table, cursor and app version — never row contents, coordinates, Device Token, server body, SQL or opaque database errors.
Sync endpoints are Online Capabilities and sit behind the existing App Status gate with every other server-backed capability.
Likely files
modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/ - the new sync package, cursor store and cursor-gated retention queries alongside the data they protect
modules/vescape-core/android/src/main/java/expo/modules/vescapecore/service/CoreForegroundService.kt - the process that stays alive during a ride
modules/vescape-core/android/src/main/java/expo/modules/vescapecore/runtime/Scheduler.kt - the existing scheduling seam, with TestScheduler as its fake
modules/vescape-core/android/src/main/java/expo/modules/vescapecore/appstatus/AppStatusCoordinator.kt - the Online Capability gate and existing OkHttp usage
modules/vescape-core/ios/telemetry/, modules/vescape-core/ios/runtime/, modules/vescape-core/ios/appstatus/ - the peers, including cursor-gated retention in TelemetryRepository
modules/vescape-core/src/index.ts - wire/status contracts native mirrors to TS
KacperKozak/vescape-server#14 - server byte/row bounds and stable error contract this uploader mirrors
docs/adr/0010-pure-logic-modules-in-native-services.md - the pattern the two pure modules follow
Implementation hints
Scheduler / TestScheduler already exist for exactly this: driving a timed loop in tests without real time. Use them rather than a new timer abstraction.
The server writes a batch as one transaction, table by table, in its own declared table order — that order is the dependency order, and the builder walking the same order is what keeps a batch valid. Favorites use the contract from KacperKozak/vescape-server#15; Favorite metadata is an ordinary mutable Sync Table, while Favorite Media bytes remain outside this uploader in #293. Read the server's batch writer before choosing an order of your own.
The server refuses a whole batch rather than half-applying it, so a wedged batch leaves every cursor untouched. Validate the 200 accepted map exactly before cursor commit; a missing table, extra table or mismatched count is a protocol failure.
Keep compact encoding deterministic enough to measure the actual payload produced on each platform. The pure builder may accept encoded row sizes, but the transport rechecks the final request bytes before sending.
GroupRideObserver shows how native holds a long-lived server connection and how it reacts to the online gate closing. The uploader has the same lifecycle concerns and should not invent a second answer to them.
Idle Pause halts sample persistence while keeping the Board Session live at a reduced poll rate. The loop's ride interval should follow whether samples are actually being produced, not merely whether a session exists.
Acceptance criteria
Native uploads pending rows, including Favorite metadata, to the server with no JS involvement
Uploads continue with the app backgrounded and the screen off during a ride
Batches fill in dependency order under 1000 total rows and 1 MiB actual compact UTF-8 JSON
Exactly-at and one-over row/byte boundaries behave identically on Android and iOS
Native wire DTOs validate strict fields, enum cases, key lengths, integer bounds and finite numbers before transport
Valid/invalid fixtures are mirrored against KacperKozak/vescape-server#14
The cursor advance set describes exactly the rows sent, and nothing more
Cursors advance only after a successful response, in their own transaction
A refused batch leaves every cursor untouched
A failure part-way through a drain leaves cursors at the last accepted batch
Re-sending the same batch changes nothing on the server
A 200 advances cursors only when its response is valid and every accepted count exactly matches the submitted batch
400, 409, 422, unknown 4xx and malformed success responses permanently pause without cursor movement
401 pauses for authentication; 429 respects Retry-After; 5xx, network and timeout use backoff
413 reduces the byte target; a single row that still fails pauses without being skipped
A timeout after server commit safely resends the identical batch
A permanent failure records one coalesced metadata-only Diagnostic Event and does not retry on ordinary kicks
A bound database retains every old row whose table cursor has not passed it
A bound database prunes old append-only rows only when id <= accepted cursor
A bound database prunes old minute buckets only when sync_seq <= accepted cursor
An old minute bucket rewritten after upload survives retention until its new sync_seq is accepted
A missing table cursor protects every row in that table
Sign-out keeps retention protection for data subsequently recorded under the bound Account
A never-bound database keeps the existing age-only cleanup behaviour
Retention reads the cursor and deletes atomically and emits no Sync Actions
Backoff grows and resets on success; a 401 stops the loop permanently
First sign-in claims the existing local database for the returned stable Account id
Signing back into the same Account preserves local data, Sync Cursors and pending Sync Actions
A different Account cannot activate sync until the Rider confirms that all local app data will be erased and cannot yet be restored
Confirming replaces the app-data database, clears cursors/actions, binds the new Account, then starts sync
The Account-change wipe emits no Sync Actions to either Account
Cancelling the warning leaves the old database and Account binding untouched
An in-flight response from the previous Account cannot advance a cursor or mutate the fresh database
Sync stops when the App Status gate closes, like every other Online Capability
Batch building and policy are pure and tested with no database, clock or network
The engine is tested against a fake transport for the wedged batch, mid-drain failure, dead token and stale previous-Account response cases
Both platforms behave identically, linked by @parity
Parent
What to build
The uploader. Native scans each table forward from its Sync Cursor, sends small Sync Batches, and advances only what the server accepted.
It runs in
vescape-coreon both platforms, inside the window the app already keeps alive: the foreground service while a Board Session or GPS is active on Android, the existing background modes on iOS. NoWorkManagerand noBGTaskScheduler— a ride that ends offline on a phone that is then never opened waits for the next app open or the next ride, and that is accepted.Runtime shape:
Immediate kicks on connectivity regained, ride end and sign-in. On a
200with rows still pending it sends again straight away rather than waiting for the next tick, so a long backlog drains instead of trickling.Two pieces carry all the interesting behaviour and both are pure, with no database, clock or network:
Cursors:
AUTOINCREMENTkey; mutable tables, including Favorites after [Sync] 3 - Add sync_seq to six tables #281, scan onsync_seqRetention safety
A retention cutoff is only a candidate cutoff once the database belongs to an Account. Cleanup must not remove a row the uploader has not yet delivered.
For append-only
telemetry_frames,telemetry_markers,diagnostic_eventsandmetric_exclusion_ranges, the row cursor is the existingAUTOINCREMENTid. For mutabletelemetry_minute_buckets, it issync_seq; this protects an old bucket that was rewritten after its earlier version uploaded. A missing cursor is treated as0, meaning no rows in that table are safe to prune.Each retention sweep reads its table cursor and deletes inside one database transaction. Racing an upload therefore fails safe: before cursor commit it retains the rows, after cursor commit the server has accepted them. Retention emits no Sync Actions. Sign-out does not remove the Account binding, so data recorded while signed out remains protected for the same Account.
Account change reset
The Device Token exchange returns a stable server Account id. The first Account claims the existing local database. Signing back into that same Account keeps all local data and cursors.
A different Account does not reset cursors over the existing rows — that would upload the previous Account's Boards, Ride History, locations and settings to the new Account. Native instead reports
accountChangeRequiresReset; JS shows an explicit destructive warning that cloud restore is not available in this version. Only after confirmation does native perform one ordered transition:The wipe is local maintenance, emits no Sync Actions, and must not race an old response. A generation captured by each request is checked before cursor commit, so a response from the previous Account becomes a no-op. Cancelling the warning leaves the old database and Account binding untouched.
Wire validation and poison batches
Native builds strongly typed wire DTOs and validates the constraints mirrored from
KacperKozak/vescape-server#14: required and unknown fields, enum/target names, key lengths, integer bounds, finite numbers, total row count and actual compact UTF-8 bytes. Contract fixtures include valid and invalid boundary rows and run against Kotlin, Swift and the server decoder.A batch is whole or refused. Native never advances past, skips or silently quarantines a bad row in this version. A single row that cannot encode or exceeds 1 MiB is a permanent local protocol error and pauses the engine with that row retained.
Transport/result policy:
App Status gate responses keep their existing Online Capability handling. A malformed or unexpected
2xxresponse is a permanent protocol failure. A timeout can happen after the server committed, so the unchanged batch is resent idempotently.A permanent pause writes one coalesced Diagnostic Event keyed by failure class, table and cursor, then stops automatic retries; ordinary timer/connectivity/sign-in kicks do not bypass it. The event contains only error code, table, cursor and app version — never row contents, coordinates, Device Token, server body, SQL or opaque database errors.
Sync endpoints are Online Capabilities and sit behind the existing App Status gate with every other server-backed capability.
Likely files
modules/vescape-core/android/src/main/java/expo/modules/vescapecore/telemetry/- the new sync package, cursor store and cursor-gated retention queries alongside the data they protectmodules/vescape-core/android/src/main/java/expo/modules/vescapecore/service/CoreForegroundService.kt- the process that stays alive during a ridemodules/vescape-core/android/src/main/java/expo/modules/vescapecore/runtime/Scheduler.kt- the existing scheduling seam, withTestScheduleras its fakemodules/vescape-core/android/src/main/java/expo/modules/vescapecore/appstatus/AppStatusCoordinator.kt- the Online Capability gate and existing OkHttp usagemodules/vescape-core/ios/telemetry/,modules/vescape-core/ios/runtime/,modules/vescape-core/ios/appstatus/- the peers, including cursor-gated retention inTelemetryRepositorymodules/vescape-core/src/index.ts- wire/status contracts native mirrors to TSKacperKozak/vescape-server#14- server byte/row bounds and stable error contract this uploader mirrorsdocs/adr/0010-pure-logic-modules-in-native-services.md- the pattern the two pure modules followImplementation hints
Scheduler/TestScheduleralready exist for exactly this: driving a timed loop in tests without real time. Use them rather than a new timer abstraction.The server writes a batch as one transaction, table by table, in its own declared table order — that order is the dependency order, and the builder walking the same order is what keeps a batch valid. Favorites use the contract from
KacperKozak/vescape-server#15; Favorite metadata is an ordinary mutable Sync Table, while Favorite Media bytes remain outside this uploader in #293. Read the server's batch writer before choosing an order of your own.The server refuses a whole batch rather than half-applying it, so a wedged batch leaves every cursor untouched. Validate the
200 acceptedmap exactly before cursor commit; a missing table, extra table or mismatched count is a protocol failure.Keep compact encoding deterministic enough to measure the actual payload produced on each platform. The pure builder may accept encoded row sizes, but the transport rechecks the final request bytes before sending.
GroupRideObservershows how native holds a long-lived server connection and how it reacts to the online gate closing. The uploader has the same lifecycle concerns and should not invent a second answer to them.Idle Pause halts sample persistence while keeping the Board Session live at a reduced poll rate. The loop's ride interval should follow whether samples are actually being produced, not merely whether a session exists.
Acceptance criteria
KacperKozak/vescape-server#14200advances cursors only when its response is valid and every accepted count exactly matches the submitted batch400,409,422, unknown4xxand malformed success responses permanently pause without cursor movement401pauses for authentication;429respectsRetry-After;5xx, network and timeout use backoff413reduces the byte target; a single row that still fails pauses without being skippedid <= accepted cursorsync_seq <= accepted cursorsync_seqis accepted401stops the loop permanently@parityBlocked by
Related