Skip to content

feat(dev): published LIVE builds for Diff; apply reads Data.p4k only - #42

Merged
NyleGarcia merged 3 commits into
mainfrom
feat/dump-diff
Nov 20, 2026
Merged

NyleGarcia merged 3 commits into
mainfrom
feat/dump-diff

Conversation

@NyleGarcia

Copy link
Copy Markdown
Contributor

Published game data is now used only for comparing old vs new LIVE builds in Developer mode. Apply always reads Data.p4k on this computer.

Changes

  • Settings > Network > "Download published LIVE builds for Diff": opt-in, off by default, and persisted now (it used to be forced off).
  • Datasets: "Published LIVE builds" panel: lists LIVE releases of voidput/sc-gamedata-dumps, with download and remove buttons. A downloaded build becomes a dump:<tag> source and can be picked as A or B in Diff. Downloaded builds keep working with the switch off and when GitHub is unreachable.
  • Contract (real, Rust mock, TS mirror, TS mock): new dev_dump_releases -> DumpReleases { releases, error }; dev_download_dump(DevDumpRequest { target, tag }) runs as a dev job with progress; dev_remove_source deletes a downloaded build.
  • Integrity: the tag is pinned to its commit and each file is checked against its git blob SHA-1, size and _build stamp. SHA-256 is re-checked on load. A build may be at most 512 MiB, files are streamed to disk, and the switch is re-checked before every file.
  • Removed:
    • DumpFirst / PublishedDump (dumps feeding apply).
    • The CLI published-dumps feature and --use-published-data. The CLI has no HTTP client at all now.
    • The wizard's data-source step and SetupRequest.data_source. An old frontend that still sends it keeps working, and saved wizard drafts resume at the same step.
  • Docs: privacy, user guide, addendum (§1, §7.5, §7.6), todo and backlog (M3: app screens (setup wizard, strings table, enhancements, channels, backups) #15).

Verification

  • Rust tests: 985 pass. The 2 native_flow tests fail only because the game was running locally.
  • npm: 161 tests pass; svelte-check is clean; the build passes.
  • Real-network smoke (cargo test -p starorg-codex-app --test real_dev real_network_smoke -- --ignored): downloaded 4.10.197.2878 (17.6 MB) and 4.10.200.58746 (24.1 MB), then diffed them: 87 tables, 35 differ, +13,590 / ~1,908 records.

Refs #15

Rescope published dumps (voidput/sc-gamedata-dumps) to Developer mode's
Datasets/Diff: download past LIVE builds and compare old vs new. Apply
always reads Data.p4k locally.

- core: DumpTransport::releases, live_releases, tag-based fetch_verified
  over every data/*.json (blob SHA-1 + _build checks), DumpCache
  entry/list/remove; drop PublishedDump, DumpFirst, to_game_data,
  recover_loc_keys, match_build
- net: GitHub release list (published_at, drafts skipped)
- app contract: dev_dump_releases, dev_download_dump({target, tag}) as a
  dev job, dump:<tag> sources (SHA-256 re-checked on read, usable with the
  switch off, removable); mocks serve two synthetic LIVE builds
- settings: network.published_dumps is a real switch, "Download published
  LIVE builds for Diff"
- setup: drop the "Game data source" step and SetupRequest.data_source
- cli: drop the published-dumps feature and --use-published-data; no HTTP
- docs: privacy, user guide, addendum §1/§7.5/§7.6 rescope notes, todo
- core: refuse builds over 512 MiB before downloading; write each
  verified file to a per-download temp dir as it arrives; prune stale
  *.tmp-* dirs on listing; ignore meta.json entries with unsafe or
  case-colliding names; DumpError (not published / refused / network /
  stopped); progress callback can stop the download
- app: re-check the switch before every file (off = network-off, no
  further request); dev_dump_releases returns { releases, error } so an
  offline or rate-limited GitHub still lists downloaded builds; network
  failures say "Could not reach GitHub"; checksum text without the
  "archive check failed" prefix; missing or unreadable cached files put
  the source in Error
- mocks match the real backend: invalid for a bad tag, same not-found
  text, same progress steps incl. "Checking"
- net: follow up to 3 pages of 100 releases
- tests: failing transport, switch off mid-download, progress sequence,
  build cap, case collisions, stale temp dirs
@NyleGarcia
NyleGarcia merged commit f855898 into main Nov 20, 2026
15 of 16 checks passed
@NyleGarcia
NyleGarcia deleted the feat/dump-diff branch November 20, 2026 00:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant